Building an online marketplace from scratch — the dark side of Salesforce limits

This series of articles describes one of the modern approaches to designing and building e-commerce applications. It focuses on showing how to ship CRM-related features fast, so the business gets decent time to market. If you want to get along with business people or you’re thinking of becoming a CTO, then this series might well be worth reading.

This is the second installment of a series of posts, here you can quickly jump to other parts:

• Introduction
• Order processing

In the last post, we covered how to use Salesforce (SF) to embrace the chaotic and Excel-driven order management process in Manufaktura. We saw how SF can automate lots of manual jobs and reduce the amount of error handling by taking care of data integrity. But from both a business and tech point of view, SF alone is not enough to constitute a speed-oriented software architecture. In this article, we’ll show you why and then how to use SF API to connect your IT components into a well-oiled platform.

The API

It’s a little-known fact that SF, a $50 billion company, generates 50% of its revenue through its API. There has to be something in it, right? In the following chapters, we’ll try to explain what makes it so popular among businesses and how an early stage business like “Manufaktura” can use it to scale their operations.

In fact, the REST API is going to be a core concept of Manufaktura software architecture and thus of further articles in the series. It will become the main pillar of the framework that Manufaktura is built on. A leverage which allows the Manufaktura team to deliver business results more quickly while simultaneously saving enormous amounts of engineering time.

But before we use the SF REST API, let’s see how you can adjust your processes system without tapping into external services. This should give you more knowledge about what can be done within SF and its internal programming environment.

The first problem of scale

The problem the team stumbled upon is the outburst of initial not-so-successful relationships. One of the manufacturers didn’t manage to meet the designer’s expectations and so, the latter didn’t want to work with them anymore. The situation occurred a couple of times for multiple designer — manufacturer pairs and, therefore, remembering which manufacturer should be excluded from the tender became problematic. The dev team was given a task to automate this.

Fortunately, this issue can be quickly solved with an SF Apex code. All we need to do is go to the ManufacturerSelectJob, you can find it in our previous article, and add a new business rule.

The feedback will be collected by customer service agents after an order is finished/completed and they will enter the data into a newly created Order’s field — for a while we will keep data entry as manual.

So, before we change the code, we have to modify the related SF objects accordingly.

1. Add a picklist field “Feedback” to the “Order” object with the following values: POSITIVE, NEGATIVE

2. Add hidden formula field called “Feedback Filter” to “Order” object which will be used in the query, it will concat Account Id and Manufacturer Id

3. Add a hidden formula field called “Feedback Filter” to the “Order Proposal” object that will be used in the query, it will concat Account Id (from Order) and Manufacturer Id (Introduced to work around the SOQL limit — direct field to field comparison is not supported in SF, more info)

4. Remove the scheduled job to allow code modification (Setup ->Environments/Jobs/Scheduled Jobs)

5. Edit ManufacturerSelectJob: (modified lines start with the “+”)

Quick, right? We improved an important feature and therefore a whole process within 30 minutes. Plus, you can immediately generate more value out of this change by creating a report which sums up manufacturers’ performance:

‍You got the flow? A quarter of configuring and scripting here and there and bam, the business gets value. And this doesn’t require us to tap into external databases or other systems, we can store and maintain data structures like this one easily within SF objects.

What’s most important is, within an hour or so, we helped the operations team reduce the number of painful assignment mistakes which, when accumulated, might cause a lot of harm to Manufaktura’s trustworthiness.

But as you saw in the 3rd point above (the workaround), SF and its limitations don’t always allow for such a seamless experience.

When to go outside Salesforce

Limits

Sometimes you can purchase an extra pack, sometimes the limit is fixed and you have to work it around. If the latter is true or if paying more isn’t an option, you can transfer the responsibility of storing/processing/calculating to an external system, either your bespoke module or a 3rd party tool.

Dev experience

1. Poor source control integration
2. No continuous integration
3. Long deployments:

• You can’t add a new code to production immediately, you have to wait up to 30 minutes.
• Poor support for multi-developer/multi-team environmentsToo many platform limits (see above)No local execution emulator that doesn’t require server round trips of saving Apex code (this has changed with DX, see below)
• Limited Java API exposure in Apex. No binary file manipulation, image/sound APIs, or utility APIs

This translates into a cumbersome developer experience (DX). Devs can’t get into their flow like they can with different technologies. The platform reminds them of its limitations on every corner. And poor DX translates into drops in motivation and less productivity in the end.

The good news is that SF listened to the community and just released Salesforce DX — a new tooling that addresses some of these issues. Is their long-standing motto “No software” finally going to fade away? :)

Lastly, if you want to lessen your source code’s reliance on proprietary language (or you’re having problems finding devs with the SF skill set) and have to work around SF’s limits, you should consider implementing and integrating an accompanying system. Let’s see how you can do it so that it doesn’t mess with our speed-oriented software development methodology.

How to connect SF to the outside world

Input

Let’s take more of the burden off the customer service team’s shoulders. We will replace the manual data entry of new orders with an online form. We have 2 options here:

• quickly sketch up a custom website which connects to the SF API through JS Force or another SDK, or
• use a form generator and add new records through Zapier.

To speed things up, we’ll take the 2nd path. Let’s employ Typeform which is a popular online form builder. As you might’ve guessed, the first step is to create a simple form like this one: https://bandro.typeform.com/to/q2jhFL

Next, we need to send the completed forms to SF. As both SF and Typeform are on the Zapier store, the configuration is straightforward. You just need to hook on a new order and create respective Account and Order objects.

‍This setup only takes ~30–60 minutes (depending on the level of your familiarity with Typeform and Zapier).

‍Notifying external components

In this section, we’ll show you how to send automatic notifications when there’s a change of state in one of the SF objects. Let’s create a trigger which will push a JSON payload to an external API.

1. Go to Setup -> RemoteSiteSettings and whitelist a remote address you want to connect to.

2. Create a class, e.g. BackendService, which is responsible for creating an HTTP request, @future(callout=true) annotation makes the request async.

3. In Setup -> Apex Triggers, add a simple trigger watching Orders. It compares order status and invokes the BackendService, passing the relevant objects as JSON.

4. Implement a service consuming the request, example in Node.js.

Again, a couple of hours of a single developer’s time and the important feature got automated.

Data analytics unleashed

When the traffic rises to millions of requests and the SF API becomes too slow or too costly, you can always resort to Heroku Connect (HC). This platform allows you to mirror an SF database as a regular Postgres instance. In this way, you can point data-sensitive applications to the follower database.

HC gives you 10k records for free right after you install the addon. If you want more than this, you should contact your SF sales rep and negotiate. There are no clear rules here, but the more user licenses you buy, the more HC records you get.

How it works — you need to buy a database in the Heroku marketplace, add the HC addon, and finally map which objects should be synchronized. HC allows you to sync objects in the range <10, 60> minutes. What you get as a result is the ability to run SQL-based reports without utilizing any SF limits, e.g. you can connect one of the popular BI tools like tableau.

What can you configure about synchronization:

• Which objects
• Which fields
• Poll frequency

Moreover, HC supports bi-directional synchronization which means that we can modify data within the postgres instance and the changes will be automatically applied on SF records. Speaking of synchronization, I should also say that HC takes care of metadata changes propagation. If you remove a field in SF, it disappears in the postgres table too. Note: SF is always the master of data.

Surprisingly, every call to SF from HC doesn’t utilize the daily API limit either! This is then a perfect tool to build data exposing UI applications.

Although HC is a mature and powerful tool, it’s still under development. New features come, and some go — they’ve recently withdrawn support for formula or roll-up summary type. So, consult the docs before you decide to give it a chance. Also, when it comes to pricing, here’s an interesting fact: the negotiated limits are not strict and HC won’t block you when you exceed them. All in all, you should discuss it with your account manager again.

Scaling & pricing

Lastly, we’d like to share some pricing guidelines/watchouts which help you prepare your SF billing forecast and perhaps decide which functionalities might be taken out to external service.

Let’s start with the list of limits you should be aware of:

• 15k API calls for starter
• Strict email limits
• Maximum amount of code used by all Apex code in an org — 3MB, see other code-related limits
• Data/file storage — e.g. data storage is 1GB per one user license
• Number of custom reports
• 4000 characters for a db query and other SOQL limits

Studying the limits and their impact on the bill is a key exercise to be done when deciding if something should be coded using SF objects and APEX or outsourced to your bespoke system via the API.

Summary

Within a day or so, we managed to implement 2 crucial features for Manufaktura. Step by step, our platform is automating more and more processes, allowing customer service, marketing, and the operations team to take care of other hot topics in the fast-growing company.

We’ve also learned when and how to move outside SF (with REST API and Heroku Connect) to do the job in the relevant/favorite technology or/and without hitting SF limits.

The small codebase and flexible architecture allow us to introduce changes super-fast and scale. On this basis, we can now jump into automating subsequent department processes. Almost, because there’s one thing which doesn’t scale and it’s gonna hit us when new features are released fast and pile up — it’s the error handling. In the next post, we’ll suggest a solution which helps you build a safety net which is tight enough to keep the business running and lean enough to be maintainable when your source code base starts to swell.

Originally published at www.voucherify.io.