Building an online marketplace from scratch — order processing

This series of articles describes one of the modern approaches to designing and building e-commerce applications. It focuses on showing how to ship CRM-related features fast, so the business gets decent time to market. If you want to get along with business people or you’re thinking of becoming a CTO, then this series might well be worth reading.

This is the second installment of a series of posts, here you can quickly jump to other parts:

• Introduction

Welcome back!

In this post, we’ll describe how to design and implement order management functionality for Manufaktura (our online marketplace business).

As you read in the previous post, Manufaktura is an early stage business. This means that orders are handled manually, emails go back and forth, customers and manufacturers want to be updated about the status, and new “exceptions” keep popping up every day. In such conditions, building an order management system is like dismantling a bomb. You need a short-term and a long-term solution. Short — turn off the timer, long — take away the explosives otherwise the problem isn’t solved.

Translating this into the software world, it means you need to figure out how to automate order processing as asap as possible. Otherwise, your team will sink in manual work. At the same time, you need to keep in mind business scale and plan how you’re going to map long-term data analysis out. Without the latter, your quick & dirty solution will start generating more and more problems in operations and will impede drawing insights from data.

So exactly how do you approach order management with a 3rd party SaaS & API platform approach then?

Meet Salesforce and its API platform.

We’d like to show you how an early business can use a SaaS giant to:

• build a base for a software platform,
• respond to ever changing requirements,
• decrease time to market.

But before we jump into architectural stuff, let’s get an overview of what the business requirements are. This is crucial to learn how Salesforce is going to help us solve the problems and to see what makes it a leader in its category.

Requirements in a nutshell

There’re a couple of tasks which make this project challenging. We have several people involved in the process, and each of them brings their must-haves to the table. Even though it’s an early stage business, order processing won’t be efficient without taking into account the following facets:

• Lead/contact management — marketing
• Order data validation — operations
• Marketing channel tracking — marketing
• Status tracking and pipeline management — sales/operations
• Responsible agent — customer service
• Reporting — sales/operations/finance

Apart from these basic CRM features, we need quote management to cover the order workflow. Speaking of which, here’s a sketch of the order process we want to map with the platform:

ROLES:

• Hardware designer (HD) — wants to have their product manufactured
• Manufaktura (M) — our business, online marketplace
• Manufacturers (MS) — bid for the contract

WORKFLOW:

1. HD submits RFP
2. M screens the RFP and requests an in-advance partial payment from HD
3. M starts a call for bids
4. MS submit their proposals
5. M selects the winner according to a current strategy
6. The manufacturing starts and will run through several stages (see the scheme below)
7. Product Validation Tests are finished, mass production starts
8. The first batch is delivered, M charges HD full price
9. HD runs a QA session — if all is fine, MS gets paid by M

As you can imagine, in reality, the process has many corner cases and potential problems at every step. It’s our role as developers to solve them with the software and so, off we go to the planning session! Let’s design the database scheme, choose the database vendor, count story points in JIRA, argue over front-end frameworks, and hold an hour-long discussion on how to tackle a super specific corner case, twice.

That’s one way of doing it, but before you gather up your team in front of a whiteboard, take a minute to go over these points:

• FIRST, although this approach gives value by letting developers adapt to each other and improve their social skills, like the art of reasoning and arguing, it takes time. And the business people can’t wait that long. They are very eager to have their orders automated.
• SECOND, remember it’s a new business and a lot of theories and, therefore, requirements have not been validated with the market. The organizational structure of the company isn’t known yet either. All this brings the risk that your design and estimations won’t be valid in a month or so.
• THIRD, at this stage of the business, some specific and rarely occurring cases can be done manually by non-tech people. Or, at least you can use some smart bypass like a magic Excel formula or a 10-line bash script invoked when necessary. With small scale, there’s no need for full-blown automation for every cumbersome job.

If these points are valid for your case, let’s put off a big planning meeting and think of a faster way to get the order management system up and running.

Prioritize & iterate

To achieve decent time to market you should do 3 things:

• Narrow down the real business pain points
• Research tools that can fix these pain points and can be adapted to your context
• Solve the problem with the least amount of code

So, instead of hand-picking tools for your dream tech stack and building a skeleton of the application with Jenkins, one-click builds, triple-layer test net, let’s outline the showstoppers, break them down to what’s really crying out for automating and plan the very next iteration.

In the next sections, we’re going to show you how you can embrace the “prioritize & iterate” process and finally how to implement the critical elements for the business — fast.

We’ll focus on the first and most cumbersome part of Manufaktura’s process, so:

• Validating and storing requests details from HDs
• Handling bids
• Announcing the winner and commissioning the manufacturing process

(We’re leaving some parts of the process for the next posts, e.g. payments, customer support, promotions)

But first of all, we need to replace orders.xlsx because it’s becoming more and more difficult to maintain by the ops and marketing folks.

Alright, so let’s recap — the business requirements emerged and now we can get to the heart of the matter, let’s look around for tools which can help us out. If your case sounds similar (you need to basically build wrap an order management process with software), give Salesforce a chance.

Let’s take a tour around the Salesforce platform to understand its features and ultimately to see why it’s a tool worth considering for fast pace software development.

Salesforce — the mighty platform

Ok, you probably had enough of business requirements talk by now. But we had to go through at least the basic ones to show you how much Salesforce gives you out of the box.

If you (like me 3 years ago) have paid little attention to what Salesforce has produced over the last decade, you might think they’re just a Software as a Service CRM application. Another app you have to deal with because VP Sales signed up for it. But it’s not the case anymore. Salesforce has evolved into a massive software platform for building business applications.

If you’re willing to spend some time learning their ins and outs, it will turn your software tech team into a productivity monster. For example, pretty much everyone I know involved in implementing business apps complaints about a helluva lot of tedious jobs:

• Connecting application user interface to DBs
• User data validation
• The process of DB schema change
• Handling users, roles, and profiles
• Controlling access to records and fields, adapting user interface to different roles
• Handling concurrency and transactions
• Allowing for end-user reconfigurability

At the same time, these mundane problems are super important for companies because data integrity is one of the pillars of sound business. And the online marketplace we’re building here is no different.

Now imagine that you can implement all of them, saving enormous time and energy for your team. How? Salesforce team has already solved these issues for you. As Dan Appleman points out in the article on why Salesforce might become another VB:

• When you add tables or fields to the Salesforce platform database, they appear in the language as objects with properties, already strongly tied to the correct data type.
• If you reference those objects and properties anywhere, the underlying tables and fields can’t be deleted or changed to an incompatible type. The platform creates and enforces data integrity.
• When fields are displayed on the user interface, which is easily customizable without programming, validation and the correct user interface for the fields just happens automatically.
• Security is baked in — if a given profile isn’t allowed to access a table or field, they just quietly disappear from the user interface for users that aren’t allowed to access them. That applies to forms and reports (reporting and charting being built-in as well).

Adding a custom field

And Salesforce’s approach to mapping custom, more advanced business processes also works like a charm. Most of the complex scenarios can be built with the interactive process builders. Want to come up with a multiscreen form with validated inputs? No problem, open up the builder and define your scenario using the drag and drop wizards. No coding…

… unless you have to. If you need more flexibility or a custom feature, then there’s a built-in Java-like programming language (Apex) and the ability to connect to 3rd party apps through the API. More about that in the next post. Let’s get back to what Salesforce offers for Manufaktura’s requirements out of the box.

Salesforce — what’s inside the box

When it comes to Manufaktura, Salesforce offers robust built-in order management features. What we get is a range of standard CRM functionalities, including:

• Contact, account, and order management
• Order status tracking
• Sales pipeline visualization
• Business reports
• Agents management

Salesforce offers almost 30 built-in objects which help you model your entire business

In case you didn’t notice, this list covers almost all the items our stakeholders requested for Manufaktura. As we learned in the previous paragraph, these built-in features can be customized without coding. You can add/modify/remove fields, link different entities, restrict access to particular records, create custom dashboards, and finally build custom reports on top of it all.

With these features, you can get Manufaktura’s workflow mapped and ready to handle new orders by the end of the day. This would include:

• Configuring built-in objects to match the requirements
• Defining order statuses according to the process
• Automating the manufacturer selection process

Important note: it’s all quite easy even if you don’t have Salesforce experts in your organization. SF also provides a superb site with educational resources at https://trailhead.salesforce.com. By going through the modules, you can quickly wrap your head around basic and semi-advanced topics from a user and admin point of view.

Let’s see what it takes to prepare a real-life Salesforce-based software architecture for our processes. We’re going to create a semi-automatic order management process for the following steps:

• An order is requested by a Hardware Designer and it gets PENDING VALIDATION status — the tender effective date should be at least 3 days in advance so that there’s time for submitting offers.
• After the operations team verifies the order, it goes to AWAITING PROPOSALS stage. Manufacturers can start submitting their proposals.
• Every day, in the morning, our engine selects the best price offer and the order ends up with a MANUFACTURER SELECTED, or CANCELLED NO PROPOSALS status if no proposals have been sent.

This is a simplified visualization of the relationships and the process (the entity names in square brackets stand for built-in Salesforce objects)

Relationships

Order workflow

We’ll start off by adding 2 account types for our marketplaces: Hardware Designers and Manufacturers.

1. Go to Configure Accounts (Setup -> Object Manager -> Account):
2. Add Account Record Types: Manufacturer and Hardware Designer.

The second step is to sketch out the process’ workflow; the initial stages will look like these:

• PENDING VALIDATION,
• AWAITING PROPOSALS,
• MANUFACTURER SELECTED,
• CANCELLED NO PROPOSALS.

Here’s how to do it:

Go to Configure Orders (Setup -> Object Manager -> Order) and define Statuses in Draft Category.

Now let’s make a field for linking the Order to a winning Manufacturer. We’ll do it with a custom object Order Proposal. It’s a regular “one to many” relationship — Manufacturers submit Order Proposal for a specific Order. This is how to create it:

Go to Object Manager -> Create Custom Object and:

1. Define name as Auto Number with the following pattern: OP-{0000000000}

2. Enable the following features: Allow Reports, Allow Activities, Track Field History, and Allow object in Search Result. We can also allow adding Notes and Attachments for future reference.

3. Add new fields to the created object:

• Order as a required Lookup Relationship to Account Order
• Manufacturer as a required Lookup Relationship to Account and define a filter for Record Type equals to Manufacturer
• Description as a LongText field
• Price as a required Currency field

Now let’s setup this relation from the Order point of view. Add the following fields (see the screenshots below):

• Proposal as a Lookup Relationship to Order Proposal and define a filter for Order ID at Proposal (it must match the current Order ID)
• Manufacturer as Lookup Relationship to Account and define a filter for Record Type equals to Manufacturer
• Manufacturer Selected Date as DateTime field

Great, we’ve just got the basic relationships modeled. Now, let’s add some constraints so our process is more robust and less error-prone. First, add a Validation Rule for orders with AWAITING PROPOSAL status which enforces that the Order effective date must be at least 3 days in the future. In this way, we avoid sending outdated RFPs to Manufacturers.

Then, add a rule saying: when the Order gets MANUFACTURER SELECTED status, the Manufacturer and Proposal must be chosen. Again, this time the system won’t bother HDs if somebody from the operations department forgot to fill in the necessary information. It’s just a bit of extra validation measures to reduce the communication chaos.

Apart from rapid business logic development, Salesforce gives you pretty decent control over how the objects present the information to the user. The powerful Layout Designer lets you configure multiple layouts for the same object. You can improve the UX by revealing/hiding different information for different user roles. On top of that, it offers a dozen or so built-in widgets, Preview As…option, and even Undo/Redo functionality.

The basic structure and constraints have been modeled, let’s move the needle now. The next section describes how the selection of the winning Manufacturer works. The process looks as follows:

1. The selection process is run every day at 7 am.
2. The script goes through all AWAITING PROPOSALS orders and selects the proposal with the best price (this is business criteria for the moment).
3. The orders which didn’t get any traction are marked as CANCELLED NO PROPOSALS.

Because this is a slightly more complex scenario, you can’t do it with the built-in process builder. We simply have to code it. As mentioned, Salesforce has got you covered here. Let’s see how easy it is to instruct Salesforce how to handle our processes. It’s actually no different than most popular programming environments (see this guide to learn more).

Let’s create a class in the built-in sort of IDE. Go to Setup -> Custom Code -> Apex Classes and put the logic inside:

global class ManufacturerSelectJob implements Schedulable {

global void execute(SchedulableContext SC) {

List<order_proposal__c> acceptedProposals =

[ SELECT Id, manufacturer__c, order__c

FROM order_proposal__c

WHERE order__r.EffectiveDate < TODAY

AND order__r.Status = 'AWAITING PROPOSALS'

ORDER BY order__c ASC, price__c ASC ];

List<Order> acceptedOrders = new List<Order>();

Set<String> acceptedOrderIds = new Set<String>();

for (order_proposal__c acceptedOrderProposal : acceptedProposals) {

if (acceptedOrderIds.contains(acceptedOrderProposal.order__c)) { continue; }

acceptedOrderIds.add(acceptedOrderProposal.order__c);

Order acceptedOrder = new Order();

acceptedOrder.Id = acceptedOrderProposal.order__c;

acceptedOrder.proposal__c = acceptedOrderProposal.Id;

acceptedOrder.manufacturer__c = acceptedOrderProposal.manufacturer__c;

acceptedOrder.manufacturer_selected_date__c = System.now();

acceptedOrder.Status = 'MANUFACTURER SELECTED';

acceptedOrders.add(acceptedOrder);

}

update acceptedOrders;

List<Order> cancelledOrders =

[ SELECT Id, Status

FROM Order

WHERE EffectiveDate < TODAY

AND Status = 'AWAITING PROPOSALS'

AND Id NOT IN (SELECT order__c

FROM order_proposal__c) ];

for (Order cancelledOrder : cancelledOrders) {

cancelledOrder.Status = 'CANCELLED NO PROPOSALS';

}

update cancelledOrders;

}

}

We won’t be talking much about language specifics, you can learn more here, but a few comments to point out the nice things about Apex are:

• it looks like Java/C#
• it has an SQL-like query builder with intellisense
• it offers ORM-like methods for CRUD operations on objects

Now that we have the class with the job, let’s schedule it. Again, Salesforce is bundled with a scheduler which makes triggering super easy. Visit Setup -> Environment -> Jobs -> Scheduled Jobs and with a click of the button, you can set/schedule the job to be triggered every day.

Salesforce also gives you a standalone and easy to use testing environment. Here are 2 features which help us verify our process seamlessly:

• Sandbox — as you might’ve noticed (top panel in every screenshot), each action we have taken in the course of this tutorial is done in a sandbox environment. This helps you safely test the changes before they end up in production. Modifications are grouped in changesets which you can deploy if you’re 100% sure.
• Developer console — you can programmatically access all objects and classes through an embedded console. Plus, you can see all debug logs in there too.

Now, we can deactivate the Order Start Date rule and add some test orders manually:

1. Add a few Accounts (Hardware Designers and Manufacturers)
2. Add Orders with the yesterday Order Start Date and change their status to AWAITING PROPOSALS
3. Simulate that Manufacturers submit proposals by creating assigning OrderProposals to Orders
4. Then, go to Developer Console / Debug / Open Execute Anonymous Window and type:

new ManufacturerSelectJob ().execute(null);

Our code should do the job then — the Orders will have the Manufacturer with the best price assigned and the status will change to MANUFACTURER SELECTED. In case there were no proposals, the status will be set to CANCELLED NO PROPOSALS.

If we have basic business objects set up, we should finally transfer the current spreadsheet-based order database to Salesforce.

By the way, how about treating Salesforce as a master of data. No external database? That’s right, storage flexibility, performance, and backup capabilities at Salesforce are good enough to set it up as the only data storage in the early stage of the project (our experience shows this setup might be enough for an online marketplace for 2 years). But what then? Worry not, in the next post we’ll show you how to get the data out with Heroku Connect when the scale hits.

Let’s get back to migration. The Salesforce Business App Store offers a handful of reliable tools which help you design, manage, and migrate data. Take Data Loader for example.

This simple tool gives you the ability to easily import data into Salesforce. It has an intuitive field mapping tool and gives a list of well-explained errors when the import cannot be done. With Data Loader, you can redesign your database in no time at all. Just add new fields to the object, export records to xlsx, munge the data so they match the new schema, and import them back.

Yep, in the early stages, when the number of records is at max dozens of thousands, you can do it all in Excel. We found it to be a huge productivity boost when the schema changes frequently.

For brevity, we won’t showcase how to transfer the Excel with orders via Data Loader, but you can see for yourself how simple it is with this video.

Additionally, SF gives you record history (audit log) out of the box. You can see who changed what and what was the previous value. These features (custom fields with validation, easy migrations, and backup with audit logs) ensure your data integrity without investing time in countless schema modifications and mundane migrations.

This all means, you really can treat Salesforce as core data storage for an early stage, yet-to-be-predictable business.

Recap

Using mostly the out of the box capabilities of Salesforce, we configured the fundamental processes related to the order management in our online marketplace. The orders.xlsx has been successfully migrated and now the operations people can manage data in a streamlined way. All fields are validated and monitored, so you can always ensure data integrity.

We have achieved this without implementing or using any other external system or applications — even getting the external database wasn’t necessary. Within a couple of hours, we have responded to the real business needs.

Now the marketing/sales/ops/customer service people save tons of time and invest their time in creative tasks. They also have an opportunity to use the collected data to improve their processes even more.

And finally, all of the data model changes and consecutive data migrations are super easy to handle. Adding a new field or creating a validation rule is a 1-minute no-coding job. In other words, the system is open for change and scale — 2 things we’ll cover in the next articles.

Coming next

We have a basic setup of entities and workflows. But what if we want to model more complex scenarios? Let’s imagine we want to update the policy for selecting the winner of the tender. What if we want to automate some mundane tasks such as sending out email confirmations? This is where Salesforce really shines. In the next post, we’re going to showcase Apex and API’s awesome capabilities, best practices, and caveats. And what’s more important, we’re going to cover Salesforce licensing, cost estimates, and support possibilities so you can learn how to communicate all this with management.

Lastly, let’s update our architecture diagram. Salesforce has emerged as a major SaaS platform. In the next article, we’ll connect it to the other parts of the system.

Originally published at www.voucherify.io.