Scaling Backend Development Teams

Scaling teams is one of the biggest challenges engineering leaders face, and backend teams, in particular, can be especially hard. In today’s distributed environment, backend teams are often responsible for applications that are used by both internal and external stakeholders, and documentation, deployment, and tech scaling challenges are all more pronounced than they are in frontend development.

Before I foundedDraft.dev, I spent eight years leading and growing software engineering teams, initially at Packback and then at the Graide Network. In this piece, I wanted to collect some of my observations about this process, as well as tips from other technical leads on expanding backend software engineering teams sustainably.

Whether you’re in the process of building your first backend team or scaling an established one, you should find some of these insights useful.

Identifying the Right Time to Scale

Knowing when to scale is half the battle. Typically, a team will begin to feel shorthanded as the software’s complexity rises and it takes longer to make changes. With a bigger team, you can split the work between smaller units that handle different parts of the project and work in tandem to improve application delivery.

While leaders are under pressure to hire more engineers to get more done, this tack can backfire. You might end up with a bigger, busier team who is actually not producing more of the results the business needs.

This is because delivery velocity does not grow linearly as you expand your team. In many cases, larger teams may move more slowly than smaller ones, or as The Mythical Man Month

points out:

It takes one woman nine months to make a baby, but you can’t make nine women make a baby in one month.

Think about how many parallel tracks you really have for your developers. Two devs can’t work on the exact same line of code (unless they’re pairing), so when you expand your team, you have to factor in the loss in productivity that comes when more people need to communicate to complete the project.

Proceed systematically when you grow and be clear on three things:

1. The changes and features you’d like to introduce
2. How quickly you want to introduce them
3. How many engineers it’s going to take to make those changes

There’s no perfect formula for doing this, but as a leader, your job is to make the best stab at it that you can.

Structuring the Backend

Typically, there are a couple of different ways that companies will try to scale backend teams.

The first way is to develop multiple APIs with a team supporting each of them.

Let’s say in the early days, a company has a single API that powers the entire app. Eventually, it will get unwieldy and hard to change, so the engineering team might introduce a second API that’s maintained by a different team. This essentially splits the functionality between the two, where team A and B are each responsible for a specified set of endpoints and everything in between is up for collaboration.

The second way to structure backend teams is to use a single large backend API, and put different teams in charge of different parts of it.

For instance, team A might be responsible for user authentication, team B for the shopping cart, team C for the catalog, and so on. They’re each going to be in charge of a group of endpoints within the same codebase.

The first approach is typically used by large companies with complex, distributed applications and teams like Google and Amazon. Small-to-mid-sized companies are usually better off with the second option because creating multiple backends adds a lot of maintenance overhead. Once you have multiple APIs, you have to have multiple continuous integration flows, multiple server environments, extra security controls, and so on.

Tips for Scaling the Right Way

Beyond having a well-thought-out structure, there are a few things engineering leaders can do to help scale their backend teams and operations. Some best practices I’ve observed include:

Build Smaller Teams

Having smaller teams reduces communication overhead while helping your employees develop expertise and ownership over their domains. The only potential drawback here is that it can concentrate knowledge into silos and require stronger cross-team collaboration. You can remedy this by regularly rotating developers between teams. This way, they can familiarize themselves with all parts of the application, giving you redundancy in case a developer suddenly becomes unavailable.

The Scrum Guide recommends between three to nine members in one team. While others tout seven as the ideal size for a good mix of experience levels and skills within the group.

Be Innovative With Team Structure

There are a few different ways to structure teams and responsibilities. One is, of course, to let them specialize in a domain. This is very useful for areas like security, infrastructure, data warehousing, and other shared services which are necessary across teams.

For your business-critical initiatives, consider a more cross-functional setup. Jimmy Paul, Co-Founder at Crafty told me, “I’ve found teams to be most successful when creating user-focused pods that consist of dedicated product management, product design, and developers. This allows the full team to truly understand who they’re building for and have empathy for the users they’re building for.”

An emphasis on product management and user benefit makes your teams self-sufficient while naturally ensuring everyone in your company is pulling together and pulling in the right direction.

Griffin Caprio, Founder at G,C & J  Group, echoes this sentiment, “Most companies struggle because they think the problems to solve are howto build something, but it’s really what to build and why X vs. Y. Every engineer has built something that was the highest priority and must-have that no one used. There’s very little accounting for opportunity cost and total cost when the PM team gets something approved.”

Document Rigorously

Keeping documentation up to date is vital so that everybody on the backend team knows how their system and the other systems they interface with work. This lets engineers help each other out and minimize errors due to misusing APIs.

Documentation is particularly useful for new hires who need to quickly familiarize themselves with the application architecture. Otherwise, they’ll have to spend time asking existing engineers questions at every turn.

According to Josh Holat, CTO, team members at __Cube__use a range of documentation to stay aligned:

“One way to help other engineers is creating READMEs within the codebase itself to outline how to get up and running with it. High-level overviews of the data model or business logic in a system like Confluence is also helpful. We make heavy use of RFCs when building larger features. This can serve as a self-documenting write-up for other engineers.”

As backend teams grow, it gets harder and harder to keep documentation up to date and maintain communication pathways between teams. It helps to use third-party tools to automate some of this. Postman, for instance, is a tool that I’ve used a lot to document and test APIs and streamline collaboration.

Test Often and Autonomously

When someone new joins your team, there are two ways to keep them on track. Option one is documentation, but even with extensive documentation, something will inevitably go wrong. A bunch of different people are likely working on the code, so there are bound to be discrepancies. The only surefire way to catch these discrepancies is through automated testing.

Every time someone makes a change to the backend and tries to push it out to the servers, you should automatically run your test suite to make sure that they didn’t break something in the process. This prevents individual developers from messing things up too badly, but also gives the whole team confidence to let other people work on the backend without sacrificing quality or code integrity.

Initially, the cost of maintaining code with manual tests or no tests is a lot lower than that of writing automated tests. However, as the diagram below illustrates, it soon becomes prohibitively expensive to scale your application without automated testing.

Hire for Long-term Fit

The big thing to think about with hiring software engineers is their existing skills and how they want to grow. For example, if your backend is written in PHP, a diehard Ruby engineer is probably not a good fit for you. You want somebody who wants to work in the language and framework that you’re using.

Also, keep an eye out for people who’ve designed similar systems in the past. If you’re an e-commerce business, you’ll want someone who understands the data model and API structure in that industry because they’ll be able to catch on and contribute much faster than someone without that domain expertise.

Beyond that, all the basics for organizational hiring such as culture fit and value-alignment matter. This is good to keep in mind especially if your teams are amalgams of developers and non-developers who need to work together. New hires will need to — should want to — communicate effectively with non-technical people.

Choose Leaders Carefully

If there’s one thing I’ve learned managing other developers, it’s that the leaders of engineering teams need to be excellent communicators who can empathize with their team. They have to be really good at managing and communicating up and down the chain of command.

This tends to be one of the biggest challenges for engineering managers becauseengineering managers are essentially a buffer between developers and business-minded upper management. Keeping both of these groups happy can be a challenge.

Promoting from within is usually easier than making external hires. This ensures new leaders are familiar with your application ecosystem and have established relationships with the other team members.

But, only do this when there are willing and viable candidates available. I’ve made the mistake of taking a great individual contributor and putting them into a management role only to lose the employee when they realized leadership wasn’t a good fit.

Keep an Eye On Productivity

When scaling from one to many teams, it’s easy to overlook the complexity this adds to your communication and deployment workflows. As you’re growing, look for ways to improve collaboration and measure velocity changes across your teams.

Caprio advocates for smaller projects and simpler technologies to facilitate this:

“Teams have a tendency to throw new technologies at problems that are actually focus and communication issues. As a result, they get mired in rewriting/reinventing/rehashing solutions to known problems in different, often immature, technologies. Bottlenecks are typically the biggest slow downs in terms of independent contributions and a lot of that comes from tribal knowledge.”

There’s a lot of merit in using simpler technologies for improved teamwork. This is a big reason why shared codebases, for instance, are making such a comeback.

Valentin Vasilyev, Co-Founder at Fingerprint explains it this way:

“We like to break down the application into services, but store the code as a monorepo to reuse code and move faster. Each team runs a service that plugs into a common event stream. Shared, cross-cutting responsibilities are maintained by the team that is considered to be the most skilled or mature, because they own parts of the monorepo that will be reused by all teams.”

Common Mistakes Teams Make When Scaling

One of the biggest problems you’ll face while scaling a backend organization is that you likely have to work in systems that weren’t designed to accommodate multiple teams. Future-proofing is nearly impossible.

For instance, if your application is designed so that the codebases for your user profiles and shopping carts are tightly-coupled, a change in one will likely impact the other. This means that every time a team changes something with the shopping cart, the team handling the user base would have to make a corresponding change to their code to prevent merge conflicts down the line.

This limits productivity and defeats the whole purpose of having multiple teams. Well-planned systems that allow teams to work independently and reduce the need for cross-collaboration are key to successful scaling.

My advice is to spend time planning for the future early in your development journey. While slightly less “agile,” I’ve found that having a long-term architectural vision can help make choices clearer and prevent corner-cutting that leads to poorly decoupled code.

Too many product leaders make the mistake of prioritizing progress in two-week sprints over long-term goals. A short-term focus can undercut your ability to grow sustainably in the long-term, and may even prove counterproductive in the short-term if engineers don’t know what they’re working towards.

Final Takeaway

At its heart, scaling a team — any team — is about matching skills to problems. Sometimes the challenge is finding engineers with skills to solve the problems you have, and other times, the challenge is figuring out what your most pressing engineering problems really are.

If you’ve found these insights useful or have some of your own you’d like to share, I’d love to hear from you. Leave a comment below or reach out to me on Twitter to continue the conversation.

Photo by Annie Spratt on Unsplash