How to Scale Global Infrastructure Teams

The customer support jigsaw is made up of 24/7 infrastructure and services support. This article focus is on the former, discussing the challenges and solutions to maintain performance as teams scale. Scaling your infrastructure/pipeline teams globally occurs around the point that sufficient customer traction has been gained. The need for 24/7 coverage for many "AlwaysUp" products often occurs before global sales have materialised. 

Early Stage

In the case of a product that has a single infrastructure team handling their environments and delivery pipeline, the intuitive starting position to support global users and infrastructure, is to add 24/7 on-call to your existing co-located team.

Initially, this works well until the team reaches a certain out-of-hours call frequency and the team gets weary. A large local team can maintain this support over a long period of time but individuals will eventually fall foul of Herzberg's dissatisfier hygiene factors, impacting negatively on team morale.

Below I provide insight on how to put your organisation on the path to easily support global sales by thinking ahead and side-stepping common scaling challenges.

Next Stage - multiple geographical located teams

A second infrastructure team is set up in order to globally scale an organisation in this situation to its next geographical sales target market.  Starting from a single co-located team doing everything, add a smaller secondary team based out of new regional location that incorporates a time zone overlap so that both communication and handover can occur.  Multinationals regularly choose geographically distributed locations based out of the US, Canada, Europe or New Zealand. 

Four Productivity Challenges

1. Mindset Challenge - Treat your locations as equal partners

The evolution towards globally available teams, may begin by treating both locations as one big team with a sixteen-hour workday spread across two sites. This will result in the second site not being as effective, due to the increased overheads involved. It's important, as the second team grows in members to intentionally shape this big team thinking and create a shared culture of one strongly bonded team.

At the same time, this allows each location to operate as semi-autonomous cooperating partner teams, capable of taking the initiative on defined areas. Adopting the mindset of two eight-hour days, rather than one sixteen-hour day is really critical to success.

2. Culture Challenge - Getting the Culture Right First Time

Actively create a team super culture that acknowledges and incorporates site cultures, cultivates strong team branding by celebrating the success of individual contributions, while emphazising the project success as belonging to the whole team. Hold x-team hackathons, workshops, design competitions for team-specific merchandising logos, team colours and use on the team's t-shirts, cups, jackets, etc.

3 Communication Challenge - Informed, Consulted and Involved

Cross-team communication needs to be a priority, coupled with strong agile practices. It's important to bring some or all of the team members physically together on a scheduled basis. Face-to-face time creates a stronger connection and avoids miscommunication from ever arising in the first place.

Every location needs to host their own local project planning, technical refinement, and Show & Tell ceremonies (aim for 10 minute recordings and centrally published), follow the same formatting, coding standards, testing practices and continuously hold scheduled team and individual bonding sessions to establish and maintain a trust relationship.

A strong trust relationship needs to be established at both management and engineering levels. Some agile ceremonies need to be duplicated, while others need to be shared. It's really important for the shared ceremonies that everyone does pre-work so as not to waste people's time in meetings.

The daily scrum of scrums, project prioritisation, and retrospectives are shared ceremonies. Reactive non-production support issues need to be handled close to the team raising it, but some less urgent issues should be included in daily hand-offs. Expect and accept it takes time to build strong trust, actively work to build and maintain high trust levels.

Each location should provide daily hand-offs message posted in the team IM channel and should have a consistent format and may include

1. Urgent tasks
2. Outstanding tasks raised through email/Instant messenger/ walk-ups
3. Outstanding issue tracker tickets

Distributed teams (outside of timezones intersections) need to communicate asynchronously between themselves and with the developer teams e.g. shared instant messenger channels. Email is best for out of hours communications. Co-editable documents should be utilised for inputs and wikis for processes. Project specific documentation should be kept alongside the code repositories as a general rule.

4. Team Topologies Challenge

Teams will often have their specialisations. The team co-located with development teams will have more focus on front-end and reactive support work, whereas remote teams need to focus on autonomous backend systems projects that require less time-sensitive interactions with development teams. Each site should have their own local manager, they understand the environment, local culture, and legalities. In the absent of a local manager, you have a satellite team whom will struggle to innovate or reach any level of self-determination.

Each location needs to be allowed to create their own gravity.  This can be accomplished by setting up semi-autonomous projects (projects not services) for each location.  Rather than both locations working on everything, have different sites deliver phases of projects, co-located team members can code review for each other.  Projects should be intentionally planned so that specific phases can be switched between sites in order to create a true sense of group ownership and accountability and the ability to support the end systems.  This allows a full partnership to emerge and avoids the emergence of the competitive mindset.  Project developer experience is really important to cultivate to ensure success.

Conclusion

To achieve a very high-performing globally distributed infrastructure and pipeline function, it is essential from the beginning to proactively embed collaborative and cooperation mindsets, create firm agile practices, utilise distributed communication technologies and instill a strong team branding.