Order out of chaos

At Qumulo, we build a highly scalable distributed file system. Qumulo’s flagship product — Qumulo Core — is a scale-out distributed file-system. That system is predominantly written in C and is comprised of several millions lines of code. It’s a highly complex and mission critical system. Enterprise storage systems, similar to networking infrastructure, have to be available all the time and cannot malfunction. Our customers are typically Fortune 100 companies with extremely high availability and performance requirements.

We chose to build this system by adhering to a few tenets:

• We release our software every two weeks
• We follow an Agile software development methodology
• We TDD, unit test and are maniacal about quality
• We work in self-directed and autonomous teams

It is the last point that I want to spent some time describing in a bit more detail.

The Qumulo engineering team is comprised of ~40 software engineers, 3 QA engineers, 2 hardware engineers and 3 agile coaches. These individuals self-organize into Scrum teams ranging in size from 4–6. Each team is assigned a Product Owner (from the Product Management team) along with an Agile Coach. Within these teams there is no assigned leader. No dev manager. No tech lead. Sounds like a recipe for chaos, but it really is not.

Team creation: New teams are created in response to either new business needs (building a new feature) or, more typically, due to the increase in the size of the engineering team. Whenever a new team is formed it galvanizes around a charter or mission. That mission is driven by the feature the team is tasked to build. Feature prioritization, scoping and delivery targets are predominantly set by the PM team.

It is important to note that the team selects its mission and not the other way round. We do this to encourage agency, autonomy and purpose, which research has shown time and time again, foster highly motivated and engaged employees, which in turn leads to higher productivity.

Once a new team is created, they set out to create a plan for delivering the feature they selected to implement. This would typically involve working with the PO (and potentially customers) to ensure proper scoping of the feature and likely a design process, depending on the complexity of the feature. Teams usually spend anywhere from a few days to weeks in this phase, again, depending on the complexity of the feature. One of the main artefacts of this phase is a high level estimate (HLE) of when the team expects to deliver the feature.

The team is responsible for the full development of their feature, from design, implementation and testing. While incremental delivery is highly encouraged, incremental quality is highly discouraged. We strive to keep the tip of our code base at shippable level quality at all times.

Team sprints: All our dev work is done in 2 week sprints. As I mentioned earlier, we release new versions of our software every 2 weeks. How this works, might be a good article in of itself. Every team (we have ~10) aligns to this release cadence with no exceptions.

At the outset of every sprint, teams go through a planning phase to outline what they expect to accomplish within the sprint. Similarly, teams hold a review (anyone can attend) at the end of every sprint during which the team presents what they accomplished and evaluates it against what they sought to achieve. This cadence of plan/review allows teams to gain confidence in their HLE and ensures that they are making good progress towards their delivery goal.

Team movement: We encourage team movements along two axes so to speak. First, team membership can change. And second, the team charter will change, typically when a team has completed its current charter. There have been very few exceptions, whereby a team had to abandon its charter and work on a new one, which is driven by business needs.

The change in team membership is driven by the individual and is another way of fostering agency and choice. It also helps individuals try working on different parts of the product, helping them develop new skills. Surprisingly team movement isn’t as high as you’d imagine it to be; a handful per quarter.

Team structure: As was mentioned earlier, our teams of comprised of software/hardware engineers, an Agile Coach and a Product Owner. We have no team leads. No dev managers and no assigned lead per team. In fact, we have no hierarchy within the engineering organization.

Fixed charter teams aka Special Teams: Some of our teams have a fixed charter. These are typically for teams that we expect to need all the time. We have a few of those, which are:

• Performance: Makes our software zoom zoom
• Support: We adhere to “You break it. You fix it” mentality and as such we have a support team comprised of software engineers
• Engineering Infrastructure: Build system, CI and many other tools we depend to on to create our product
• Platforms: Evaluates new physical and virtual platforms for our software to run on as well as plethora of new hardware goodies
• Certification: This is a team comprised of our 3 QA engineers who work tirelessly to ensure that our software can be safely released to our customers.

With the exception of the certification team, membership of the above teams is voluntary. Yes, even our support team is 100% based on individuals selecting to join that team. The support team has one requirement: minimum stay of 3 months. And we have a waiting list for that team as well.

We’ve been adopting this model since our inception (~5 years ago) and it has worked out incredibly well for us. We’ve hired smart, creative individuals who want to contribute to Qumulo’s success. By letting the individuals select which group of people they work with and what projects those teams work on within the constraints of the business needs, we are able to keep individuals engaged and able to do their best work. By favoring a process that involves choice to determine work groups and assignments, we foster autonomy and ownership of results.