Order out of chaos

At Qumulo, we build a highly scalable distributed file system. Qumulo’s flagship product — — is a distributed file-system. That system is predominantly written in C and is comprised of several millions lines of code. It’s a highly complex and mission critical system. Enterprise storage systems, similar to networking infrastructure, have to be available all the time and cannot malfunction. Our customers are typically Fortune 100 companies with extremely high availability and performance requirements. Qumulo Core scale-out We chose to build this system by adhering to a few tenets: We release our software every two weeks We follow an software development methodology Agile We TDD, unit test and are maniacal about quality We work in self-directed and autonomous teams It is the last point that I want to spent some time describing in a bit more detail. The Qumulo engineering team is comprised of ~40 software engineers, 3 QA engineers, 2 hardware engineers and 3 agile coaches. These individuals self-organize into Scrum teams ranging in size from 4–6. Each team is assigned a Product Owner (from the Product team) along with an Agile Coach. Within these teams there is no assigned leader. No dev manager. No tech lead. Sounds like a recipe for chaos, but it really is not. Management New teams are created in response to either new business needs (building a new feature) or, more typically, due to the increase in the size of the engineering team. Whenever a new team is formed it galvanizes around a charter or mission. That mission is driven by the feature the team is tasked to build. Feature prioritization, scoping and delivery targets are predominantly set by the PM team. Team creation: It is important to note that the team selects its mission and not the other way round. We do this to encourage agency, autonomy and purpose, which has shown time and time again, foster highly motivated and engaged employees, which in turn leads to higher productivity. research Once a new team is created, they set out to create a plan for delivering the feature they selected to implement. This would typically involve working with the PO (and potentially customers) to ensure proper scoping of the feature and likely a design process, depending on the complexity of the feature. Teams usually spend anywhere from a few days to weeks in this phase, again, depending on the complexity of the feature. One of the main artefacts of this phase is a high level estimate (HLE) of when the team expects to deliver the feature. The team is responsible for the full development of their feature, from design, implementation and testing. While incremental delivery is highly encouraged, incremental quality is highly discouraged. We strive to keep the tip of our code base at shippable level quality at all times. All our dev work is done in 2 week sprints. As I mentioned earlier, we release new versions of our software every 2 weeks. How this works, might be a good article in of itself. Every team (we have ~10) aligns to this release cadence with no exceptions. Team sprints: At the outset of every sprint, teams go through a planning phase to outline what they expect to accomplish within the sprint. Similarly, teams hold a review (anyone can attend) at the end of every sprint during which the team presents what they accomplished and evaluates it against what they sought to achieve. This cadence of plan/review allows teams to gain confidence in their HLE and ensures that they are making good progress towards their delivery goal. We encourage team movements along two axes so to speak. First, team membership can change. And second, the team charter will change, typically when a team has completed its current charter. There have been very few exceptions, whereby a team had to abandon its charter and work on a new one, which is driven by business needs. Team movement: The change in team membership is driven by the individual and is another way of fostering agency and choice. It also helps individuals try working on different parts of the product, helping them develop new skills. Surprisingly team movement isn’t as high as you’d imagine it to be; a handful per quarter. As was mentioned earlier, our teams of comprised of software/hardware engineers, an Agile Coach and a Product Owner. We have no team leads. No dev managers and no assigned lead per team. In fact, we have no hierarchy within the engineering organization. Team structure: Some of our teams have a fixed charter. These are typically for teams that we expect to need all the time. We have a few of those, which are: Fixed charter teams aka Special Teams: Performance: Makes our software zoom zoom Support: We adhere to “You break it. You fix it” mentality and as such we have a support team comprised of software engineers Engineering Infrastructure: Build system, CI and many other tools we depend to on to create our product Platforms: Evaluates new physical and virtual platforms for our software to run on as well as plethora of new hardware goodies Certification: This is a team comprised of our 3 QA engineers who work tirelessly to ensure that our software can be safely released to our customers. With the exception of the certification team, membership of the above teams is voluntary. Yes, even our support team is 100% based on individuals selecting to join that team. The support team has one requirement: minimum stay of 3 months. And we have a waiting list for that team as well. We’ve been adopting this model since our inception (~5 years ago) and it has worked out incredibly well for us. We’ve hired smart, creative individuals who want to contribute to Qumulo’s success. By letting the individuals select which group of people they work with and what projects those teams work on within the constraints of the business needs, we are able to keep individuals engaged and able to do their best work. By favoring a process that involves choice to determine work groups and assignments, we foster autonomy and ownership of results.