The 2018 State of DevOps Report from DORA was released last week. This is the first one I’ve read all the way through (failure on my part!), as opposed to just skimming the highlights.. and there are some very interesting things to be found inside.
Being better at shipping software is being better at business.
This feels true in the filter bubble of startups and Silicon Valley. But going back to the report of 2014, the data shows that high performers at shipping code have at least 50% better odds at beating their one business goals, like rev targets, than low performers.
Our analysis shows that elite performers are 1.53 times more likely to meet or exceed their goals for organizational performance, and high performers are 1.38 times more likely to meet or exceed their goals. — 2018
This year we looked at both financial and non-financial measures of organizational performance. We found that high performers were twice as likely to achieve their own reported goals across both financial and non-financial measures. — 2017
In the 2014 DevOps survey, just over 1,000 respondents voluntarily identified the companies they worked for. Of these, 355 were publicly traded. We analyzed three-year results for these companies and found that they all outperformed the S&P 500 over the same three-year period. Of this group, those with high-performing IT teams, as identified by our analysis, had 50 percent higher market capitalization over three years than the public companies with lower-performing IT teams. The companies with high-performing IT teams were also twice as likely to have exceeded their own goals for profitability, market share and productivity. — 2016
This year, we hoped to validate this preliminary finding, but our sample size of stock ticker symbols was too small to conduct a meaningful analysis. However, we did find that this year’s high performers were 1.5 times more likely than their peers to exceed their organization’s’ profitability, market share and productivity goals, compared to last year’s high performers, which were 1.9 times more likely to exceed goals. — 2015
High performers deploy faster
("Elite” performers vs low performers)
What counts as “low” performance is getting faster
In 2016 low performers deploying every one to six months.
In 2017 they were deploying at least once per month.
In 2018 some cohort got fast but shy of on-demand fast, while the low performance bar was stable.
What do you call teams that today are slower than low?
The more/faster you ship, the less you go down
In the 2018 report, better availability is explicitly correlated with high performance. And high performance teams are those that deploy faster. Those same teams also recover faster and have smaller lead times from dev to prod.
Analysis showed the availability measures are significantly correlated with software delivery performance profiles, and elite and high performers consistently reported superior availability, with elite performers being 3.55 times more likely to have strong availability practices.
Doing less toil is good for business
High performers do significantly less manual work across all vectors, spend more time doing new work
This is toil. Reading from the book of Site Reliability Engineering [Google]:
Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows
High performers uniformly minimize or outright eliminate toil. And remember, high performers also are twice as likely to outperform their against their own business goals (revenue, profitability, etc).
I’m amongst the first to be skeptical about anything coming out of FANG [etc] companies being applicable to anyone else. But this is a clear case where it’s not just true for Google; it is also true for all of us.
Focus on quality early leads to better performance
Because [high performers] build quality in, they spend less time fixing problems downstream, freeing up more time to do value-add work.
This one is a truism. And yet, it’s always sacrificed to the gods of false promises your boss made to their boss made to their boss made to…investors. Break the cycle.
What does focusing on quality mean?
- Write observable code
- Build to scale one order of magnitude beyond where you are today
- Remove toil
- Reserve capacity buffers for bounded resources
- Spend time finding out what resources your code is bound on
- Reserve time for maintenance, paying down debts, refactoring, bugfixing
Overhead is Real
At the “elite” level the best case scenario for doing “new work”, aka creating business value, is 50% of the time. This means that the highest performers have 50% overhead. FIFTY PERCENT.
Considering the anecdata I hear from many engineering leaders, we’re all lying to ourselves and each other.
The Maddening Middle
Something super interesting happens between the low and high performers. We get stuck.
Medium performers have the highest amount of manual work on all dimensions.
This same pattern happens in startup go to market growth, hype cycles, or becoming an expert in any field of endeavor.
- Low hanging fruit are picked
- A lot of quick and dirty, but effective work, is executed
- We start running out of easy wins
- We start feeling the weight and inertia of the debt, technical and otherwise, from the early quick and dirty decisions
- The real work begins — now we have to make incremental and deeply unsatisfying but necessary progress on paying down technical debt, adding process, automating toil, figuring out what does and does not scale, generalizing just enough to get beyond early adopters, etc
In my experience, very few organizations want to do that kind of work or create the culture and incentives necessary for it. It’s not sexy. It’s not motivating. There aren’t quick wins anymore. It’s mostly manual labor.
This, to me, is the most interesting point from the report precisely because it’s such a common pattern that crosses disciplines. The only answer is: do the effing work.
Futher Questions
- How many (what percentage) of the low performers from 2014 (and every subsequent year) are now mid or high performers?
- How many (what percentage) of the low performers in 2014 (and every subsequent year) are new (as in they were not counted before or were beyond low or moved from the low end of the low to nearer to mid but not quite)?
- How many (what percentage) of mid performers in 2014 (and every subsequent year) made the transition to high in following years?
- What is the generalized rate of evolution low -> mid -> high -> elite?