Agile Is Not Enough: Empowered On Innovation

"If you think that by moving to Agile you've already done some form of digital transformation, I am sorry to tell you, but you haven't even gotten started.” - Marty Cagan and Chris Jones, Empowered (Silicon Valley Product Group) (p. 4).

Marty Cagan’s claim that 'Agile is not enough to achieve innovation at scale' might come as a surprise to both the Agile community and the executives investing heavily in Agile transformation programs. His claim is grounded in an analysis of what he claims are the key elements allowing top tech companies to innovate at scale (he especially emphasizes Google, Amazon, Apple, and Netflix).

So what is needed for innovation at a scale that goes beyond Agile as Cagan sees it?

Empowered on Innovation

Superficially, the key insight of Empowered can seem a lot like it could be from the Agile manifesto:

“In your journey to empowered teams, if I had to pick just one concept from this entire book that I hope you would take to heart, it would be the idea of an empowered engineer… Empowerment of an engineer means that you provide the engineers with the problem to solve and the strategic context, and they are able to leverage technology to figure out the best solution to the problem.” Marty Cagan and Chris Jones, Empowered (p.390)

However, the meaning is quite different from how this looks. It is not saying to just remove red tape and let engineers work out what to do. Many people think Agile means less management. Cagan argues that empowerment needs better management. Specifically, this means:

• Setting strategic direction based on insights.
• Ensuring structures are in place to keep gathering and assimilating product insights.
• Organizing teams so that each is empowered to pursue key problems.
• Staffing and skilling teams for an empowered approach (including coaching).

Some elements of this are not in the Agile manifesto, at least not in the specific way that Cagan intends them. A key difference between Cagan’s aims and the typical focus in Agile circles is that Cagan is explicitly not talking about the delivery of projects. He has looked at the best tech products and how the best tech product organizations can keep improving and innovating on those products. Central to what he finds is ongoing product discovery.

Product Discovery

“[W]ith software, even though we all know it is very hard, and we know that the majority of software releases fail to meet their objectives, we still insist on scheduling the discovery phase like we’re planning the construction of a house.” Marty Cagan

Continuous product discovery means doing continuous research on customer problems conducted by the team. Talking to users through focus groups and customer councils. Analyzing survey results, understanding feedback from prospective customers. Putting prototypes in front of the key users to find out which approach might solve their problem.

The key idea with product discovery is to avoid building the wrong thing. To get there, we need to really know what problems our users have and how our proposed solution will solve them. Somehow we need to know this in advance of building anything as once we’ve built something, then we’ve already spent a lot of engineering effort on something that may not provide any value.

Ideally, we can reduce our risk with prototypes. Not fully featured prototypes that require multiple engineers to build. Simple low-fidelity designs that can be put together quickly by a designer. If we put multiple options in front of users, then they can tell us the strengths and weaknesses of each (whereas if we just offer one then they may not scrutinize it enough to see its weaknesses).

To get to the point of putting good prototypes in front of users, we need to understand what their key problems are and the value they get from our product. We can get that from looking at data like product usage analytics, customer feedback, surveys, and competitor analysis. We should also be looking at emerging technology trends and be looking for new opportunities to provide value.

Product discovery in the empowered team model involves the whole team but is the central responsibility of the product manager, designer, and team lead. It’s important that the team has enough context to make informed decisions. The discovery work itself can be spread around in different ways to suit the context so long as there is enough collaboration and sharing for everyone to understand the core needs and how the proposed solutions really do meet them.

Note that this is not typically just part of how Agile is practiced. The scrum product owner role is normally regarded as more of a prioritization role and ongoing discovery is rarely considered an explicit part of Agile practice.

Product discovery can lead to lots of insights that can help to reframe what is most important for the product. It also needs to be directed towards what we currently think are the most important problems. There needs to be strategic thinking with active participation from management beyond the team implementing the software. Let’s understand how Cagan sees this direction-setting.

Direction-Setting (Vision and Strategy)

“Good teams have a compelling product vision that they pursue with a missionary-like passion. Bad teams are mercenaries.” Marty Cagan, Inspired (2nd ed.), p.311

The product vision serves multiple purposes. It aligns stakeholders across the company. It provides clarity to the product teams. It acts as a recruitment tool to attract talent who are inspired by the product concept. It can help formulate the value narratives for customers.

A product vision does not come out of anywhere. It is based on research and is a response to the current market conditions. As those conditions change, then the vision will change also but it is typically to be aimed at a time horizon of a few years.

Typically product visions are formulated as a few sentences that succinctly describe the product and its aims. It is complemented with a set of product principles, which go into more detail on the nature of the product, who it is targeted at, and what is most important about it. Empowered recommends going beyond written vision statements in using vision types. These are videos that capture what you hope the product to be in a compelling way. The svpg.com website (maintained by the Empowered authors) gives examples.

The strategy addresses the shorter time horizons of less than a year. It is about focusing on particular problem areas that you think are of special importance to growing your product. This needs to be a focused set of initiatives in the region of five. Too many initiatives and none will get enough focus. Trade-offs have to be made on which areas to focus on for this phase of the product’s growth.

The strategy should be grounded on insights. You should be studying competitors and technology trends. You should be analyzing data from customers and from market research. You need to assimilate this and make informed guesses about where the most value can be found. Those guesses are problems for teams to work on. Let’s understand how Cagan sees problem-setting.

Setting Problems to Solve (OKRs)

“Conventional roadmaps are all about output. Strong teams know it's not only about implementing a solution. They must ensure that solution solves the underlying problem. It's about business results.” Marty Cagan, Inspired (2nd ed.), (p. 24)

OKRs are now pretty popular, and it is not surprising for Empowered to recommend them. Many companies think OKRs are pretty easy. On paper, it looks straightforward enough to set an objective like “dramatically reduce the time it takes for a new customer to go live" and a key result like “average onboarding time less than 3 hours.” But getting value from OKRs is not straightforward for companies not doing ongoing product discovery.

If you’re not practising ongoing product discovery, you likely won’t have the insights needed to frame good OKRs. You won’t know what customers are struggling with (either within your product or in its general space) or how those problems can be framed as opportunities. You likely won’t have the kind of data you’d need to be able to focus on outcomes. Then the key results just end up being the delivery of features. But that’s a roadmap, and that defeats the point. That’s effectively ‘big design upfront’ by the back door.

“Typical roadmaps are the root cause of most waste and failed efforts in product organizations.” Marty Cagan, Inspired (2nd. ed.), (p. 108)

OKRs are meant to be strategic. They capture problem areas where we’ll focus our energy to create value, whereas roadmaps are more like bucket lists. That does not mean we should do both. Empowered suggests using OKRs for quarterly planning instead of roadmaps.

Note that OKRs should not be something that management produces and then cascades down. Instead, OKRs should be a negotiation. Top-level goals are set, and then teams and management work together to see which teams can cover which objectives and what it’s realistic for the teams to do in a quarter.

Note how different this is from how Agile is often practiced. This is not assigning stories from a backlog. This is identifying problem areas and empowering teams to determine the best solutions to those problems. This requires a range of skills on the teams (as well as strategic context and the right leadership practices).

Team and Individual Growth

“Your company depends on successful products. And successful products come from strong product teams. Coaching is what turns ordinary people into extraordinary product teams.” Marty Cagan and Chris Jones. Empowered, (p. 32)

Innovative product work does not require genius, but it does require being comfortable with ambiguity and embracing a collaborative way of working. Teams need to switch between a research-oriented way of working in discovery and an execution-oriented way of working in delivery. Teams also need to be comfortable drilling into the real problems that stakeholders have, rather than just acquiescing to the solutions that the stakeholders think that they want.

It is hard to hire people who have already worked in an empowered product team setting because most companies tend to work in a more siloed way than this. You can make your company more attractive by showing how compelling your mission is. You can make your hiring more personal by having hiring managers make the approaches themselves. But the talent pool with this kind of experience is still limited.

“[T]he best product companies hire competent people of character, and then coach and develop them into members of extraordinary teams.” Marty Cagan, Empowered, (p. 141)

Coaching is a key element of how companies using the Empowered Team model operate. Managers take responsibility for growing the members of their team through regular one-to-ones and personalized on-the-job training. 

The kind of coaching Cagan is talking about is not just about growing technical skills or people skills. It is about showing how to practice empowered working. Engineers, for example, might need coaching on being comfortable with ambiguity and tackling problems with a product mindset. Designers might need coaching to collaborate with engineers to determine what is possible. The empowered approach of actively seeking feedback and customer insights is likely to be unfamiliar to most and can require coaching on how to grow the necessary processes and relationships.

In this model, managers are responsible for coaching their teams, and their managers are responsible for coaching the managers. Empowered emphasizes growing the careers of reports as the first responsibility of managers. Managers also have a responsibility to ensure the teams' structures are suitable for empowered working, the team’s place in the product strategy and goals makes sense, and obstacles are coming to light and being tackled.

So Where Is Agile Missing?

“The best product teams I know have already moved past how most teams practice these methods—leveraging the core principles of Lean and Agile, but raising the bar on what they're trying to achieve and how they work.” Marty Cagan, Inspired (2nd ed.), (p. 24)

That Agile needs the right kind of setting to flourish is perhaps not new. Agile surveys regularly find teams reporting organizational resistance and clashes between Agile practices and company culture. What is new is Cagan’s diagnosis of the specifics of what most companies are missing and how the most innovative companies are operating differently. Let’s understand this in more detail.

Cagan on Scrum

Cagan refers to the Scrum Product Owner role as “essentially a backlog administrator” (Inspired, p. 41) or ‘feature jockey’. His criticism of many companies practicing scrum is that they are not adequately doing product discovery, vision-setting, strategy, or setting the team problems to solve. Instead, they are balancing stakeholder interests and essentially putting features into a prioritized backlog as stories.

The way Cagan phrases the criticism comes across as a criticism of Scrum as well as the companies practicing it. It is natural that companies have not seen the need to add discovery and strategy to Scrum because Scrum does not clarify that there's a need for these things.  

Cagan on SAFe®

Here’s a snippet of Cagan’s criticism of SAFe (Scaled Agile Framework):

“In SAFe, this concept of a true product team has been undermined and demoted, and the core concept is now a Program, which has a top-down model of a product manager, an architect, and a release train manager, and these people make all the key decisions, and then some number of engineering teams with a low-level product owner are assigned various parts to build.” svpg.com, accessed 29/06/21

A similar criticism is made in Empowered (p.12):

“A delivery team doesn't even pretend to be a true product team. They are not cross‐functional, and they are not empowered. There is a product owner (responsible for administering the product backlog) and some number of engineers. They are purely about output (code and ship). If you're running a process like SAFe, then this is unfortunately you, and truthfully, I have no idea why you would want to read this book, since what I describe here is the polar opposite both philosophically and practically.”

So now we see the key issue for Cagan. What’s missing from both SAFe and Scrum is:

• Insofar as product discovery figures at all, it is separate from the engineering teams.
• Empowered teams should be given goals as problems, not features from a backlog.
• The organization needs to be set up to work like this, which especially includes leadership, staffing, team structures, and career growth.

Running Tech Product Organisations, Not Just Software Delivery

Now we understand how Cagan’s analysis leads to the conclusion that Agile is not enough for product innovation. Let’s come back to that quote from Empowered that we started with, which claimed that empowered engineers are the key concept of the book:

“Empowerment of an engineer means that you provide the engineers with the problem to solve and the strategic context, and they are able to leverage technology to figure out the best solution to the problem.” Marty Cagan and Chris Jones, Empowered (p.390)

We can now see that this is not just about giving engineers autonomy. It is about structuring work around problems rather than features, gathering insights and framing a strategy, and structuring the organization so that teams can best tackle the key problems.

Nothing in this is against the spirit of the Agile manifesto. But the manifesto comes across as pitched at a team level. If organizations need to change to support the best application of Agile, then the manifesto in itself does not tell us what organizational change is needed. 

Surveys have long been finding a missing ‘organizational culture’ ingredient to be problematic for agile adoption. Empowered and its analysis of what works for tech’s top companies offers one way of fleshing out this missing ingredient.

Title image by waferboard, CC BY 2.0, via Wikimedia Commons.