Kaspar Lyngsie

A socially calibrated guy with uncompromising passion for IT.

Ownership and Responsibility in Software Development Teams

This is part 1 of a 3-part series on things I've learned working in an incubator together with a high-performing, truly agile software development team.
If I was to make a top 20 over the most talented software developers I've ever worked with, most of the people on that list would all be engineers originating from the same project I worked on recently. The project was a startup-backed-by-big-business endeavour, which nowadays commonly shares the sobriquet "incubator", and has seen many variations of implementation across the years in the digital market. The business managed to recruit some of the best and brightest in the relatively small talent pool available in the country (as well as an even larger talent pool recruited internationally, but this piece will focus on the those whom I worked with locally).
For these engineers, on average, the churn rate was just over a year. After approximately 12 to 14 months, they left the company again. After I had been involved in the project for about 2 years, there was none of them left. Why?

Define incubator

Detaching an innovation department from a large, established corporation is backed by the hypothesis that large, complex and traditionally managed corporations does not handle innovation well. Decisions are made top-down by the people least in touch with both product and customer, employees have fixed roles with defined career paths and new projects are commanded and controlled through a series of decisions designed primarily to avoid accountability.
This constellation continually dilutes and inhibits any and all attempts at innovation aimed at market disruption or challenge of the status quo. Despite the buzz, the idea isn't actually that new. A lot's been written about it, see for instance this article. The book Adhocism was first released in 1972, containing many of the notions and ideas utilised by incubators today.

The grown-ups will take it from here

During my employment, I observed a significant change in the applied management, which had a decisive factor in the engagement and dedication of the employees. At the incubator's inception, the coherence between management's words and actions was 1:1. As the projects and culture spawned by the incubator started to mature, a top-down decision was made by the traditional management to, as they called it, "graduate" the incubator projects into first-class citizens of the corporation.
Spoiler alert: That's when resignation rates started increasing.
Even if I had not spoken to anyone about their decisions, their actions usually spoke louder than words. Any HR or management responsible does well in trying to listen to the silent delta between old job and new. Some colleagues even left without having found a new job at all. That really ought to be a wake-up call if I ever knew one.
I did, however, inquire. All resignations shared (at least) one common pain as a reason for seeking new opportunities: Where they before had enjoyed ownership, they now suffered responsibility. Out of the fleeing talent pool previously mentioned, five of the engineers went to become CTOs in other incubators like the one they had just left, four or five others were poached by said newly-elected CTOs, three joined other incubator-startups as senior developers and finally two decided to become independent contractors. When asked what they gained in their new positions, the answers were the same: Ownership.
"I like to make my own decisions", one of them told me, when I asked him why he had resigned, after he was poached by an ex-colleague-turned-CTO, "while going 120 mph". What he meant was, that he liked to be trusted to take charge, while allowed a minimal time-to-market. He was owning the development, deployment process and quality assurance. But how can you be in charge if you are just a simple software engineer? Surely the CTO or the technical management is the one in charge?
Right? Wrong! Let me try and explain why, after these messages:
The entire original management of the incubator was in this process of "graduation" completely replaced. I will in this article series refer to the new management style observed as simply The Career Manager™. This is the well-known type of classical manager who are not interested in product growth, but career growth. I'm sure you all know at least one. Maybe you are one yourself? In that case I really hope you will continue the read.

Empowerment means ownership, not responsibility

Most current How-To guides on managing software teams always emphasises the same notion: Empowerment. But what does that really mean?
Empowerment is a means to include the team in decision making, to give them a participatory role which capitalizes on their own expertise and judgment, and that increases their sense of both individual worth and commitment to the organization.
Career Managers™, on the other hand, interpret this to mean blessing their employees with responsibility: "Congratulations, we've decided to make you team lead. That's a lot of responsibility!"
But it's not, really. Empowering your employees should mean ownership. So what's the difference?

Responsibility

Being "granted responsibility" is in fact an oxymoron. You're not granted anything, you take it, just like everybody else on your team has to, as well as in the management group and on the penthouse-floor offices of all the people wearing suits and sporting C-level calling cards in bone off-white color pallettes. The engineering talents we see make the most of their careers are the ones who do not see responsibility and leadership as a job title, but in the behaviour and conduct of their peers.
If you are developing a software solution, it's just as much your responsibility that this code actually runs in production, as it is everybody else's on the team. We've long since evolved past the silo-mentality of "Not My Code, Not My Problem Problem" across high-performing development teams. Countless research and articles have been published on the matter, so I won't bore you with yet another.
I've worked in many high-performing teams that make use of firefighters to handle production bugs, to avoid constantly causing context-shifts in the minds of other developers on the team. Obviously, if that is the way you like to do it, responsibility is impossible to monopolise given the very nature of this constellation, and that's the way it should be.
For Career Managers™ to empower through responsibility effectively means nothing more than assigning blame and accountability, in order for themselves to be held unaccountable. I mean, that makes sense, right? If I can defer responsibility to someone else, that must mean I myself is unaccountable, right? For example, does this sound familiar?
There's a bug in production: Who was QA-responsible during the release? Who in the development team introduced the bug? Who's to blame, so we can take away that team's responsibility in the future?
...we must create a new, dedicated deployment team to be responsible for deployment to production in the future. This cannot be trusted to the development teams, as they cannot coordinate internally themselves. This incident is indisputable proof of this.
Giving and taking responsibility has zero benefit or meaning in any modern development organisation, and it is a byproduct of misunderstood release management, which I will cover in the next part of this series. For now, just trust me on it.

Ownership

Ownership means taking the initiative to solve a task in the way you, as a team, believe is best. Best for the end-users of the product, and for the the quality and maintainability of the code for your other team members. That can be a nerve-wrecking thing to do, as it can lead to cataclysmic disasters for a product, if done wrong.
If you are working in an environment that works entirely around assigning and deferring blame, taking ownership or initiative to challenge existing ways of working is not something you're very likely to do. In fact, like I witnessed personally, it lead people to actively work against such behaviour in order to avoid bringing upon themselves unwanted scrutiny from upper management.
That is why responsibility and ownership is a team-effort. Empowering a team with ownership means the team internally share a sense responsibility of both the product and the code. In that way, that team will be able to mitigate potential disasters before they occur, and when some unavoidably do occur anyway, will be able to mitigate production errors quickly, because they themselves have ownership of the deployment pipeline. That's the original meaning of DevOps, by the way, before that got a bit diluted, just like Fake Agile.

Examples

Like everybody else who's ever read a man page, examples are the first thing you skip to. So I will end this article with two real-world scenarios displaying the difference.

Scenario 1: Full responsibility, zero ownership

Imagine you are working in a 5-6 man software development team, and you are tasked with some assignment of creating a distributed backend that needs to scale. The team decides to go with all the wonders that cloud-providers has to offer, allowing them to focus on code instead of infrastructure. They utilise some cloud provider MQ for asynchronous messaging, some auto-scaled load balancer for optimal response times and failover, a sharded object-store as backend and a distributed logging mechanism for easy debugging and finally some metric alarms to quickly be alerted of production errors.
For programming language, the entire team has for a while been following the Rust-spectale happening lately, and thought this would be a good opportunity to collectively grow in competency while learning new technology which could improve security of the platform - they know the management love to hear that.
Before getting very far, The Career Manager™ interrupts them, and demands that a thorough report on their decisions are created, minutely pointing out what (technical) decisions have been made, and why. The Career Manager™ needs approval before the team can move on. Rumours have spread around the establishment suggesting the team is about to embark on a Wild West mission using unapproved software, technologies and programming languages. It would be a plus if the team includes drawings and figures in their report. Microsoft PowerPoint, of course. None of the management uses OSX or Linux.
After the team has spent two days writing and validating the report internally, and waited a response for another week, the team is called to a meeting and informed that management has a few concerns. They are still empowered to do it the way they want. But, as good management dictates, they are required to operate inside certain boundaries to avoid them spinning out of control. The management has to think about the rest of the corporation too.
  1. Cloud is dangerous and insecure. They can use the on-premise solution that has been home-brewed internally by the IT department.
  2. The database can only be Microsoft SQL Server (no sharding, single source), as this is the only enterprise solution they have external support for, in case something goes wrong (remember, assigning blame and responsibility is pinnacle).
  3. There is no off-the-shelf solution for an MQ that they know of or support, so the team will have to come up with an implementation without this functionality.
  4. Automatic load balancing and fall-back solutions, de-facto standard in every cloud solution for at least 6 years, is not supported on-premise, so the solution will just hold a fixed number of servers all the time. If the ops department notice any performance issues, they will provision more servers. The team just has to tell the ops how many servers they want, no big deal.
  5. Automatic metrics and alerts on errors are not available due to firewall problems in the complex organisation with a tight security team. Production logs can only be viewed from a special Windows machine with special credentials (the team use whatever operating system they prefer, primarily Linux and OSX, so they will need some new machines). Production errors will be detected by a dedicated QA team on the day of deployment, or by complaints by end-users experiencing them.
  6. No Rust. The language is unknown to the IT management, and therefore all arguments made for its use irrelevant as the risk for the company is too big: Consider the possibility, that the entire 6-man team suddenly decides to quit all at once. There would be nobody to maintain the code base, as learning a new programming language is not something a senior software developer just does out of the blue. Official company policy is to only support C#, since that was Microsoft-backed and again, if something goes wrong, you can always bring in external Microsoft consultants to put out the fire. (While blame and I Told You So™ is assigned to the team).
This is an example of total accountability on behalf of the team, while zero trust is put to the team's own effort to solve the task in the best way they see fit. This is an optimal solution for The Career Manager™, as this leaves his path completely open to promotion while being kept absolute unaccountable for any and all catastrophes that could potential occur. To the development team, the situation is utterly demotivating, as they receive the full force of the blame if the implementation suffers, while receiving zero credit if it succeeds, as it was The Career Manager™ who saved the project from disaster by keeping these cowboys on tight leashes.

Scenario 2: Shared responsibility, shared ownership

Imagine the exact same scenario, only this time, the rumours have not caused a spike in panic from management, but a spike of interest
Management knows that they have hired talented developers, so they also recognise their own inadequacy to know better than their own software development team. The manager's responsibility is to show the team trust and clear the path of obstacles on the way from idea to implementation. As management has no interest nor knowledge about the fine details of the software implementation, that is left to the development team alone.
Since the company has no on-premise solutions that can remotely compare to the flexibility of a cloud provider, the team is empowered by ownership of the entire development pipeline, and is requested by management to dedicate a significant amount of time up-front to document this process, so the next team can benefit from the knowledge in a shared way.
  1. As no cloud provider is currently in use by the company, the team is requested to review and decide upon the most effective cloud provider for the task, both in terms of cost and integration into the existing system. Management requests, as security, GDPR and company brand value is important, that the team ensures all such regulations are enforced by the provider. Should people with key knowledge in the team suddenly decide to quit, hand-over procedures, like in any other case of resignation, should be documented and in place.
  2. Database technology can be chosen as the team sees fit. Management requests that the cloud provider ensures sensible snapshot and backup procedures in case of accidental deletions from trigger-happy junior developers, attacks and the like.
  3. MQ technology can be whatever the team sees fit. But since enterprise solutions can potentially be expensive, management asks that the team either try and bundle the MQ solution with the cloud provider's MQ (if any) or find something open-source that the team can vouch for.
  4. Automatic load balancing solutions can be whatever the team sees fit. The management has for a long time looked for a solution for automatic fallback and re-deployment in case of deployment errors, and requests they investigate this possibility here.
  5. Logs and error metrics can be whatever the team sees fit, but management requests that the team either develop a hook themselves or find a solution that can integrate the production alarms into the existing bug tracking tool used across the company.
  6. Programming language can be whatever the team agrees on internally. The management trusts that the team will not select obscure programming languages that are not suitable for the task at hand or unmaintainable in the future.
Ownership means trust in the team's own ability to act rationally, and it leaves the team with the type of responsibility that seems a lot more like approval than accountability. The amount of responsibility has not diminished, on the contrary, the team is now a lot more exposed. That is exactly why most Career Managers™ fail to understand the concept.
Consider the sense inside the mind of an engineer assigned a bug related to the database. In scenario 1, for all intents and purposes, he did not decide on the technology, he probably even decided against it but was shamelessly overruled by explanations of politics that did not make any sense. How responsible or motivated will that employee actually feel for fixing the bug? In scenario 2, it was his own decision to jump on the train of that database technology, the management trusted him to do it, and therefore gives him a lot more motivation to actually solve it, as well as a much higher sense of catharsis upon task completion.

Conclusion

Being blamed for failure is a traumatic experience for everyone, and something most people naturally wishes to avoid. I have personally caused a production error which rendered a snowball effect of undetected bugs which in turn caused a significant disruption to our services. If I had been working in a culture of zero ownership and total accountability, my natural reaction would probably been to meticulously comb `git blame` until all guilty commits were detected, and then ragingly insist on a more siloed release processes and stricter control of team freedom, since I now suffered from other people's lack of control of their own freedom.
Instead, the team felt a shared sense of ownership and responsibility, which caused us to do a collective post mortem of the incident, identify how to avoid it in the future, and then continue as we were. We never had a similar incident again.
It is a scary thing to give up control, especially for The Career Manager™. But if those type of command-and-control style corporations don't realise the importance of empowerment through ownership, not as responsibility, they will continue to see their pool of engineering talent rapidly diminish, and instead be replaced by people who just works to avoid blame.

Up next

This leaves us with a final burning question: Is the very nature of these corporate systems simply not compatible with this notion of empowerment? Does it even make sense to have these incubator-corporation constellations? I am going to argue yes, it does make sense, but it requires a revolutionary new way of thinking (this is sarcasm) from The Career Manager™ and their peers working with software developers: Acceptance of production errors and Transparency of decision-making. That's coming up in Parts 2 and 3.

Attributions

Icons used in the images are from Vectors Market (on FlatIcon, licensed by Creative Commons BY 3.0)

Tags

Topics of interest