CEO & Co-founder Waydev
The Agile Data-Driven methodology was born from the need of engineering leaders to be able to measure their teams’ performance with real-time metrics and reports, aggregated directly from Git repos, by using Git Analytics tools.
The Agile Data-Driven methodology is not meant to replace the Agile and Waterfall methodologies. It’s meant to add a layer of data to the current Agile processes, bringing a real-time, quantifiable view of engineering output. By switching to the Agile Data-Driven methodology, managers and executives will be able to gain visibility into potential roadblocks, reduce time spent creating reports, and increase delivery velocity.
The Agile Data-Driven method also helps software engineers remove distractions. The importance of engineers’ workflow is highlighted by Paul Graham in his essay “Maker’s schedule, Manager’s schedule.” According to Graham, the way engineers and managers work is very different. While a manager has no problems in organizing his work around meetings, for an engineer this has a high productivity impact, as they usually end up losing half a day over a simple meeting.
He recommends taking into account the Maker’s schedule as not to interrupt the engineers’ workflow and help them achieve a higher productivity. The Agile Data-Driven methodology helps its users do that by providing automated reports for each team member and their peak productivity hours. Doing this as an engineering manager will help run non-disruptive meetings while making engineers less likely to dread the necessary meetings.
In 1985, the United States Department of Defense mentioned the Waterfall method in their standards for working with software development contractors. The Waterfall methodology’s reign was stopped short by the appearance of a new methodology in 2001 – Agile.
It was born in the USA when people came to the conclusion that companies lost sight of what really matters – delivering products that please their customers. The companies needed a shift in their focus, from unending planning and documenting their software development cycles to a more flexible approach.
In 2020, we believe that the current Agile methodology needs some tweaking. With the Agile Data-Driven approach, we want individuals, teams, and organizations, to reach their full engineering potential.
In 1985, the United States Department of Defense mentioned the Waterfall method in their standards for working with software development contractors. The nature of the waterfall approach is highly sequential and consists of six phases, each following the other, that have to be completed one by one. One phase can’t start without the previous one being completed.
The phases of the waterfall methodology are:
Agile was born in the early 2000s, in the USA, when 17 people met to discuss their discontent about the current state of project management within software. While they didn’t agree about everything, they came to the conclusion that companies lost sight of what really matters – delivering products that please their customers. The companies needed a shift in their focus, from unending planning and documenting of their software development cycles to a more flexible approach.
Agile is based on the idea of iterative development. Agile teams deliver work in small but consumable increments. One of its strengths is the ability to respond to change quickly because Agile teams are able to evaluate requirements, plans, and results continuously.
The Agile development process generally looks like this:
Agile is a method that comes in a few frameworks. The most widely used frameworks are:
The Agile Data-Driven methodology is enabled by Git tools. Thousands of organizations store code in the cloud, which opens the opportunity to track the engineering output and help managers make decisions based on data.
We believe that the Agile Data-Driven methodology is the next step in software development, whether you are using Agile, Waterfall, or a mix between the two of them.
It is designed to help engineering leaders spot roadblocks, identify coaching opportunities, provide concrete feedback, drive constructive conversations, and increase the overall velocity and quality of their delivery.
The end result of incorporating data into your teams’ workflow is the improved quality and velocity of delivery. There are quite a few reasons why we believe that the Agile Data-Driven methodology is better than traditional Agile approaches. We have studied and documented some of its benefits:
The Agile Manifesto and the 12 Principles were written by a group of software developers (and a tester) to address the issues in software development. Here is our take on the original 12 principles of the Agile Manifesto:
Then: Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
Now: Our highest priorities are satisfying the customer through early and continuous delivery of valuable software, while keeping team members happy.
Then: Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
Now: Welcome changing requirements, even late in development, but acknowledging the underlying costs of changing requirements, such as increasing costs and delayed deadlines.
Then: Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
Now: Deliver working software frequently, while maintaining a steady cadence.
Then: Business people and developers must work together daily throughout the project.
Now: Provide business people and engineering leaders access to Git Analytics reports, while allowing developers to focus on code
Then: Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
Now: Build projects around motivated individuals. Give them the environment and support they need, trust them to get the job done, and provide them with the feedback required for growth.
Then: The most efficient and effective method of conveying information to and within a development team is a face-to-face conversation.
Now: The most efficient way to communicate within a development team is through one-to-ones, supported by individual data and reports.
Then: Working software is the primary measure of progress.
Now: Working software is one of the measures of progress, along with the insights provided by Git Analytics tools.
Then: Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
Now: Agile Data-Driven processes promote sustainable development. Managers should help engineers to maintain a constant pace.
Then: Continuous attention to technical excellence and good design enhances agility.
Now: Technical excellence and good design are byproducts of the Agile Data-Driven methodology.
Then: Simplicity — the art of maximizing the amount of work not done — is essential.
Now: Maximize the amount of work not done and aim to optimize the work in progress.
Then: The best architectures, requirements, and designs emerge from self-organizing teams.
Now: Encourage autonomy within teams by letting data speak for their performance.
Then: At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Now: Engineering managers should use data and reports when providing feedback, consequently helping engineers become more effective.
Scrum was created as a framework that helps teams work together more efficiently. Scrum’s main goal is to help teams continuously improve by learning through experience and self-organizing when working on projects.
Data-Driven Scrum, compared to the traditional Scrum, is an Agile Data-Driven framework improved by the addition of real-time data and automated reports generated directly from Git repos. Used together, they help engineering teams manage their work more effectively.
Adding the data-driven aspect to the Scrum framework doesn’t impact its simplicity. The rules, artifacts, events, and roles remain the same, but with an added notion of data. Data-Driven Scrum users can analyze the effectiveness of their team members to tailor the Scrum framework to the organization’s needs and capabilities.
The semi-prescriptive approach and the use of real-time reports will remove the ambiguities in the development process. The Data-Driven Scrum will allow enough room for organizations to introduce their individual flavor to it.
Data-Driven Scrum will facilitate the organization of complex tasks into feasible user stories. The development cycle will benefit from the use of the framework. The clear delimitation of roles and the planning of the events will translate into transparency and ownership throughout the process. Quick releases will keep the team motivated and the users happy, as they can see constant, incremental progress.
If already familiar with the Scrum framework, mastering Data-Driven Scrum will not be a challenge, especially if the development team is already adapted to it. The concepts of smaller iterations, daily scrum meetings, sprint reviews, and identifying a scrum master will remain the same. Teams will just benefit from the addition of data to Scrum.
In the end, Scrum’s success in developing complex software products across diverse industries and verticals makes Data-Driven Scrum a great framework for any organization to adopt.
Successful sprints are often measured by the completion of items. That’s the most basic measure of success. Success is also tied to technical debt. We suggest avoiding introducing technical debt with reopening tickets, bugs spawns, quality errors, and more, via the newly created code.
Engineering teams now have data available, to monitor and diagnose any outliers in the software development process. This enables them to deliver consistently and avoid introducing new technical debt.
Future Scrum ceremonies will be influenced by the patterns that engineering managers identify in retrospectives, the actions they agree to take in response, and the target ranges they will set.
Data-Driven Scrum helps you run effective retrospectives and plan sprints better. Using the Data-Driven Scrum framework, engineering managers can spot developers that may need more assistance during the next sprint, make more informed resource allocation calls, and ultimately help their team align toward common goals.
The time spent in a data-driven retrospective brings teams closer to those recurring meetings, mission, alignment, and calibration. Engineering managers can run a more actionable retrospective by integrating the key metrics they’ve identified as the most relevant for their team and then driving conversations around those trends.
Daily standups are meetings that gather the core team: developers, product owners, and the scrum master. These meetings unfold differently for each team. Most oftenly, their structure is defined by these questions:
The reason why these questions are used frequently by teams is that they focus on progress and finding the blockers.
However, the Agile Data-Driven methodology supports a different approach to daily standups. It suggests that engineering teams run two daily standups per week – one at the start of the week and one at its end.
The reason why daily standups can be reduced to two per week is because Git Analytics tools can answer the questions mentioned above, with no input required from engineers.
Moreover, as Paul Graham highlighted in his essay, meetings have a negative impact on engineers’ productivity. They usually end up losing half a day over a simple meeting. The Agile Data-Driven is designed to reduce the meeting load of engineers, so they can focus on delivering business value.
Some of the reports that enable data-driven daily standups are:
One-on-ones are scheduled meetings between engineers and managers to check-in and see how everything is going. These are meant to build rapport, uncover potential issues, discuss and diagnose performance, and recognize achievements.
The Agile Data-Driven methodology supports a data-oriented approach to one-to-ones. With data, engineering managers can drive conversations to uncover and address potential issues, making one-to-ones more meaningful. Git Analytics reports, such as the Developer Summary, help managers discuss, diagnose, and recognize performance, helping them build rapport with engineers.
When engineering managers support their team’s narrative with concrete data, they can communicate to the organization before the release is delayed, so there are no surprises. They can show the consequences of ambiguous requests. They can advocate on behalf of specific team members, justify additional budget needs, and connect the dots between their team and the outcomes of the business.
Upward communication is different from team communication. We could say that it’s the inverse of it. If communicating with the team is about autonomy and over-communicating may prove to be distracting or condescending, upward communication is the opposite. There is no such thing as too much communication, especially in large organizations. The multitude of updates serves as valuable reminders to the stakeholders about all pieces currently in play.
Senior leaders need managers to push information up to them. They want to know more about what each team is focused on and how they’re progressing. One-on-ones can be a great place to showcase the work each team is doing.
When engineering managers are pushing information up, we recommend using data to support their narrative. By coupling data with qualitative analysis, they can help nontechnical stakeholders and senior leaders process the information more effectively. In doing so, in pushing visibility up while using concrete data, managers are simultaneously building trust and credibility for their team.
Code review is intended to find and fix unnoticed mistakes in the development phase, improving both software quality and the engineers’ skills. In an ideal environment, everyone on the team is actively involved in the review process, collaborating to improve solutions to problems and ensure that the work meets the team standards.
Reviewers are providing thoughtful and timely feedback, and submitters are responding to that feedback, engaging in discussion, and incorporating suggestions. But oftentimes, real-life processes don’t work like that. This is where the input of data can help organizations improve their code review workflow.
The code review process is intended to improve the quality of work before it goes into production. A healthy code review process reduces the risk of introducing bugs and enables knowledge transfer and code collaboration within a team.
Some of the reports that can help engineering managers improve their teams’ code review workflow are:
The cycle time is tied directly to concepts that many engineering organizations identify with, such as creating smaller stories, providing quick, meaningful feedback, and iterating.
While Scrum is designed to help engineers iterate swiftly and ship faster, undetected bottlenecks along the way can upset any development schedule. This is why the Agile Data-Driven approach to Scrum aims to bring visibility into the engineering teams’ code review workflow.
Retrospectives are an occasion for the team to reflect on the previous sprint. Engineering teams discuss their successes and failures. Then, they collaborate on what actions they can take to improve the process and the product in the next iteration.
A commonly used framework for facilitating team discussion in retrospectives is ‘start, stop, and keep.’ Engineering managers might use this framework to drive a round table discussion. During this discussion, team members are asked to share approaches they would like to be added to the team’s process, such as specifying early acceptance tests with customers.
When asked what they would stop doing, team members will express changes that would help the team save time or find better solutions. This could mean a change in how the team interfaces with external stakeholders or when they run daily standups every day.
Managers can prepare for data-driven team retrospectives by identifying relevant data points in their team’s workflow using data aggregated by the reports mentioned above. Then, they need to gain an understanding of what target ranges signal healthy development.
After that, the Agile Data-Driven framework suggests using that information to facilitate discussions around any differences between the target ranges and actual values. Afterward, engineering managers can have discussions about what they’ve learned, and the learnings can be applied to the next sprint.
Photo sources: https://agile-lounge.com/18-years-of-agile-manifesto-for-software-development/
(Disclaimer: The author is CEO & Co-founder Waydev)
Create your free account to unlock your custom reading experience.