The Diffusion of Responsibility Phenomenon Analyzed: Unravel the Truth

As a project manager for a software development team Do you know the team structure? Do you know who is working on what? Do you know whether the respective developer has the best knowledge prerequisites? “You don’t have to”, say the advocates of agile development, because “the team organises itself”. And by the way, the principle of “collective code ownership” prevails, i.e. everyone is responsible for everything in the code . [1] . With a little thought, one starts to wonder, e.g. because “collective code ownership” contradicts the principle of modularisation and efficient development [2] But the principle is not only dubious, it also causes harm. Among other things, the phenomenon occurs, in which no one really feels responsible any more. Individual developers lose sight of the bigger picture because many of their fellow developers have also changed the same code. “diffusion of responsibility” [7] In the process, the different development styles mix into a cacophony of code, and the cognitive effort required to understand the code increases to the point of incomprehensibility. How true it is that many cooks can spoil the broth. An erosion of the quality of the work product becomes inevitable and ultimately recognisable by the fact that escalating bug-fixing efforts occur. Now it is obvious to ask about the alternatives to “collective code ownership”. What are good distributions of responsibility and knowledge in an (agile) team instead How can you recognise early enough that a team is not (any longer) working together effectively? What other alarm signals are there for problematic projects and what are the answers to them? In this article, we will not make purely theoretical observations, but analyse the as a current and relevant example. We examined the knowledge structures of the two iOS and Android development teams step by step to identify good and risky patterns such as . Experience shows that the conclusions and recommendations from such an analysis can be used by self-organised software teams as well as by project managers for more effective team cooperation and leadership. German Corona-Warn-App Knowledge Balances, Knowledge Islands and Knowledge Tangles Please find more insights about the architecture quality and technical debt of the Corona-Warn-App in our blog: Corona-Warn-App – On the Way To Critical Technical Debt? Assessment Criteria for the Team and Knowledge Structure The fundamental challenge in any development project is to ensure an effective and robust team and knowledge structure. After several studies , we have found that intuitively develop a good kind of knowledge sharing and knowledge structure. We refer to this pattern, when exactly two developers work on a not-too-small shared set of source files, as . [5] teams of two Knowledge Balance Such an ideal distribution of knowledge and skills does not always necessarily arise in reality, nor is it always sought. We therefore recommend analysing this aspect in good time for larger and urgent projects. From our analyses and experiences, we have compiled some patterns (knowledge patterns) of knowledge distribution in Table 1: Table 1: Knowledge Patterns and Explanations To visually identify these patterns of knowledge distribution, we use network diagrams from the analysis data. The diagrams allow first qualitative indications of possible challenges regarding a too inhomogeneous knowledge distribution. Fig. 1 shows an example for the iOS Corona-Warn-App in Q4/2020. DETANGLE Figure 1: Network diagram showing the relationship between source code files and developers (committers) for iOS and Q4/2020 Source code files are shown as rectangles, developers (committers) are shown as circles. An edge between a developer and a source code file means that the developer has worked on the file and made changes during the period in question. In view of more than 20 developers, the diagrams are quite confusing without further filters. Nevertheless, the aforementioned large (committer62, top centre) stands out easily. Furthermore, figures 1 and 2 confirm the values of the effort distribution already mentioned, namely that committer2, committer62 and committer 24 account for more than 61% of the development effort in iOS app development. Knowledge Island Figure 2: Files outlined in green were processed by committer2 in Q4/2020 Fig. 3 shows the network diagram for the Android variant of the Corona-Warn-App. Here, too, knowledge concentrations can be easily identified as clusters in the middle and at the edge. Figure 3: Network diagram showing the connection between source code files and developers (committers) for Android and Q4/2020 The visual impression of is confirmed by the values on the inhomogeneous effort distribution (Fig. 4). Knowledge Islands Figure 4: Comparison of development efforts between Android (left) and iOS developers (right) in Q4/2020 Assessing the Effectiveness of Knowledge Sharing However, in order to take appropriate countermeasures, the effectiveness of the knowledge structure at hand must be assessed in detail. To do this, we will apply certain views or filters to these network diagrams: (e.g. by bugs or features/enhancements): source code files, developers and edges are only displayed if the files have only been worked on in the context of an issue of the selected issue type. Issue type : filtering by certain patterns such as and . Knowledge Patterns Knowledge Islands Knowledge Balances Knowledge sharing is effective when: Developers should be enabled to ramp up or expand their knowledge. They should be able to gain a level of knowledge beyond initial bug-fixing that enables them to contribute to the implementation of new features to a high degree. Ultimately, as many developers as possible should have a global system knowledge to be able to carry out improvements (e.g. refactorings) in the code or major architectural changes independently. At the same time, the necessary coordination effort and the communication paths between developers should remain limited. And the development work can still be distributed and run in parallel as far as possible. At the end of the period under review (Q4/2020), the majority of developers (15 out of 21 for the iOS app and 15 out of 28 developers for the Android app respectively) were involved in bug fixing. But a relevant perspective is formed by the meaningful views on feature development. In Figure 5 (for iOS), and are very visible. Knowledge Islands Knowledge Balances Figure 5: Source code files and developers of the feature and enhancement development (for iOS) The following insights for the iOS app and its feature development result from the analysis data: Only 11 (out of 21) developers add value by developing features or enhancements. One of the main developers already mentioned (committer2) forms a large (circled in red in the middle). Knowledge Island There are four other smaller (circled in red). Knowledge Islands All developers work more or less intensively with the main developer and form small or larger with him (marked with a green rectangle). Knowledge Balances The main developer (committer2) takes on the role of coordinator in feature development. Therefore, it could be qualitatively concluded that nine of these eleven developers are each involved in feature development together. Those four who are islands of knowledge themselves already have more system knowledge, so they can work independently on a larger set of source files. Other developers mainly work with the coordinator on feature development. One could therefore draw the positive conclusion that various developers in the project team are either being built up and/or already have advanced system knowledge. It would even be possible that even the independent knowledge in the of the coordinator and the said four other developers is balanced out by the shared knowledge in the . Knowledge Islands Knowledge Balances The DETANGLE analysis of the Android app comes to very similar results as for the iOS app and is therefore not explained further here. Key Figures for Evaluating the Effectiveness in Knowledge Distribution For further analysis, the more qualitative visual impressions of the network diagrams are supplemented by quantitative evaluations of the data. Two indices corresponding to the respective patterns, the and the are measured for this purpose. Knowledge Island LOC Index Knowledge Balance LOC Index, Knowledge-Island-LOC Index (KIL) This is the percentage of Lines of Code (LOC) of source code files on which has worked compared to the LOC of all source code files on which development has generally taken place in a given period. The number is measured per total system or per directory. exactly one developer Knowledge-Balance-LOC Index (KBL) This is the percentage of the LOC of the source code files on which have worked, compared to the LOC of all source code files on which development has generally taken place in a given period. Different pairs of developers may have worked on different source code files from the same directory. exactly two developers The two and index values together result in a value below 1.0 (i.e. below 100%). The delta to 1.0 (100%) gives the LOC share of files in a directory in which at least three developers were actively involved. The KIL and KBL values for the entire iOS app are now shown in Figure 6. Knowledge Island LOC (KIL) Knowledge Balance LOC (KBL) Figure 6: Change in the KIL and KBL values of the iOS app from Q2 to Q4 / 2020 The above conclusions can be confirmed quantitatively: The is 27% (approx. ¼) in Q4 / 2020 for the entire iOS app. The KIL value even fell by 32% in Q4 (compared to Q3 / 2020), which initially reflects a good trend and shows that the islands of knowledge behind the main developer are decreasing. Knowledge Island LOC Index In contrast, the of the iOS app remained just above or equal to the critical threshold of 20% (approx. ⅕) throughout 2020 even after the team change. This means that the cooperation visible in the network diagrams as a team of two is still far from sufficient to achieve at least an average value of 30% or a good value of 40% of the code base of for good knowledge sharing. Knowledge Balance LOC Index Knowledge Balances In comparison, we now consider the values for the Android app in Figure 7: 1. Here, the even fell after the team change. However, this is still at a value of 28%, i.e. close to an average value of 30%. This means that working in teams of two is a little more advanced on Android than on iOS. This is already indicated by the direct comparison of the network diagrams of the iOS and Android feature development. Knowledge Balance LOC Index 2. The better picture regarding is canceled out by a constantly critical of the Android app. This amounts to a very high 57% in Q4 / 2020 and implies that the coordinator continues to represent an oversized in Android feature development. Knowledge Balances Knowledge Island LOC Index Knowledge Island Figure 7: Change in the KIL and KBL values of the Android app from Q2 to Q4 / 2020 Recommendations to Improve Effectiveness of Development Even with limited access to the SAP JIRA project and the lack of opportunity to talk to the developers, the following recommendations are made to improve the effectiveness of the development of new features in the Corona Warning app: Table 2: Recommendations for improving the knowledge and effort distribution in the iOS and Android versions of the Corona-Warn-App Collective Code Ownership Considered Harmful The KIL and KBL index values related to feature development can also be broken down into source code directories as shown in Figure 8 for Q4/2020. The KIL values are plotted on the horizontal axis and the KBL values for source code directories are plotted on the vertical axis. Each circle represents a source code directory, the area of the circle reflecting the LOC of all changed source code files in the directory. In other words, the larger the circle, the more LOC are the source code files that were changed in the directory in the period under consideration for feature development. Figure 8: Source code directories and their KIL and KBL values (iOS in Q4/2020) The diagram is divided into the following four squares, each with a different frame color: Directories with a low value (less than 30%) and a medium to high proportion (at least 30%) of files. So it is a very good knowledge structure. Green: Knowledge Island Knowledge Balance Directories with a value of medium to high (over 30%), but which are offset by a medium to high proportion (at least 30%) of . These directories still have a good knowledge structure. Yellow: Knowledge Island Knowledge Balances Directories whose high value is not offset by . Here we are dealing with critical . Red (right quadrant): Knowledge Island Knowledge Balances Knowledge Islands Directories on which at least three developers have worked that cannot be attributed to either critical or good . These directories require a further critical review of how many developers have actually worked on them, and whether it is a rather “orderly” or “chaotic” way of working for developers on common source code files. Dark red (left quadrant): Knowledge Islands Knowledge Balances The reasons for viewing these directories as dark red, i.e. as very critical, are discussed below considering the principle of “Collective Code Ownership”. This categorization can be understood using extreme examples. One of them are the “green” directories on the top left (KIL value of 0, KBL value of 1). These circles represent directories in which only two people were involved in development. In contrast, the circles at the bottom right of the diagram (KIL value of 1, KBL value of 0) represent “red”, critical directories with a single developer. Knowledge Island Now let’s look at the selection of directories that we have already identified in our blog under with regard to architecture quality, documentation and technical debts as in need of improvement. [6] These are the and directories of the iOS app source code, which cover features for risk assessment and handling of Covid test results. These directories show a global coupling of these features and should also be viewed from the point of view of knowledge distribution. Risk Exposure Submission Table 3: KIL and KBL values of the Risk and Exposure Submission directories and LOC of the changed source code files in Q4/2020 The consideration of their knowledge index values allows several conclusions to be drawn: The directory from the green quadrant represents a very good knowledge structure with a good and low shares. src/xcode/ENA/ENA/Source/Scenes/ExposureSubmission/__tests__ Knowledge Balance Knowledge Island In the directory from the yellow quadrant, the property as a is offset by a high value. src/xcode/ENA/ENA/Source/Services/Risk/Calculation Knowledge Island Knowledge Balance LOC The vast majority of these directories are in the dark red quadrant from Fig. 8. These are neither nor . This means that more than three developers were involved in the majority of the files being edited. Knowledge Islands Knowledge Balances For these directories it can be measured whether several to many developers have worked “together” in a chaotic manner. From our project experience we know why such source code directories “magically” attract maintenance effort. These are source code files that many developers have had to make more or less changes to over and over again. Ultimately, the “Diffusion of Responsibility” phenomenon occurs that at some point no one feels really responsible for it any more. So developers sometimes lose perspective because many of the co-developers have also changed the code. [7] The Committer Friction Index (CFI) can also be measured using the DETANGLE analysis methods. We have plotted the diagram with the committer friction values and the number of developers for Q4/2020 in Figure 9: Figure 9: CFI values of the Risk and Exposure Submission directories; x-axis: CFI, y-axis: number of developers Looking at the values in Fig. 9 allows the following observations: The highest number of developers have worked on directory . src/xcode/ENA/ENA/Source/Scenes/ExposureSubmission/__tests__ Although it has a high , its Committer Friction index is still in the medium range. This means that a directory with a large number of developers can simultaneously have a high proportion of source code files as part of and, on the other hand, several source files that have been edited in an unstructured manner by many developers. Knowledge Balance Index Knowledge Balances The following directories mentioned so far have high Committer Friction index values: src/xcode/ENA/ENA/Source/Services/Risk/Provider, src/xcode/ENA/ENA/Source/Scenes/ExposureSubmission/View, src/xcode/ENA/ENA/Source/Services/ExposureSubmission The effect on these directories with a high Committer Friction index is that they become maintenance hotspots with high bug-fixing efforts. These source code areas, like and , can be made visually visible as Knowledge Tangles (as shown in Table 1) in the network diagrams below. Knowledge Islands Balances Figure 10: KI / KBs of the iOS app Q4 2020, Risk and Exposure Submission files outlined On the one hand, Figure 10 shows of the iOS app in Q4/2020 in connection with bugs as red triangles. A few source files are affected by three or two bugs, several by one bug, but the vast majority (approx. 95%) are free of them. Knowledge Islands/Balances Figure 11: Knowledge Tangles of the iOS-App Q4/2020, Risk und Exposure Submission Source Files are outlined in dark colour In contrast, Figure 11 shows a large Knowledge Tangle for the iOS app, which is heavily affected by bugs compared to and . In both diagrams, source files from the and directories have a black border. Knowledge Islands Balances Risk Exposure Submission It can be seen that, as assumed from the values from Table 3, they are represented to a large extent in the aforementioned Knowledge Tangle. Approx. 85% of these source code files originate from the three source code directories mentioned above and located on the right in Fig. 9 with a high Committer Friction Index. The analysis and visualization with DETANGLE makes it impressively clear that Collective Code Ownership has shown its pitfalls and, in this case, markedly negative effects. Summary and Recommendations In this article, the authors have used an analysis of the development of the Corona-Warn-App in the period from Q2 to Q4 2020 to present ways and means by which teams and project managers can measure and monitor the relationship between development efforts and the effectiveness of knowledge distribution. The findings can be used to identify project and product risks at an early stage and to initiate appropriate countermeasures in a timely manner. In detail, the analysis of the code history revealed that knowledge and team structures were not or hardly have been explicitly analysed. The extraordinarily inhomogeneous distribution of knowledge ( ) in the teams of the two sub-projects iOS app and Android app was detected and visualised with the DETANGLE software analysis suite. Knowledge Islands It seems that the principle of self-organisation of teams using reviews and giving mutual feedback was relied on only. That the hoped effect did not happen gets evident by the also very inhomogeneous distribution of the development load within the two teams. In case of the Android app, 60% of this load was the responsibility of only one developer. In case of the iOS app 50% was the responsibility of two developers, and this with a team size of up to 28 developers. Such ineffective knowledge and inhomogeneous effort distribution leads to general inefficiencies in the development process and also to undesirable dependencies on individual developers. The loss of one of these developers can jeopardise the quality, the launch date or the entire success of the project. In addition to the easily ascertainable patterns of , and were introduced as further patterns that can be visualised via network diagrams and quantified down to the level of source code directories. This allows us to know where and when and individual coordinators represent a risk and how they can be compensated for by strengthening the . Knowledge Islands Knowledge Balances Knowledge Coordinators Knowledge Islands Knowledge Balances Last but not least, it has been shown that the principle of Collective Code Ownership can even do harm and affected areas of the code can become hotspots of maintenance. Bug fixing is a kind of non-productive work. To ensure sustainable value creation in feature development, the authors recommend conducting these analyses concurrently across iterations of development. The results would be an essential input for team reviews to initiate improvement measures or for the purpose of transparency for project managers, clients and users. Our diagnosis that the distribution of knowledge and development efforts in the development teams of the Corona-Warn-App is problematic, possibly has been confirmed by reality, because when new features were launched, delays and functional restrictions occurred repeatedly, up to and including the failure of the app . [8] Note: The presented results and insights are only a small part of the relevant information that could have been achieved if the authors had been given open access to the development teams and to the non-public SAP JIRA projects used by the project team. About the Approach The analysis of the development of the Corona-WarnApp was carried out with the data available on GitHub as an open source project, but without the involvement of the companies commissioned with the development. For the analysis we used the DETANGLE Analysis Suite, which allows an evaluation of the GitHub issues/pull requests, the developers involved and the code changes resulting from them over the entire code history of the two apps. The data from the non-public SAP JIRA issue project could not be used. A detailed description of the procedure can be found at . [3] We focus on data resulting from changes to common source code files for new features. We do not take into account merges of the commit history that result from pull requests, as these are usually related to reviews and would “dilute” the data in terms of knowledge distribution. March 10, 2021 by Egon Wuchner and Konstantin Sokolov This article first appeared on the Cape of Good Code blog at: https://capeofgoodcode.com/2021/03/11/rethinking-collective-code-ownership-knowledge-distribution-risks-demonstrated-with-the-german-corona-warn-app/ Links [1] Agile Alliance, Collective Code Ownership https://www.agilealliance.org/glossary/collective-ownership [2] heise developer, “Collective Code Ownership: Ein Anti-Pattern? https://www.heise.de/developer/artikel/Collective-Code-Ownership-Ein-Anti-Pattern-3909449.html [3] Blog Cape of Good Code, Details of the analysis approach https://capeofgoodcode.com/2021/02/23/corona-warn-app-on-the-way-to-critical-technical-debt/#About_the_Approach [4] JCON 2020, “Feature and Time-Based Software Analysis” Measuring Development Effort: 5 Minutes: 13:10–18:27 https://youtu.be/aD4OQScGILo?t=790 [5] Blog Cape of Good Code, “Cooperation and Knowledge Sharing for Remote Agile Development” https://capeofgoodcode.com/2020/06/01/cooperation-and-knowledge-sharing-for-remote-agile-development/ [6] Blog Cape of Good Code, “Corona Warn App — On the Way to Critical Technical Debt?” https://capeofgoodcode.com/2021/02/23/corona-warn-app-on-the-way-to-critical-technical-debt/ [7] Diffusion of Responsibility https://web.archive.org/web/20100227092905/http://greatergood.berkeley.edu/greatergood/archive/2006fallwinter/keltnermarsh.html [8] Handelsblatt, “Die Fehler der Corona-App offenbaren eine gefährliche Schweigsamkeit der Firmen” https://www.handelsblatt.com/meinung/kommentare/kommentar-die-fehler-der-corona-app-offenbaren-eine-gefaehrliche-schweigsamkeit-der-firmen/26038162.html