Should Your Next Software Engineering Study Be Ethnographic?

Table of links

Abstract

1 Introduction

2 Is an ethnographic study the right choice?

• The context of your research
• The kind of research questions you want to answer
• What ethnographic studies require from the researcher

3 Planning an ethnographic study

• Finding a site for field work
• Participant or non-participant observation
• Duration of field work
• Space and Location
• Theoretical underpinning

4 Implementing your ethnographic study

• Gaining access and starting up
• Handling your preconceptions
• During the study
• Going Native
• Leaving the field

5 From Data to Text

• Reflective and inductive analysis
• Writing Ethnography for Software Engineering Audiences - Reporting the Results

6 Ethnography and Research Ethics

7 Final comments, Further reading and References

2 Is an ethnographic study the right choice?

Here we consider the question of whether or not an ethnographic study is the right choice from three different perspectives:

1. the context of your research
2. the kind of research question you want to answer
3. what it means for you as the researcher, because in an ethnographic study the researcher is the research instrument This section will discuss these three perspectives to help you to decide whether to use ethnography as a research method.

The context of your research

This perspective considers where the intended study fits in the wider context: what role might an ethnographic study play in the research? In social sciences, where ethnography originated, studies focus on understanding phenomena. They are used to understand better how (sub) cultures operate, or how groups of people organise their social life. In HCI, ethnographic studies are used to inform the design of technology from a user’s point of view so that the designers might better appreciate who they are designing for. In software engineering, ethnographic studies may also be used to understand or to inform design, but research often goes beyond understanding towards improving methods, techniques and tools. Understanding current practice is a good starting point to inform changes of any kind, not so that current practice can be replicated in the new order, but to appreciate why things are done the way they are, so that there are no unintended consequences when things change. A quite famous example is the long time it has taken to replace the paper flight strips used by air traffic controllers with digital versions. In 1999 Wendy MacKay commented that “Our observations have convinced us that we do not know enough to simply get rid of paper strips, nor can we easily replace the physical interaction between controllers and paper strips” [38]. She wrote this after being inspired by her work and previous studies about flight strips which started in 1992. A blog written in 2017 on the subject demonstrates how cautiously this change was eventually made [18]. In our analysis of ethnographic research in empirical software engineering [51], we identified four roles for ethnographic research, all of which start by considering “how things work in practice”

2.1.1 To strengthen investigations into the social and human aspects of software engineering

Since ethnography’s origins are in understanding cultures and communities, it is not a surprise that ethnography has been used in this way. It is inevitable that some insights regarding social and human aspects will emerge whatever research question is pursued since the approach champions the members’ point of view. For example, Lopez et al. [35] were interested in investigating how non-specialist software developers engage with security in practice. This explicitly focuses on the human aspect of software developers. They used a multi-sited study and a mixture of research methods. Their results included a set of episodes where non-specialists engaged in security activity, and five typical behaviours that characterise how individual developers respond to security concerns to meet the demands of particular circumstances. A key characteristic of this work was that it emphasised security from the developers’ perspective. The results form a framework that managers and teams can use to recognize, understand, and alter security activity in their environments (motivatingjenny.org).

2.1.2 To inform the design of software engineering tools

For a new tool to be successful it must support the task being performed, e.g. bug localisation. But for it to be useful to practitioners it must also fit into their workflow and support how they and their organisation performs the task. Focusing on a task in isolation gives only a partial view and however clever it may be, if it’s not usable then it won’t be used. So understanding the context of tool use is as important as understanding its technical and functional requirements. But there are other considerations too. As with much software development, it can be easy to jump straight into designing a system that seems to address a user’s problem [48]. However if the goal is to develop new software engineering tools then don’t start the design until you are sure that the proposal will be useful. This can be achieved by sketching a number of potential tools based on observations, and then collecting evidence to support or refute their value, e.g. by working with practitioners and analysing your data overall to evaluate the ideas. For example deSouza et al. [9]were interested in the role of application programming interfaces (APIs) in the coordination of software development work in large scale organisations. An initial observation led them to think that integrating the versioning software with email might facilitate the team’s work, but further data collection made it clear that this was only part of the situation, and focusing on a tool to support the team’s overall communication would bring them more benefit. So instead of developing something to integrate two existing tools, they developed a tool that supported the identification of dependencies between developers which allowed developers to coordinate their efforts more effectively and reduce unnecessary work.

2.1.3 To improve development processes

Ethnography can be a starting point, together with the practitioners observed, to discuss and implement improvements of their work practices. Ethnographic research then becomes part of action research. Two chapters in this book by Staron and by Dittrich, Bolmsten and Seilein introduce action research. A critical point is that the decision about the intervention and the evaluation of its implementation should keep the members’ point of view, both for ethical and methodological reasons. Dittrich et al. developed Cooperative Method Development (CMD), an action research approach combining ethnographical and ethnomethodological inspired empirical research with the improvement of software engineering tools, methods, and processes [17, 13]. CMD is explicitly designed to keep both the work practice focus and the developers’ point of view of the initial research throughout the action research cycle(s). Unphon’s thesis provides an example of action research [61]. The research cooperation with a company developing software products for simulating hydraulic systems focused on the evolvability of these software products. The researchers collaborated with the team responsible for the product simulating open one-dimensional water systems like rivers and creeks. The action research introduced light-weight software architecture techniques for high-level design, developed a light-weight Architecture Level Evolvability Assessment method for focussed discussions of design decisions with relevant stakeholders, and introduced light-weight architecture compliance techniques using the built system. Introductory research with the team in order to understand development practices as well as investigate the structure of the existing software product resulted in a framework for understanding the influence of the organisation of software development and business on the architecture of the software [62]. The research results emphasise the need to adapt software architecture methods and tools to support the continuous evolution of software products: architecture design and evolution take place as part of everyday evolution; architectural practices need to support the software architect to keep up with the changes of the software and the emerging requirements that might challenge the architecture; evolvability should be a quality to be considered in regular software architecture design discussions.

2.1.4 To inform research programmes that do not have ethnography at their core

It is common for research outputs such as tools and new processes to be ignored by practitioners, in favour of innovations suggested by other practitioners. Understanding the context of software engineering takes the researcher one step nearer to suggesting an innovation that is acceptable to practice. In particular, an ethnographic study may help a research programme by articulating more specific, relevant research questions. For example, our programme on agile software development started by looking at the realities of XP in practice, i.e. how was the approach implemented [53]? Ethnographic studies in this context led to other studies and research questions that focused on the role of physical artefacts [55], information flow between stakeholders [65] and collaboration in dispersed teams [11]. But ethnography can also be used to inform research programmes by providing context grounded in practice for an existing research focus. For example, Capiluppi et al. [6] investigated the evolution of code within an agile development project. This led to a number of observations regarding how the agile code base evolved. Using an ethnographic study as context allowed the researchers to:

• Characterise the development process and practices clearly, and

• Provide alternative interpretations of phenomena found in the data, e.g. where the size of the code base changed significantly they could trace an event through the ethnographic study to explain that a new library was loaded

Ethnographic observation in this context has the benefit that it is not dependent on the report of practitioners, as it is in interviews. When collecting reports from practitioners, their responses will inevitably be partial and informed by current events. For example, they might not consider certain aspects of everyday practice to be relevant to the questions asked, or they might not report practices that they consider to be informal or not accepted. However, these aspects may be exactly the characteristics that render the specific way of developing software possible, and hence would be important from a research point of view. The design of studies complementing and providing context to other research may not be driven by an independent research question, but the research question might be decided based on the overall research interest.

• When conducting an ethnographic study with the aim of investigating social and human aspects of software engineering it is easy to become overwhelmed by the huge number of focus points SO track the possible foci and address them one at a time.

• When conducting an ethnographic study with the aim of producing a new tool, it can be tempting to start designing before a full picture has been obtained SO start by hypothesising what kind of tools would be useful and collect data to validate those hypotheses.

• When conducting an ethnographic study with the aim of producing a new tool, it can be tempting to focus on specific instances rather than the broader picture SO focus on the analytical results rather than the data when hypothesising about potential tools.

• When conducting an ethnographic study with the aim of improving development processes it can be easy to suggest changes based on textbook versions of processes SO make sure to engage practitioners in any suggested modifications.

• When conducting an ethnographic study with the aim of informing other programmes of research, confirmation bias can easily cloud your observations SO be sure to seek confirming and disconfirming evidence.

2.2 The kind of research questions you want to answer

This subsection considers the kind of research questions you want to answer. The specific detail of the question may change as the study progresses (see section 4 below) but whether an ethnographic study is the right choice depends on the kind of question you want to answer. The first thing to note is that ethnographic research, like other qualitative research, is driven by research questions rather than hypotheses derived from theory. Research questions ask ‘How’ and ‘Why’ and ‘What are the characteristics of’ questions rather than ‘Is X better than Y’ or ‘Will this technique make programmers more productive?’ kind of questions. For example ‘How do software practitioners develop systems using XP?’ rather than ‘Is single programmer coding more productive than pair programming?’, and ‘Why don’t developers adhere to the company’s security policies?’ rather than ‘Does structuring the manual in this way help developers produce more secure code?’ and ‘What are the characteristics of a technology adoption?’ rather than ‘How did the ideas of Simula develop into Java? Other techniques and methods introduced in this book will support answering other types of question.

The strength of an ethnographic approach is that it emphasises the point of view of the participants, i.e. the members of the community under study, who are often called informants or interlocutors in ethnographic work. It therefore allows the researcher to understand better why things are the way they are. It brings research closer to practice and hence can make the results more acceptable within practitioner communities. For example, it may seem strange to a researcher to see agile practitioners using both a physical storyboard and a virtual one, especially because this entails duplicate work in keeping them both up to date. However, this is common practice where co-located or hybrid teams are deployed because of the different and complementary collaboration and communication benefits it provides.

2.3 What ethnographic studies require from the researcher

To help decide whether or not an ethnographic study is the right choice, this section considers the four main features of ethnographic work: the member’s point of view; the ordinary detail of life as it happens, the analytic stance and thick descriptions. For each of these we present considerations on whether the researcher (you) are prepared and willing to apply the approach appropriately, e.g. to write thick descriptions, listen to the informants’ point of view and modify your research focus. The information in this section will help to assess this, and will set your expectations for what it means to implement an ethnographic study.

2.3.1 The members’ point of view

We would argue that it is always important to understand the point of view of practitioners whatever is the focus of the research, for example whether developing a new tool, looking to improve the development process, or simply trying to understand better how something works in practice. As discussed in Section 2.1, taking account of the members’ point of view doesn’t restrict innovation, but instead understanding the rationale for the current way of working before introducing changes helps to avoid unintended consequences. Understanding the member’s point of view is all about understanding what is important for the informants so that this can be taken into account. In other disciplines, ethnographic studies involve the ethnographer studying a culture that is unfamiliar to them, e.g. in terms of language, customs, cultural norms etc. These studies may need to last for months or years. As a software engineer conducting an ethnographic study in a software development environment, this will be less of a challenge to overcome. Understanding someone else’s perspective is still hard, but much of the basic context of software development, e.g. IDEs, development processes, programming languages etc will be familiar. One of the benefits of this is that ethnographic studies don’t necessarily need to take months or years. Focusing on the members’ point of view will entail getting used to the organisation’s environment and the specific details of the team’s work but the learning curve is less steep than it would be for someone unfamiliar with software development.

The familiarity of the researcher with the tools, techniques and methods of the informants can make it difficult to keep the member’s point of view in mind when observing how software development takes place. It is very easy to apply the knowledge from your own education and research and slide into a “this can be done better” attitude. Such an attitude might even be provoked by the practitioners asking the researcher, who supposedly is an expert, whether they have any recommendations. Here it is important to separate observations from analysis for the duration of the field work and put your own judgement aside.

2.3.2 The ordinary detail of everyday life

The ordinary detail of everyday life is important because it exposes how tasks are addressed, the issues that are relevant or not and how informants approach tasks. For example, a series of studies with agile software development teams exposed that the colour of cards placed on a physical scrum board, and the way in which they are handled carries meaning beyond what was written on them [54]. This raised questions of whether digital storyboards would support the same kind of processing and collaboration as physical boards. This also means that the research will be conducted in the field or “in the wild” rather than in a controlled environment, and the research will aim to not disturb or control normal behaviour.

Equally, informants’ language is relevant. For example if the informants’ native language is different to the ethnographer, what might the impact be? If the domain of study is highly technical then the ethnographer would do well to understand the domain too. Performing an ethnographic study in an unfamiliar technical domain will have an impact on the timescale (as mentioned above) but also may lead to misinterpretation of activities and tasks. The gender of the researcher might both influence the field situation and the interpretation of the observation. Given that ethnography emphasises what is important from the members’ point of view, and that the researchers won’t know what they might find before entering the field, there is a strong chance that the research focus will evolve. A key issue to consider is whether the research context and design is suitably flexible to allow for the research focus to change? For example, if your focus is developing a new tool to identify areas of technical debt but your informants explain that their concerns are driven by improving the security of the code, how would that affect the research? Although the detail of everyday life is observed, it is not the detail itself that is important, but the significance of the detail (see section 2.3.3 below).

2.3.3 The analytic stance

An ethnographic account is not just a description of what is seen but is crafted out of an analysis and interpretation of what was found in the field, explicating why things are the way they are. The analytic stance is related to the members’ perspective. The idea is to understand and uncover the rationalities of practices and how the practitioners’ behaviour makes sense, even though it does not match the researcher’s own understanding of how software development should take place. This can be problematic to those unused to the approach. One example to illustrate this analytical stance can be found in [49], where a recording of a steering group meeting addressing a major change in a method and tool development project is analysed. The analysis shows in detail how the steering group relates the actions to take the development further to both the new plan for the project and the company-wide project model. The analysis shows that the steering group takes a decision to deviate from the project model in order to assure that the work in the project can progress using, nonetheless, the very same company-wide project model to make the deviation visible and accountable in the organisation. The analysis thus sheds light on the rationale behind deviations from plans and company-wide project models, and, at the same time, it shows the importance of a company-wide project model to communicate both behaviour according to it and deviations from it to other actors in the organisation. Not every analysis needs to be as fine grained as the interaction analysis performed in that article. However, the analytical stance requires the researcher to be able to relate the observations here and now to the spatial and temporal context and to the goal and purpose of the observed activities.

2.3.4 Thick descriptions

An ethnographic study results in a comprehensive and detailed set of data, and a detailed account of the findings. The account aims to communicate a broad picture of the study environment and activity. It needs to show how the researcher has arrived at their conclusions or recommendations. The set of data is specific to that site and hence is not suitable for statistical generalisation although it may be suitable for analytical generalisation. If you decide to undertake an ethnographic study this is an important point to remember. In an area where generalisation and statistical significance of results are expected, having a core ethnographic study may be challenging to convince others. The thick description can be used to defend the study and its results, but it cannot justify a statistical generalisation. An ethnographic researcher needs to acquire a different perspective on justification. This may be particularly pertinent for publication and if you are a PhD candidate