Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Background and Related Work Research Method Results Discussion Conclusion and References Background and Related Work Background and Related Work Background and Related Work Research Method Research Method Research Method Results Results Results Discussion Discussion Discussion Conclusion and References Conclusion and References Conclusion and References 4 Results The main insights from the interviews and questionnaires are outlined below. We present opportunities and challenges that BPM industry experts perceive regarding the AB-BPM method. As software vendor employees, the study participants’ answers, to some extent, reflect the experience of the wider industry, i.e., the customers’ challenges. Statements of the experts are marked as quotations. Opportunities. The sentiment towards AB-BPM was mainly positive. Multiple consultants brought up that some companies they worked with tried testing new process versions and comparing them with the status quo. However, the tests were mostly unstructured and considered only of a few instances or even no “real” instances (i.e., only tests). This means that any drawn conclusions are not dependable, due to the low number of instances and lack of statistical rigor when it comes to controlling confounding factors. Thus, AB testing provides a useful process improvement method that supports the structured testing of alternative versions. Opportunities sentiment One question to the study participants was about the suitability of the method regarding contextual factors. The study participants were not presented with a list of categories but were free to elaborate on their intuition. More concretely, they were asked for what type of processes and what surrounding circumstances (company, market, industry) they think the methodology would be well- or ill-suited. Their statements were then mapped to the categorization of BPM contexts by [29]. The result can be seen in Table 1. The characteristics in italics present special cases for factors where every characteristic was deemed suitable, which we will outline in the following. suitability Focus. No agreement could be reached on whether AB-BPM was suitable for radical changes. [24] present the method primarily for evolutionary changes, while some study participants believe it is suitable for both. However, most consider the method more appropriate for small process changes due to the ease and speed of implementation. AB-BPM would be incompatible with fundamental changes that require “lengthy discussions” and expensive financial obligations, making rapid testing difficult. Somewhat larger changes within the same information system may be feasible, but smaller changes are generally preferred. Focus Value contribution. Using the AB-BPM method might be especially useful in core processes. This is because other processes are found at many companies and cannot be used for meaningful differentiation. One study participant noted that it is advisable to “differentiate where you differ.” They said, “as a sports shoe company, we could strive to have the best finance processes, but that won’t make people buy our shoes – we need better shoes and better shoe quality to win in the marketplace.” They, therefore, recommended using standard processes for everything but the core processes. This is already common practice and also suggested by academic studies [25]. For core processes, however, experimentation with the AB-BPM method would be highly favorable. Value contribution. Competitiveness. In general, there were no opinions indicating that any level of market competitiveness would lead to less suitability of the method. Study participants noted, however, that highly competitive markets would increase the need for such a tool to allow for faster process iterations, “to stay competitive.” Competitiveness The elicitation of requirements for a tool that executes and supports the AB-BPM method was also part of the study, and the identified feature requirements are presented in the following. First, we present the ranking of the items (see Table 2). Afterward, more details on the items ranked as most important are provided. requirements The ranking is based on the importance Likert scale presented in Section 3. The average importance scores (AVG) are accompanied by the standard deviations (SD) to give an insight into the level of agreement among the experts. Furthermore, the feature requirements have been categorized into presentation, procedure, and support. This categorization has been created after and based on the interviews, during the coding of the interviews (GT phase intermediate coding [4]). Presentation includes features regarding the presentation of data, or features that are more focused on the front end of the tool in general; Procedure are features regarding the underlying technical or methodological procedure; Support includes features that already exist in the AB-BPM method but that have not been presented to the study participants during the introduction to AB-BPM (see Section 2); they, therefore, support the equivalent suggestions by [24]. In the following, the three tool features ranked as most important are described in more detail. See potential impact beforehand (amount and business-wise). According to the study participants, process experts should be able to see estimations on possible impacts beforehand to support an informed decision-making process, e.g., how many customers or what order volume would be affected by the test. See potential impact beforehand (amount and business-wise). BPMN diff viewer. One study participant emphasized the importance of human experts having a clear understanding of changes in the current initiative. They suggested a ”diff viewer for the diagrams” - a graphical representation of changes made to a document from one version to another, commonly used in software engineering. In business process management, this could involve versions of a BPMN diagram, with changes highlighted in different colors. Diff viewers are well-researched and applied in this context, for example in [19,8]. BPMN diff viewer Communicating process changes efficiently for teaching and enablement of employees. The need for process participants to learn how new versions have to be executed was stressed by multiple interviewees. One study participant stated that “one needs to notify the people working on steps in the process of the changes.” More “enablement is needed to teach employees the changes,” and another study participant noted that “seeing how this [aspect of change management] can be integrated would be an interesting question.” This would go beyond just teaching single steps but also create openness and transparency about goals and project setup, allowing for “a lot of change, even in parallel, without people being lost.” Similar to the diff viewer, change notification management is a feature that has already received research attention in the context of business process management software [30,14]. Communicating process changes efficiently for teaching and enablement of employees. Challenges. It is vital to know the core challenges to advance the AB-BPM methodology and adjacent endeavors. Only then can they be addressed and mitigated adequately. The risks and further challenges that the expert panel has voiced are, therefore, outlined below. Challenges A critical goal of this work is to determine the AB-BPM method’s principal risks since they hinder its usage and implementation in organizations. In the following, we will present the results from the experts’ ranking and then give a more detailed outline of the most highly ranked individual risks. The risks, alongside their AVG risk scores and the SD of those scores, can be seen in Table 3. Furthermore, the risks have been categorized as follows. Culture are risks regarding the working culture and employees of the company. Results include risks regarding results, decisions, and outcomes; Operations consists of risks regarding the implementation and execution of the AB-BPM method itself, but also the normal business operations; Legal includes risks regarding the cost and loss of income caused by legal uncertainty [27]. risks Results Operations Legal In the following, details on the three most highly ranked risks are explained in more detail. Unclear results due to high process variance and process drift. As mentioned before, the execution of business processes can differ from how they were intended to be executed and it is subject to (unintended) changes over time. This phenomenon, called process drift [23], leads to a high variance of executed process versions. This could pose a risk for the AB-BPM method since it is then unclear whether process participants execute the two versions as they are intended. A process participant is a company-internal actor performing tasks of a process [6], i.e., an employee of the organization executing the process. If the process cases vary from the intended way of execution, it is hard to draw conclusions from the results since they might be based on a change that occurred spontaneously instead of the planned process changes. One example might be that “people exchange emails instead of following the steps in the process execution software.” Unclear results due to high process variance and process drift. Erroneous machine-generated analysis results are blindly followed. Many interviewees noted that solely relying on the algorithm’s interpretation of the data might cause problems. One study participant noted that “such models are always an abstraction of reality [...] and relying on them completely can lead to mistakes.” This topic also came up during the discussion of bad prior experiences, when a study participant noted that sometimes wrong decisions were made because of a lack of understanding of data. One potential example is the use of team performance metrics, which are often highly subjective (e.g., workload estimates in some project management methods), without context. Putting data into context and not blindly following statistical calculations is, therefore, a core challenge that needs to be addressed. Erroneous machine-generated analysis results are blindly followed. Cultural change management problems during variant roll-out. This risk was added after the validation survey since one study participant remarked that this item was missing. It can be understood as incorporating any other cultural change management issues not yet included in the item list (see blue items in Table 3). The high rating of this item can be seen as an indicator that the human side of the method and adjacent tools must not be left out of research and development efforts. The importance of culture also became very apparent when asked about prerequisites for the use of the AB-BPM method. Many study participants noted that the organization would need to have an experiment culture, meaning that they should be open to trying new things and handling failures as learning opportunities. Furthermore, they stated a need for organizational transparency and trust. Cultural change management problems during variant roll-out. prerequisites for the use of the AB-BPM method The implementation and adoption of AB-BPM as presented in [24] assumes the existence of a Business Process Management System (BPMS) that allows for the direct deployment of BPMN models. A BPMS is an information system that uses an explicit description of an executable process model in the form of a BPMN model to execute business processes and manage relevant resources and data. It presents a centralized, model-driven way of business process execution and intelligence [6]. However, most processes are executed by non-BPMS software, i.e., they are not executed from models directly [9]. Therefore, whether the usage of a BPMS is a requirement for technical feasibility is a research question of this study. Altogether, AB testing of business processes seems technologically feasible without a BPMS, one interviewee noted: “I do not think that it is a problem that processes are executed over several IT systems since you only need to be able to start either process version. The route they are going afterward, even if it is ten more systems, is no longer relevant.” However, if we want to use live analytics to route incoming process instantiation requests (e.g., as proposed with RL) without a BPMS, we would need an Extract-Transform-Load (ETL) tool. ETL software is responsible for retrieving relevant data from various sources while bringing the data into a suitable format [28]. Relying on a BPMS would not only have the benefit of easier data collection and access, it would also make deploying and executing experimental processes more straightforward. Furthermore, such an ETL tool might also be highly complex due to the many systems processes can potentially touch. One study participant noted that when a BPMS does not exist, “you will have to put a lot of effort into mining performance data; it would be more difficult to get the same data from process mining, covering every path and system.” In fact, most study participants did deem a BPMS, or something similar, to be a prerequisite. One study participant stated, however, that while some “central execution platform” would probably be needed, it remains unclear whether these have to be in the shape of current BPMS. Overall, there seemed to be the notion that the integrated, model-driven way of orchestrating and executing business processes offered by BPMS is the direction the industry should move towards. technical feasibility Besides the risks, the study participants also mentioned other challenges. Here, we highlight some of them. Regarding the question of bad prior experiences when conducting BPI initiatives, some study participants criticized the unclear impact of process improvements during/after BPI projects. This is due to constantly changing environmental factors and the resulting difficulty to compare process data that has been collected at different points in time. This highlights the possible positive impact that AB-BPM could have on BPI efforts, by giving BPM experts a better data basis to evaluate improvement efforts. Regarding the prerequisites for the use of the AB-BPM method, on the more technical side, the interviewees noted that many companies would not offer the level of continuous data metrics needed for the dynamic, RL-driven routing during the experiments. This again highlights the need for an integrated process execution (e.g., a BPMS). other challenges. bad prior experiences when conducting BPI initiatives prerequisites for the use of the AB-BPM method Authors: (1) Aaron Friedrich Kurz [0000 −0002 −2547 −6780], Technische Universitat Berlin, Berlin, Germany and SAP Signavio, Berlin, Germany (aaron.kurz@sap.com); (2) Timotheus Kampik, SAP Signavio, Berlin, Germany (timotheus.kampik@sap.com); (3) Luise Pufahl, Technische Universitat Munchen, Munich, Germany (luise.pufahl@tum.de); (4) Ingo Weber [0000 −0002 −4833 −5921], Technische Universitat Munchen, Munich, Germany and Fraunhofer Gesellschaft, Munich, Germany (ingo.weber@tum.de). Authors: Authors: (1) Aaron Friedrich Kurz [0000 −0002 −2547 −6780], Technische Universitat Berlin, Berlin, Germany and SAP Signavio, Berlin, Germany (aaron.kurz@sap.com); (2) Timotheus Kampik, SAP Signavio, Berlin, Germany (timotheus.kampik@sap.com); (3) Luise Pufahl, Technische Universitat Munchen, Munich, Germany (luise.pufahl@tum.de); (4) Ingo Weber [0000 −0002 −4833 −5921], Technische Universitat Munchen, Munich, Germany and Fraunhofer Gesellschaft, Munich, Germany (ingo.weber@tum.de). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv
