Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Background and Related Work Research Method Results Discussion Conclusion and References Background and Related Work Background and Related Work Research Method Research Method Results Results Discussion Discussion Conclusion and References Conclusion and References 5 Discussion Emerging patterns in the coded items can be brought together to form tool and methodological suggestions. This analysis of patterns in the coded interview data can be seen as an application of the GT phase advanced coding [4]. In the following, we present the inferred suggestions and concepts in more detail. These serve as both concrete suggestions and examples of how the requirements can be used when implementing AB-BPM tools. advanced coding Integrated Process Model Repository. The idea of integrating the ABBPM tool with a process model repository (PMR) came up in one interview. It is implicitly supported by statements of most of the other study participants. The main aim of IPMR is to provide a fast and efficient way of introducing new process variants to process participants while allowing them to provide feedback. Once a new business process variant is added, the responsible process expert would need to add additional material to help process participants understand the new version. Suppose a specific process participant is part of an experimental process case (or a new version is permanently rolled out). In that case, they would receive a notification (e.g., via email) with the most important information regarding the process update, with a link to more material. This could be extended with the need for process participants to pass a short test to avoid improvement effort failures due to misunderstanding. They could also provide feedback, which could then be evaluated by the process experts or even considered for the RL experiment. Integrated Process Model Repository. Instance Recovery Mechanism. The analysis of the interviews made it clear that the danger of failed process instances has to be mitigated in some way. Based on that data, we propose an instance recovery mechanism (IRM). Before starting an experiment, process experts could set thresholds for specific KPIs. The expert would be notified when a process instance reaches such a threshold. They could then intervene in the process execution to make sure that the process consumers still get their desired product or service. The thresholds should be set relatively high, to only intervene in erroneous cases. Otherwise, it could create a bias in the experiment. This is why one also has to consider how such instances are evaluated in calculating the rewards of the RL. Leaving them out of the RL calculations could lead to problems: Imagine only one process version sometimes exceeds the thresholds and has to be salvaged manually. By leaving these instances out, we might misjudge that version. A better approach could include these manually salvaged versions in the model with a certain penalty value. Considering the scientific state-of-the-art, IRM is closely related to the notion of (real-time) business process monitoring, as for example covered in [18,10]. Instance Recovery Mechanism. Another pattern that can be observed throughout the interviews is the need for more human control. This can be seen in the perceived risks (e.g., BLE, LRE), as well as in the desired tool features (e.g., EES, CPS, PSN, MRL, XRL). In a previous paper, we have already presented an initial prototype that enhances the AB-BPM methodology with additional features for human control [13]. The presented study has several limitations. One possible threat to the validity of results, especially the ranking, is that we employed a shortened RTDM. Usually, there are multiple ranking rounds until the concordance among the experts has reached a satisfactory level. As mentioned above, we decided not to conduct multiple ranking rounds due to the exploratory nature of the study and the extensive interview process, which makes it difficult to be certain in the ranking. Preliminary statistical analysis of the rating differences between the items shows that there is no significant difference between the listed risk items, but there is a significant difference between the ratings of the different feature items. However, given the qualitative nature of the study the rankings should only be seen as rough guidance. The focus of the work is the qualitative elicitation of items. This has been taken into consideration for the discussion and conclusion, meaning that we took all the items into account, not just the most highly ranked. limitations Another possible threat to the validity of the results is that the participation in the validation round dropped from ten to five. We tried to mitigate validity issues by ensuring higher participation in the ranking round (eight experts participated) and giving study participants the option to also give feedback on the coding in that round. Since no more remarks were made about the coding, we conclude that the coding was satisfactory for all eight study participants of the ranking round. Furthermore, all study participants are employed by the same company. We tried to attenuate this by selecting experts with extensive experience from various teams and backgrounds. Additionally, the consultants brought in their experience with business process improvement projects from additional companies. 6 Conclusion The main aim of this study was to obtain practitioners’ perspectives on the ABBPM method. Using mostly qualitative research methods, we shed light on the requirements for the further development of AB-BPM tools and the underlying method. Overall, the study participants perceived the methodology as advantageous in comparison to the status quo. The three main conclusions of the study are: i) more possibilities of human intervention, and interaction between the RL agent and the human expert, are a core requirement,; ii) transparency and features for the participation of process participants are needed to make AB-BPM culturally viable; iii) integrated process execution is necessary to facilitate the seamless deployment of parallel process variants and deliver the real-time data needed for dynamic RL and routing. The openness of the semi-structured interviews facilitated the discovery of future research opportunities, e.g., studying companies carrying out unstructured process tests in a production environment, and tool-driven training of process participants. While we focused on a single process improvement method, practitioners’ insights on different kinds of business improvement methods [15] could provide insights for studies that compare AB-BPM to other methodologies. i ii iii References Aguilar-Sav´en, R.S.: Business process modelling: Review and framework. International Journal of Production Economics 90(2), 129–149 (Jul 2004) Bass, L., Weber, I., Zhu, L.: DevOps: A Software Architect’s Perspective. AddisonWesley, 1st edition edn. (May 2015) Brinkmann, S., Kvale, S.: InterViews: Learning the Craft of Qualitative Research Interviewing. SAGE Publications, Inc, Los Angeles, third edition edn. (Apr 2014) Chun Tie, Y., Birks, M., Francis, K.: Grounded theory research: A design framework for novice researchers. SAGE Open Medicine 7, 205031211882292 (Jan 2019) Dalkey, N., Helmer, O.: An Experimental Application of the DELPHI Method to the Use of Experts. Management Science 9(3), 458–467 (Apr 1963), publisher: INFORMS Dumas, M., La Rosa, M., Mendling, J., Reijers, H.A.: Fundamentals of Business Process Management. Springer Berlin Heidelberg, Berlin, Heidelberg (2018) Holland, C.W., Cochran, D.: Breakthrough Business Results With MVT: A Fast, Cost-Free, ”Secret Weapon” for Boosting Sales, Cutting Expenses, and Improving Any Business Process. Wiley, Hoboken, NJ, 1st edition edn. (Apr 2005) Ivanov, S.Y., Kalenkova, A.A.: BPMNDiffViz: A Tool for BPMN Models Comparison. Proceedings of the Demo Session of the 13th International Conference on Business Process Management p. 5 (2015) Kampik, T., Weske, M.: Event log generation: An industry perspective. In: Augusto, A., Gill, A., Bork, D., Nurcan, S., Reinhartz-Berger, I., Schmidt, R. (eds.) Enterprise, Business-Process and Information Systems Modeling. pp. 123–136. Springer International Publishing, Cham (2022) Kang, B., Kim, D., Kang, S.H.: Real-time business process monitoring method for prediction of abnormal termination using KNNI-based LOF prediction. Expert Systems with Applications 39(5), 6061–6068 (Apr 2012) Kohavi, R., Longbotham, R.: Online Controlled Experiments and A/B Testing. In: Sammut, C., Webb, G.I. (eds.) Encyclopedia of Machine Learning and Data Mining, pp. 922–929. Springer US, Boston, MA (2017) Kohavi, R., Longbotham, R., Sommerfield, D., Henne, R.M.: Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery 18(1), 140–181 (Feb 2009) Kurz, A.F., Santelmann, B., Großmann, T., Kampik, T., Pufahl, L., Weber, I.: HITL-AB-BPM: Business Process Improvement with AB Testing and Human-inthe-Loop. Proceedings of the Demo Session of the 20th International Conference on Business Process Management (2022) La Rosa, M., Reijers, H.A., Van Der Aalst, W.M., Dijkman, R.M., Mendling, J., Dumas, M., Garcia-Banuelos, L.: Apromore: An advanced process model repository. Expert Systems with Applications 38(6), 7029–7040 (2011) Malinova, M., Gross, S., Mendling, J.: A study into the contingencies of process improvement methods. Information Systems 104, 101880 (2022) Okoli, C., Pawlowski, S.D.: The Delphi method as a research tool: an example, design considerations and applications. Information & Management 42(1), 15–29 (Dec 2004) Par´e, G., Cameron, A.F., Poba-Nzaou, P., Templier, M.: A systematic assessment of rigor in information systems ranking-type Delphi studies. Information & Management 50(5), 207–217 (Jul 2013) Pedrinaci, C., Lambert, D., Wetzstein, B., van Lessen, T., Cekov, L., Dimitrov, M.: SENTINEL: a semantic business process monitoring tool. In: Proceedings of the first international workshop on Ontology-supported business intelligence. pp. 1–12. OBI ’08, Association for Computing Machinery, New York, NY, USA (Oct 2008) Pietsch, P., Wenzel, S.: Comparison of BPMN2 Diagrams. In: Mendling, J., Weidlich, M. (eds.) Business Process Model and Notation. pp. 83 97. Lecture Notes in Business Information Processing, Springer, Berlin, Heidelberg (2012) Renn, O.: Concepts of Risk: An Interdisciplinary Review Part 1: Disciplinary Risk Concepts. GAIA - Ecological Perspectives for Science and Society 17(1), 50–66 (Mar 2008) Robson, C.: Real world research: a resource for social scientists and practitioner researchers. Blackwell, Oxford u.a., reprint. edn. (1999) von Rosing, M., White, S., Cummins, F., de Man, H.: Business Process Model and Notation—BPMN. In: von Rosing, M., Scheer, A.W., von Scheel, H. (eds.) The Complete Business Process Handbook, pp. 433–457. Morgan Kaufmann, Boston (Jan 2015) Sato, D.M.V., De Freitas, S.C., Barddal, J.P., Scalabrin, E.E.: A Survey on Concept Drift in Process Mining. ACM Computing Surveys 54(9), 189:1–189:38 (Oct 2021) Satyal, S., Weber, I., young Paik, H., Di Ciccio, C., Mendling, J.: Business process improvement with the ab-bpm methodology. Information Systems 84, 283–298 (2019) Stiehl, V., Danei, M., Elliott, J., Heiler, M., Kerwien, T.: Effectively and Efficiently Implementing Complex Business Processes: A Case Study. In: Lubke, D., Pautasso, C. (eds.) Empirical Studies on the Development of Executable Business Processes, pp. 33–57. Springer International Publishing, Cham (2019) Sutton, R.S., Barto, A.G.: Reinforcement Learning, second edition: An Introduction. MIT Press (Nov 2018) Tsui, T.C.: Experience from the Anti-Monopoly Law Decision in China (Cost and Benefit of Rule of Law) (Apr 2013 Vassiliadis, P.: A survey of extract–transform–load technology. International Journal of Data Warehousing and Mining (IJDWM) 5(3), 1–27 (2009) Vom Brocke, J., Zelt, S., Schmiedel, T.: On the role of context in business process management. International Journal of Information Management 36(3), 486–495 (Jun 2016) Yan, Z., Dijkman, R., Grefen, P.: Business process model repositories – Framework and survey. Information and Software Technology p. 16 (2012) Aguilar-Sav´en, R.S.: Business process modelling: Review and framework. International Journal of Production Economics 90(2), 129–149 (Jul 2004) Aguilar-Sav´en, R.S.: Business process modelling: Review and framework. International Journal of Production Economics 90(2), 129–149 (Jul 2004) Bass, L., Weber, I., Zhu, L.: DevOps: A Software Architect’s Perspective. AddisonWesley, 1st edition edn. (May 2015) Bass, L., Weber, I., Zhu, L.: DevOps: A Software Architect’s Perspective. AddisonWesley, 1st edition edn. (May 2015) Brinkmann, S., Kvale, S.: InterViews: Learning the Craft of Qualitative Research Interviewing. SAGE Publications, Inc, Los Angeles, third edition edn. (Apr 2014) Brinkmann, S., Kvale, S.: InterViews: Learning the Craft of Qualitative Research Interviewing. SAGE Publications, Inc, Los Angeles, third edition edn. (Apr 2014) Chun Tie, Y., Birks, M., Francis, K.: Grounded theory research: A design framework for novice researchers. SAGE Open Medicine 7, 205031211882292 (Jan 2019) Chun Tie, Y., Birks, M., Francis, K.: Grounded theory research: A design framework for novice researchers. SAGE Open Medicine 7, 205031211882292 (Jan 2019) Dalkey, N., Helmer, O.: An Experimental Application of the DELPHI Method to the Use of Experts. Management Science 9(3), 458–467 (Apr 1963), publisher: INFORMS Dalkey, N., Helmer, O.: An Experimental Application of the DELPHI Method to the Use of Experts. Management Science 9(3), 458–467 (Apr 1963), publisher: INFORMS Dumas, M., La Rosa, M., Mendling, J., Reijers, H.A.: Fundamentals of Business Process Management. Springer Berlin Heidelberg, Berlin, Heidelberg (2018) Dumas, M., La Rosa, M., Mendling, J., Reijers, H.A.: Fundamentals of Business Process Management. Springer Berlin Heidelberg, Berlin, Heidelberg (2018) Holland, C.W., Cochran, D.: Breakthrough Business Results With MVT: A Fast, Cost-Free, ”Secret Weapon” for Boosting Sales, Cutting Expenses, and Improving Any Business Process. Wiley, Hoboken, NJ, 1st edition edn. (Apr 2005) Holland, C.W., Cochran, D.: Breakthrough Business Results With MVT: A Fast, Cost-Free, ”Secret Weapon” for Boosting Sales, Cutting Expenses, and Improving Any Business Process. Wiley, Hoboken, NJ, 1st edition edn. (Apr 2005) Ivanov, S.Y., Kalenkova, A.A.: BPMNDiffViz: A Tool for BPMN Models Comparison. Proceedings of the Demo Session of the 13th International Conference on Business Process Management p. 5 (2015) Ivanov, S.Y., Kalenkova, A.A.: BPMNDiffViz: A Tool for BPMN Models Comparison. Proceedings of the Demo Session of the 13th International Conference on Business Process Management p. 5 (2015) Kampik, T., Weske, M.: Event log generation: An industry perspective. In: Augusto, A., Gill, A., Bork, D., Nurcan, S., Reinhartz-Berger, I., Schmidt, R. (eds.) Enterprise, Business-Process and Information Systems Modeling. pp. 123–136. Springer International Publishing, Cham (2022) Kampik, T., Weske, M.: Event log generation: An industry perspective. In: Augusto, A., Gill, A., Bork, D., Nurcan, S., Reinhartz-Berger, I., Schmidt, R. (eds.) Enterprise, Business-Process and Information Systems Modeling. pp. 123–136. Springer International Publishing, Cham (2022) Kang, B., Kim, D., Kang, S.H.: Real-time business process monitoring method for prediction of abnormal termination using KNNI-based LOF prediction. Expert Systems with Applications 39(5), 6061–6068 (Apr 2012) Kang, B., Kim, D., Kang, S.H.: Real-time business process monitoring method for prediction of abnormal termination using KNNI-based LOF prediction. Expert Systems with Applications 39(5), 6061–6068 (Apr 2012) Kohavi, R., Longbotham, R.: Online Controlled Experiments and A/B Testing. In: Sammut, C., Webb, G.I. (eds.) Encyclopedia of Machine Learning and Data Mining, pp. 922–929. Springer US, Boston, MA (2017) Kohavi, R., Longbotham, R.: Online Controlled Experiments and A/B Testing. In: Sammut, C., Webb, G.I. (eds.) Encyclopedia of Machine Learning and Data Mining, pp. 922–929. Springer US, Boston, MA (2017) Kohavi, R., Longbotham, R., Sommerfield, D., Henne, R.M.: Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery 18(1), 140–181 (Feb 2009) Kohavi, R., Longbotham, R., Sommerfield, D., Henne, R.M.: Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery 18(1), 140–181 (Feb 2009) Kurz, A.F., Santelmann, B., Großmann, T., Kampik, T., Pufahl, L., Weber, I.: HITL-AB-BPM: Business Process Improvement with AB Testing and Human-inthe-Loop. Proceedings of the Demo Session of the 20th International Conference on Business Process Management (2022) Kurz, A.F., Santelmann, B., Großmann, T., Kampik, T., Pufahl, L., Weber, I.: HITL-AB-BPM: Business Process Improvement with AB Testing and Human-inthe-Loop. Proceedings of the Demo Session of the 20th International Conference on Business Process Management (2022) La Rosa, M., Reijers, H.A., Van Der Aalst, W.M., Dijkman, R.M., Mendling, J., Dumas, M., Garcia-Banuelos, L.: Apromore: An advanced process model repository. Expert Systems with Applications 38(6), 7029–7040 (2011) La Rosa, M., Reijers, H.A., Van Der Aalst, W.M., Dijkman, R.M., Mendling, J., Dumas, M., Garcia-Banuelos, L.: Apromore: An advanced process model repository. Expert Systems with Applications 38(6), 7029–7040 (2011) Malinova, M., Gross, S., Mendling, J.: A study into the contingencies of process improvement methods. Information Systems 104, 101880 (2022) Malinova, M., Gross, S., Mendling, J.: A study into the contingencies of process improvement methods. Information Systems 104, 101880 (2022) Okoli, C., Pawlowski, S.D.: The Delphi method as a research tool: an example, design considerations and applications. Information & Management 42(1), 15–29 (Dec 2004) Okoli, C., Pawlowski, S.D.: The Delphi method as a research tool: an example, design considerations and applications. Information & Management 42(1), 15–29 (Dec 2004) Par´e, G., Cameron, A.F., Poba-Nzaou, P., Templier, M.: A systematic assessment of rigor in information systems ranking-type Delphi studies. Information & Management 50(5), 207–217 (Jul 2013) Par´e, G., Cameron, A.F., Poba-Nzaou, P., Templier, M.: A systematic assessment of rigor in information systems ranking-type Delphi studies. Information & Management 50(5), 207–217 (Jul 2013) Pedrinaci, C., Lambert, D., Wetzstein, B., van Lessen, T., Cekov, L., Dimitrov, M.: SENTINEL: a semantic business process monitoring tool. In: Proceedings of the first international workshop on Ontology-supported business intelligence. pp. 1–12. OBI ’08, Association for Computing Machinery, New York, NY, USA (Oct 2008) Pedrinaci, C., Lambert, D., Wetzstein, B., van Lessen, T., Cekov, L., Dimitrov, M.: SENTINEL: a semantic business process monitoring tool. In: Proceedings of the first international workshop on Ontology-supported business intelligence. pp. 1–12. OBI ’08, Association for Computing Machinery, New York, NY, USA (Oct 2008) Pietsch, P., Wenzel, S.: Comparison of BPMN2 Diagrams. In: Mendling, J., Weidlich, M. (eds.) Business Process Model and Notation. pp. 83 97. Lecture Notes in Business Information Processing, Springer, Berlin, Heidelberg (2012) Pietsch, P., Wenzel, S.: Comparison of BPMN2 Diagrams. In: Mendling, J., Weidlich, M. (eds.) Business Process Model and Notation. pp. 83 97. Lecture Notes in Business Information Processing, Springer, Berlin, Heidelberg (2012) Renn, O.: Concepts of Risk: An Interdisciplinary Review Part 1: Disciplinary Risk Concepts. GAIA - Ecological Perspectives for Science and Society 17(1), 50–66 (Mar 2008) Renn, O.: Concepts of Risk: An Interdisciplinary Review Part 1: Disciplinary Risk Concepts. GAIA - Ecological Perspectives for Science and Society 17(1), 50–66 (Mar 2008) Robson, C.: Real world research: a resource for social scientists and practitioner researchers. Blackwell, Oxford u.a., reprint. edn. (1999) Robson, C.: Real world research: a resource for social scientists and practitioner researchers. Blackwell, Oxford u.a., reprint. edn. (1999) von Rosing, M., White, S., Cummins, F., de Man, H.: Business Process Model and Notation—BPMN. In: von Rosing, M., Scheer, A.W., von Scheel, H. (eds.) The Complete Business Process Handbook, pp. 433–457. Morgan Kaufmann, Boston (Jan 2015) von Rosing, M., White, S., Cummins, F., de Man, H.: Business Process Model and Notation—BPMN. In: von Rosing, M., Scheer, A.W., von Scheel, H. (eds.) The Complete Business Process Handbook, pp. 433–457. Morgan Kaufmann, Boston (Jan 2015) Sato, D.M.V., De Freitas, S.C., Barddal, J.P., Scalabrin, E.E.: A Survey on Concept Drift in Process Mining. ACM Computing Surveys 54(9), 189:1–189:38 (Oct 2021) Sato, D.M.V., De Freitas, S.C., Barddal, J.P., Scalabrin, E.E.: A Survey on Concept Drift in Process Mining. ACM Computing Surveys 54(9), 189:1–189:38 (Oct 2021) Satyal, S., Weber, I., young Paik, H., Di Ciccio, C., Mendling, J.: Business process improvement with the ab-bpm methodology. Information Systems 84, 283–298 (2019) Satyal, S., Weber, I., young Paik, H., Di Ciccio, C., Mendling, J.: Business process improvement with the ab-bpm methodology. Information Systems 84, 283–298 (2019) Stiehl, V., Danei, M., Elliott, J., Heiler, M., Kerwien, T.: Effectively and Efficiently Implementing Complex Business Processes: A Case Study. In: Lubke, D., Pautasso, C. (eds.) Empirical Studies on the Development of Executable Business Processes, pp. 33–57. Springer International Publishing, Cham (2019) Stiehl, V., Danei, M., Elliott, J., Heiler, M., Kerwien, T.: Effectively and Efficiently Implementing Complex Business Processes: A Case Study. In: Lubke, D., Pautasso, C. (eds.) Empirical Studies on the Development of Executable Business Processes, pp. 33–57. Springer International Publishing, Cham (2019) Sutton, R.S., Barto, A.G.: Reinforcement Learning, second edition: An Introduction. MIT Press (Nov 2018) Sutton, R.S., Barto, A.G.: Reinforcement Learning, second edition: An Introduction. MIT Press (Nov 2018) Tsui, T.C.: Experience from the Anti-Monopoly Law Decision in China (Cost and Benefit of Rule of Law) (Apr 2013 Tsui, T.C.: Experience from the Anti-Monopoly Law Decision in China (Cost and Benefit of Rule of Law) (Apr 2013 Vassiliadis, P.: A survey of extract–transform–load technology. International Journal of Data Warehousing and Mining (IJDWM) 5(3), 1–27 (2009) Vassiliadis, P.: A survey of extract–transform–load technology. International Journal of Data Warehousing and Mining (IJDWM) 5(3), 1–27 (2009) Vom Brocke, J., Zelt, S., Schmiedel, T.: On the role of context in business process management. International Journal of Information Management 36(3), 486–495 (Jun 2016) Vom Brocke, J., Zelt, S., Schmiedel, T.: On the role of context in business process management. International Journal of Information Management 36(3), 486–495 (Jun 2016) Yan, Z., Dijkman, R., Grefen, P.: Business process model repositories – Framework and survey. Information and Software Technology p. 16 (2012) Yan, Z., Dijkman, R., Grefen, P.: Business process model repositories – Framework and survey. Information and Software Technology p. 16 (2012) Authors: (1) Aaron Friedrich Kurz [0000 −0002 −2547 −6780], Technische Universitat Berlin, Berlin, Germany and SAP Signavio, Berlin, Germany (aaron.kurz@sap.com); (2) Timotheus Kampik, SAP Signavio, Berlin, Germany (timotheus.kampik@sap.com); (3) Luise Pufahl, Technische Universitat Munchen, Munich, Germany (luise.pufahl@tum.de); (4) Ingo Weber [0000 −0002 −4833 −5921], Technische Universitat Munchen, Munich, Germany and Fraunhofer Gesellschaft, Munich, Germany (ingo.weber@tum.de). Authors: Authors: (1) Aaron Friedrich Kurz [0000 −0002 −2547 −6780], Technische Universitat Berlin, Berlin, Germany and SAP Signavio, Berlin, Germany (aaron.kurz@sap.com); (2) Timotheus Kampik, SAP Signavio, Berlin, Germany (timotheus.kampik@sap.com); (3) Luise Pufahl, Technische Universitat Munchen, Munich, Germany (luise.pufahl@tum.de); (4) Ingo Weber [0000 −0002 −4833 −5921], Technische Universitat Munchen, Munich, Germany and Fraunhofer Gesellschaft, Munich, Germany (ingo.weber@tum.de). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv
