System Interoperability Types: A Tertiary Study: The Research Method That We Used and How We Used It

Table of Links

Introduction

Related Work

Research Method

Results

Discussions

Conclusions & References

3 RESEARCH METHOD

We performed the three-phase process — planning, execution, and results synthesis [50] — as presented in Figure 1. Sections 3.1 and 3.2 summarize the planning and execution phases respectively, while Section 4 presents the results.

3.1 Planning

The main elements of the tertiary study protocol are the study goal, research questions (RQs), search string, selected databases, and studies selection criteria. Concerning the goal, we used GQM (Goal Question Metric) to define it. The goal is to analyze secondary studies of SIS interoperability for the purpose of identifying the panorama with respect to interoperability types, models, frameworks, domains, and solutions from the point of view of researchers in the context of academic literature. We also defined three RQs, as listed below:

• RQ1: Which interoperability types have been addressed? Rationale: Several interoperability types exist, but without an updated and complete panorama of them or a compilation of their definitions/understandings or classification

• RQ2: Which are the existing interoperability models and frameworks? Rationale: Interoperability models and frameworks can show how interoperability types can be organized and related to each other. Besides, a complete view of these models and frameworks is missing.

• RQ3: Which are the domains and their related solutions for interoperability? Rationale: Interoperability solutions can support the interoperability achievement in SIS in those domains (i.e., the contexts where interoperability is addressed by the studies).

After several calibrations through pilot studies, the resulting search string was: (“interoperability” OR “interoperabiliting” OR “interoperable”) AND (“review of studies” OR “structured review” OR “literature review” OR “literature analysis” OR “in-depth survey” OR “literature survey” OR “meta-analysis” OR "systematic mapping"). We also selected the most recommended publications databases for conducting literature reviews: Scopus[8] , ACM DL, and IEEE Xplore. For the studies selection, we defined two inclusion criteria (IC) and six exclusion criteria (EC):

• IC1: Study addresses interoperability in SIS.

• IC2: Study addresses interoperability types.

• EC1: Study does not address interoperability in SIS.

• EC2: Study does not address interoperability types.

• EC3: Study is not a literature review.

• EC4: The full text of the study is not available.

• EC5: Study is an older version of another study.

• EC6: Study is not written in English.

We also applied the quality assessment of studies due to our tertiary study intends to present a panorama of all secondary studies, and such assessment can drawn an overall view of the quality of these studies. For this, we followed the recommendation of Kitchenham et al. [49] by using DARE-4 criteria[9] ; hence, the questions for the quality assessment (QA) were:

• QA1: Were the review’s selection criteria (inclusion and exclusion criteria) explicitly defined?

• QA2: Was the literature search likely to have covered all relevant studies?

• QA3: Did the reviewers assess the quality of the included studies?

• QA4: Was the basic studies’ information adequately described?

Each study was scored as follows: QA1 (Yes: the selection criteria were explicitly defined; Partly: the selection criteria were implicit; or No: the selection criteria were not defined and were not possible to be inferred); QA2 (Yes: the authors either searched in four or more databases and included additional search strategies or identified and referenced all journals addressing the topic of interest; Partly: the authors searched in three or four databases with no extra search strategies, or searched a defined but restricted set of journals and conference proceedings; or No: the authors searched up to two databases or a highly restricted set of journals); QA3 (Yes: the authors explicitly defined quality criteria and applied them to each primary study; Partly: quality criteria and application procedure were implicit; or No: no explicit quality assessment of primary studies was presented); and QA4 (Yes: information about each study was presented; Partly: only summarized information was presented; or No: no information was presented). Moreover, the numerical scores were Yes = 1, Partly = 0.5, No = 0, or Unknown (the information is not specified).

3.2 Execution and Update

We searched for studies in the databases in May 2022, following the protocol rigorously. To do that, we configured the search string to each database’s search engine and retrieved studies based on titles, keywords, and abstracts. We focused on studies published in the last ten years (2012 to 2022), similar to most tertiary studies in the literature. We retrieved a total of 494 studies from the databases and after duplicate removal, 414 unique studies remained. During the studies selection process, we deeply examined each study, applied the selection criteria, and selected 35 studies that were relevant to answer our RQs. Next, we applied the quality assessment; for that, the two first authors evaluated the studies separately, and the third and fourth ones came together and solved divergences.

As we finished the deep analysis of each study and results summarization in December 2022, we decided to update our tertiary study considering the period of 2022 until February 2023, as illustrated in Figure 1. We identified two new studies published in 2023. Hence, our tertiary studies selected a total of 37 studies published from January 2012 to February 2023.

[8] https://www.scopus.com

[9] https://web.archive.org/web/20070918200401/https://www.york.ac.uk/inst/crd/faq4.htm