Table of Links
II. Approach
B. Proof of Concept Implementation
III. Envisioned Usage Scenarios
IV. Experiment Design and Demographics
VII. Conclusions and Future Work, Acknowledgment, and References
IV. EXPERIMENT DESIGN AND DEMOGRAPHICS
Effectiveness is one of the most common properties used to evaluate SV approaches. In that context, Merino et al. [27] present a systematic literature review of SV evaluation. Their work analyzes the literature body of full papers that were published in the SOFTVIS/VISSOFT conferences, resulting in the examination of 181 papers. The authors focus on evaluations that validate the effectiveness of their presented approach. It is mentioned that multiple evaluations omit other variables that can contribute to or generally influence the effectiveness [28], such as recollection and emotions. We share this opinion and argue that we must first evaluate properties such as perceived usefulness, perceived usability, or feature requests to potentially refine a new, exploratory approach. Only afterwards, we should evaluate effectiveness and efficiency with a sufficiently large number of participants in controlled experiments [29]. As a result, we decided to conduct an exploratory user-study first. We designed an experiment in which participants use and evaluate our approach in a taskoriented onboarding process, i.e., in a scenario similar to SC1 (see Section III). In the future, we will also evaluate our approach in other scenarios by using a similar experiment. In this paper however, we developed the experiment with a focus on SC1 due to the approach’s prototype implementation, the exploratory nature of the study, and the duration of a single experiment run. As a result, our research questions (RQ) are not concerned about effectiveness or efficiency. Instead, we focus on several aspects to gather qualitative feedback and quantitative results, such as time spent in the embedded SV, to gain first insights into the use of our approach:
• RQ1: How do subjects use the embedded SV and code editor during task solving?
• RQ2: Is the code editor perceived as more useful than the embedded SV?
• RQ3: Do subjects recognize the usefulness of collaborative SV features for specific tasks?
• RQ4: What is the general perception of the usefulness and usability of the approach?
• RQ5: Is the approach perceived as useful in the envisioned usage scenarios?
We again emphasize that the findings of this contribution should be seen as first insights and indicators for refinements rather than statistically grounded results. However, by answering the research question, we can derive the following main contributions of our evaluation:
• Further insights regarding the perceived usefulness of software cities to comprehend runtime behavior.
• First quantitative and qualitative results regarding the perceived usefulness, perceived usability, and usage time for collaborative, code-proximal software cities.
• A supplementary package containing the evaluation’s raw results, screen recordings of all participants, and detailed instructions as well as software packages for reproduction [30].
In the following, we now present the participants’ demography and our experiment’s procedure.
Authors:
(1) Alexander Krause-Glau, Software Engineering Group, Kiel University, Kiel, Germany ([email protected]);
(2) Wilhelm Hasselbring, Software Engineering Group, Kiel University, Kiel, Germany ([email protected]).
This paper is
