Why We Misjudge Our Own Effectiveness at Finding Software Bugs

Table Of Links Table Of Links Abstract Abstract 1 Introduction 1 Introduction 1 Introduction 2 Original Study: Research Questions and Methodology 2 Original Study: Research Questions and Methodology 2 Original Study: Research Questions and Methodology 3 Original Study: Validity Threats 3 Original Study: Validity Threats 3 Original Study: Validity Threats 4 Original Study: Results 4 Original Study: Results 4 Original Study: Results 5 Replicated Study: Research Questions and Methodology 5 Replicated Study: Research Questions and Methodology 5 Replicated Study: Research Questions and Methodology 6 Replicated Study: Validity Threats 6 Replicated Study: Validity Threats 6 Replicated Study: Validity Threats 7 Replicated Study: Results 7 Replicated Study: Results 7 Replicated Study: Results 8 Discussion 8 Discussion 8 Discussion 9 Related Work 9 Related Work 9 Related Work 10 Conclusions And References 10 Conclusions And References 10 Conclusions And References 2 Original Study: Research Questions And Methodology 2.1 Research Questions 2.1 Research Questions The main goal of the original study is to assess whether participants’ perceptions of their testing effectiveness using different techniques are good predictors of real testing effectiveness. This goal has been translated into the following research question: RQ1: Should participants’ perceptions be used as predictors of testing effectiveness? This question was further decomposed into: – RQ1.1: What are participants’ perceptions of their testing effectiveness? – RQ1.1: What are participants’ perceptions of their testing effectiveness? We want to know if participants perceive a certain technique as most effective than the others. – RQ1.2: Do participants’ perceptions predict their testing effectiveness? – RQ1.2: Do participants’ perceptions predict their testing effectiveness? We want to assess if the technique each participant perceives as most effective is the most effective for him/her. – RQ1.3: Do participants find a similar amount of defects for all techniques? – RQ1.3: Do participants find a similar amount of defects for all techniques? Choosing the most effective technique can be difficult if participants find a similar amount of defects for two or all three techniques. – RQ1.4: What is the cost of any mismatch? – RQ1.4: What is the cost of any mismatch? We want to know if the cost of not correctly perceiving the most effective technique is negligible and depends on the technique perceived as most effective. – RQ1.5: What is expected project loss? – RQ1.5: What is expected project loss? Taking into consideration that some participants will correctly perceive their most effective technique (mismatch cost 0), and others will not (mismatch cost greater than 0), we calculate the overall cost of (mis)match for all participants in the empirical study and check if it depends on the technique perceived as most effective. 2.2 Study Context and Ethics 2.2 Study Context and Ethics We conducted a controlled experiment where each participant applies three defect detection techniques (two testing techniques and one code review technique) on three different programs. For testing techniques, participants report the generated test cases, later run a set of test cases that we have generated (instead of the ones they created), and report the failures found1 . For code reading they report the identified faults. At the end of the controlled experiment, each participant completes a questionnaire containing a question related to his/her perceptions of the effectiveness of the techniques applied. The course is graded based on their technique application performance (this guarantees a thorough application of the techniques). The study is embedded in an elective 6 credits Software Verification and Validation course. The regular assessment (when the experiment does not take place) is as follows: students are asked to write a specification for a program that can be coded in about 8 hours. Specifications are later interchanged so that each student codes a different program from the one (s)he proposed. Later, students are asked to perform individually (in successive weeks) code reading, and white-box testing on the code they wrote. At this point, each student delivers the code to the person who wrote the specification, so that each student performs black-box testing on the program (s)he proposed. Note that this scenario requires more effort from the student (as (s)he is asked to write first a specification and then code a program, and these tasks do not take place when the study is run). In other words, the students workload during the experiment is smaller than the workload of the regular course assessment. The only activity that takes place during the experiment that is not part of the regular course is answering the questionnaire, which can be done in less than 15 minutes. Although the study causes changes in the workflow of the course, its learning goals are not altered. All tasks required by the study, with the exception of completing the questionnaire, take place during the slots assigned to the course. Therefore, there is no additional effort for the students but attending lectures (which is mandatory in any case). Note that the students are allowed to withdraw from the controlled experiment, but this would affect their score in the course. But this also happens when the experiment is not run. If a student misses one assignment, (s)he would score 0 in that assignment and his/her course score would be affected consequently. However, they are allowed to withdraw from the study without penalty in their score, as the submission of the questionnaire is completely voluntary. No incentives are given to those students who submit the questionnaire. The fact of submitting the questionnaire implies giving consent for participating in the study. Students are aware this is a voluntary activity aiming for a research, but they can also get feedback. Those students who do not submit the questionnaire, are not considered in the study in any way, as they are not giving consent to use their data. For this reason, they will not be included in the quantitative analysis of the controlled experiment (even though their data is available for scoring purposes). The study is performed in Spanish, as it is the participants’ mother tongue. Its main characteristics are summarised in Table 1. 2.3 Constructs Operationalization 2.3 Constructs Operationalization Code evaluation technique is an experiment factor, with three treatments (or levels): equivalence partitioning (EP)—see Myers et al. [41], branch testing (BT)—see Beizer [6], and code reading by stepwise abstraction (CR)—see Linger [36]. Code evaluation technique factor The response variables are technique effectiveness, perception of effectiveness and mismatch cost. Technique effectiveness is measured as follows: response variables technique effectiveness, perception of effectiveness and mismatch cost. – For EP and BT, it is the percentage of faults exercised by the set of test cases generated by each participant. In order to measure the response variable, experimenters execute the test cases generated by each participant2 . – For CR, we calculate the percentage of faults correctly reported by each participant (false positives are discarded). Note that dynamic and code review techniques are not directly comparable as they are different technique types (dynamic techniques find failures and code review techniques find faults). However, the comparison is fair, as: – Application time is not taken into account, and participants are given enough time to complete the assigned task. – All faults injected are detectable by all techniques. Further details about faults, failures and their correspondence is given in Section 2.5. Perception of effectiveness is gathered by means of a questionnaire with one question that reads: Using which technique did you detect most defects3? Mismatch cost is measured, for each participant, as the difference between the effectiveness obtained by the participant in the technique (s)he perceives as most effective and the most effective in reality for him/her. Note that participants neither know the total amount of seeded faults, nor which techniques are best for their colleagues or themselves. This operationalization imitates the reality of testers –who lack such knowledge in real projects. Therefore, the perception is fully subjective (and made in relation with the other two techniques). Table 2 shows three examples that show how mismatch cost is measured. Cells in grey background show the technique for which highest effectiveness is observed for the given participant. The first row shows a situation where the participant perceives as most effective CR, but the most effective for him/her is EP. In this situation, there is a mismatch (misperception) and the associated cost is calculated as the difference in effectiveness between CR and EP. The second row shows a situation where the participant correctly perceives EP as the most effective technique for him/her. In this situation there is a match (correct perception) and therefore, the associated mismatch cost is 0pp. The third row shows a situation where the participant perceives BT as the most effective technique for him/her, and BT and EP are tied as his/her most effective technique. In this situation we consider that there is a match (correct perception), and therefore, the associated mismatch cost is 0pp. 2.4 Study Design 2.4 Study Design Testing techniques are applied by human beings, and no two people are the same. Due to the dissimilarities between the participants already existing prior to the experiment (degree of competences achieved in previous courses, innate testing abilities, etc.), there may exist variability between different participants applying the same treatment. Therefore, we opted for a crossover design, as described by Kuehl [34] (a within-subjects design, where each participant applies all three techniques, but different participants apply the techniques in a different order) to prevent dissimilarities between participants and technique application order from having an impact on results. The design of the experiment is shown in Table 3. crossover design The experimental procedure takes place during seven weeks, and is summarised in Table 4. The first three weeks there are training sessions in which participants learn how to apply the techniques and practice with them. Training sessions take place twice a week (Tuesdays and Thursdays) and each one lasts 2 hours. Therefore, training takes 12 hours (2 hours/session x 2 sessions/week x 3 weeks). Participants are first taught the code review technique, then white-box and finally black-box. The training does not follow any particular order, but the one we have found best to meet the learning objectives of the course. experimental procedure The following week there are no lectures, and students are asked to practice with the techniques. For this purpose, they are given 3 small programs in C (that contain faults), and are asked to apply a given technique on each program (all students apply the same technique on the same training program). The performance on these exercises is used for grading purposes. The other three are experiment execution weeks. Each experiment execution session takes place once a week (Fridays) and lasts four hours. This is equivalent to there being no time limit, as participants can complete the task in less time. Therefore, experiment execution takes 12 hours (4 hours/session x 1 session/week x 3 weeks). Training sessions take place during lecture hours and experiment execution sessions take place during laboratory hours. Those weeks in which there are lectures, there is no laboratory and vice versa. The time used for the controlled experiment is the corresponding one assigned to the course in which the study is embedded. No extra time is used. In each session, participants apply the techniques and, for equivalence partitioning and branch testing, run test cases too. They report application of technique, and generated test cases and failures (for the testing techniques) or faults (for the code review technique). At the end of the last experiment execution session (after applying the last technique), participants are surveyed about their perceptions of the techniques that they applied. They must return their answer before the following Monday, to guarantee that they remember as much as possible about the tasks performed. 2.5 Experimental Objects 2.5 Experimental Objects Program is a blocking variable. It is not a factor, because the goal of the experiment is not to study the programs, but the code evaluation techniques. However, it is a blocking variable, because we are aware that programs could be influencing results. The experiment has been designed to cancel out the influence of programs. Every participant applies each technique in a different program, and each technique is applied on different programs (by different participants). Additionally, the program by technique interaction is later analysed. The experiment uses three similar programs, written in C (used in other empirical studies about testing techniques like the ones performed by Kamsties & Lott [29] or Roper et al. [46]): – cmdline: parser that reads the input line and outputs a summary of its contents. It has 239 executable LOC and a cyclomatic complexity of 37. cmdline: – nametbl: implementation of the data structure and operations of a symbol table. It has 230 executable LOC and a cyclomatic complexity of 27. nametbl: – ntree: implementation of the data structure and operations of an n-ary tree. It has 215 executable LOC and a cyclomatic complexity of 31. ntree: Appendix A shows a complete listing of the metrics gathered by the PREST4 tool [32] on the correct programs (before faults were injected). Although the purpose of the programs is different, we can see that most of the metrics obtained by PREST are quite similar, except Halstead metrics, which are greater for ntree. At the same time, cmdline is slightly larger and more complex than the other two. Each program has been seeded with seven faults (some, but not all, are the same faults as used in previous experiments run on these programs), and there are 2 versions of each faulty program. All faults are conceptually the same in all programs (eg., a variable initialisation is missing). Some faults occurred naturally when the programs were coded, whereas others are typical programming faults. All faults: – Cause observable failures. – Can be detected by all techniques. – Are chosen so that the programs fail only on some inputs. – No fault conceals another5 . – There is a one-to-one correspondence between faults and failures. Note, however, that it is possible that a participant generates two (or more) test cases that exercise the same seeded fault, and therefore produce the same failure. Participants have been advised to report these failures (the same failure exercised by two or more different test cases) as a single one. For example, there is a fault in program ntree in the function in charge of printing the tree. This causes the failure that the tree is printed incorrectly. Every time a participant generates a test case that prints the tree (which is quite often, as this function is useful to check the contents of the tree at any time), the failure will be shown. Some examples of the seeded faults and their corresponding failures are: – Variable not initialised. The associated failure is that the number of input files is printed incorrectly in cmdline. – Incorrect boolean expression in a decision. The associated failure is that the program does not output error if the second node of the “are siblings” function does not belong to the tree. 2.6 Participants 2.6 Participants The 32 participants of the original study were fifth-(final)year undergraduate computer science students taking the elective Software Verification and Validation course at the Universidad Polit´ecnica de Madrid. The students have gone through 2 courses on Software Engineering of 6 and 12 credits respectively. They are trained in SE, have strong programming skills, have experience programming in C, have participated in small size development projects6 , and have little or no professional experience. So, they should not been considered unexperienced in programming, but good proxys of junior programmers. participants They have not formal training in any code evaluation techniques (including the ones involved in the study), as this is the course in which they are taught them. Since they have had previous coding assignments, they might have done testing previously but informally. As a consequence, they might have acquired some intuitive knowledge on how to test/review programs (developing their own techniques or procedures that could resemble the techniques), but they have never learned the techniques formally. They have never been required to do peer-reviews in coding assignments, or write test cases in the projects where they have participated. They could possibly have used assertions or informal input validation, but on their own (never under request, and have not previously been taught how to do it). All participants have a homogeneous background. The only differences could be due to the level of achievement of learning goals in previous courses, or innate ability for testing. The former could have been determined by means of scores in previous courses (which was not possible). The latter was not possible to measure. Therefore, we have not deemed necessary to do some kind of blocking, and just performed simple randomisation. Therefore, the sample used represents developers with little or no previous experience on code evaluation techniques (novice testers). The use of our students is appropriate in this study on several grounds: – We want to rule out any possible influence of previous experience on code evaluation techniques. Therefore, participants should not have any preconceived ideas or opinions about the techniques (including having a favourite one). – Falessi et al. [21] suggest that it is easier to induce a particular behaviour among students. More specifically, reinforce a high level of adherence to the treatment by experimental subjects applying the techniques. – Students are used to make predictions during development tasks, as they are continually undergoing assessment on courses related with programming, SE, networking, etc. Having said that, since our participants are not practitioners, their opinions are not based on any previous work experience on testing, but on their experience on informally testing programs for some years (they are in 5th year of a 5-year CS bachelor). Additionally, as part of the V&V training, our participants are asked to practice in small programs with the techniques used in the experiment. According to Falessi et al. [21], we (SE experiments) tend to forget practitioners’ heterogeneity. Practitioners have different academic backgrounds, SE knowledge and professional experience. For example, a developer without a computer science academic background might not have knowledge about testing techniques. We assume that for this exploratory study, the characteristics of the participants are a valid sample for developers that have little or no experience on code evaluation techniques and are junior programmers. 2.7 Data Analysis 2.7 Data Analysis The analyses conducted in response to the research questions, are explained below7 . Table 5 summarises the statistical tests used to answer each research question. First we report the analyses (descriptive statistics and hypothesis testing) of the controlled experiment. To examine participants’ perceptions (RQ1.1), we report the frequency of each technique (percentage of participants that perceive each technique as the most effective). Additionally, we determine whether all three techniques are equally frequently perceived as being the most effective. We test the null hypothesis that the frequency distribution of the perceptions is consistent with a discrete uniform distribution, i.e., all outcomes are equally likely to occur. To do this, we use a chi-square (χ 2 ) goodness-of-fit test. To examine if participants’ perceptions predict their testing effectiveness (RQ1.2), we use Cohen’s kappa coefficient along with its 95% confidence interval—calculated using bootstrap. Cohen’s kappa coefficient (κ) is a statistic that measures agreement for qualitative (categorical) variables when 2 raters are classifying different objects (units). It is calculated on the corresponding contingency table generated. Table 6 shows an example of a contingency table. Cells contain the frequencies associated to each pair of classes. Kappa is generally thought to be a more robust measure than simple percent agreement calculation, since it takes into account the agreement occurring by chance. It is not the only coefficient that can be used to measure agreement. There are others, like Krippendorff’s alpha, which is more flexible, as can be used in situations where there are more than 2 raters, or the response variable is in an interval or ratio scale. However, in our particular situation, where there are 2 raters, data in nominal scale and no missing data, Kappa behaves similarly to Krippendorff’s alpha [3], [54]. Kappa is a number from -1 to 1. Positive values are interpreted as agreement, while negative values are interpreted as disagreement. There is still some debate about how to interpret kappa. Different authors have categorised detailed ranges of values for kappa that differ with respect to the degree of agreement that they suggest (see Table 7). According to scales by Altman[1] and Landis & Koch [35], 0.6 is the value as of which there is considered to be agreement. Fleiss et al. [22] lower this value to 0.4. Each branch of science should establish its kappa value. As there are no previous studies that specifically address the issue of which is the most appropriate agreement scale and threshold for SE, and different studies in SE have used different scales8 , we use Fleiss et al.’s more generous scale as our baseline. We measure the agreement between the technique perceived as most effective by a participant, and the most effective technique for that participant for all participants. Therefore, we have 2 raters (perceptions and reality), three classes (BT, EP and CR), and as many units to be classified as participants. Since there could be agreement for some but not all techniques, we also measure kappa for each technique separately (kappa per category), following the approach described in [20]. It consists of collapsing the corresponding contingency table. Table 8 shows the collapsed contingency table for Class A from Table 6. Note that a collapsed table is always a 2x2 table. In the event of disagreement, we also study the type of mismatch between perceptions and reality—whether the disagreement leads to some sort of bias in favour of any of the techniques. To do this, we use the respective contingency table to run Stuart-Maxwell’s test of marginal homogeneity (testing the null hypothesis that the distribution of preferences match reality) and the McNemar-Bowker test for symmetry (testing the null hypothesis of symmetry) as explained in [20]. The hypothesis of marginal homogeneity corresponds to equality of row and column marginal probabilities in the corresponding contingency table. The test for symmetry determines whether observations in cells situated symmetrically about the main diagonal have the same probability of occurrence. In a 2x2 table, symmetry and marginal homogeneity are equivalent. In larger tables, symmetry implies marginal homogeneity, but the reciprocal is not true9 . Since we have injected only 7 defects in each program, there exists the possibility that if no agreement is found between perceptions and reality, it could be due to the fact that participants find a similar amount of defects for all three (or pairs of ) techniques (RQ1.3). If this is the case, then it would be difficult for them to choose the most effective technique. To check this, we will run agreement on the effectiveness obtained by participants using different techniques. Therefore we have 3 raters (techniques) and as many units as participants. This will be done with all participants, and with participants in the same experiment group, for every group; for all techniques, and for pairs of techniques. Note that kappa can no longer be used, as we are seeking agreement on interval data. For this reason, we will use Krippendorff’s alpha [26] along with its 95% confidence interval—calculated using bootstrap, and the KALPHA macro for SPSS10 . To examine the mismatch cost (RQ1.4) and project loss (RQ1.5), we report the cost of the mismatch (when it is greater than zero for RQ1.4 and in all cases for RQ1.5), associated with each technique as explained in Section 2.3. To discover whether there is a relationship between the technique perceived as being the most effective and the mismatch cost and project loss, we apply a one-way ANOVA test or a medians Kruskall-Wallis test for normal and nonnormal distributions, respectively along with visual analyses (scatter plots). Authors: Sira Vegas Patricia Riofr´ıo Esperanza Marcos Natalia Juristo Authors: Authors: Sira Vegas Patricia Riofr´ıo Esperanza Marcos Natalia Juristo Sira Vegas Patricia Riofr´ıo Esperanza Marcos Natalia Juristo This paper is available on arxiv under CC BY-NC-ND 4.0 license. This paper is available on arxiv under CC BY-NC-ND 4.0 license. available on arxiv available on arxiv