Comparison of Machine Learning Methods: Related Work

Table of Links

• Abstract and Introduction
• Background
• Approach
• Experiments and Results
• Discussion and Qualitative Analysis
• Related Work
• Conclusions and Future Work, and References

6 Related Work

Several studies in the literature have focused on issue classification, which has addressed a variety of objectives, including issue assignment, effort estimation, issue prioritization, and so on. In this section, we briefly give details regarding issue assignment studies in general and all Turkish-language issue classification studies in particular.

Several types of research have been conducted in order to automate the time-consuming task of issue assignment. In 2017, Goyal et al. [18] review and categorize 75 research papers on the automated bug assignment area. They identify seven categories: machine learning [25,47,55,5,8,36], information retrieval [58], auction [24], social network [57], tossing graph [50], fuzzy set [37] and operational research based [26] techniques. They capture the fact that for automatic bug report assignment, machine learning and information retrieval techniques are the most popular ones. In recent years, deep learning algorithms have also been successfully applied in this field, which has recently revolutionized the idea of word sequence representation and demonstrated encouraging advancements in a number of classification tasks [17]. In this section, we restrict our focus to machine learning and deep learning architectures used to train issue assignment systems.

The machine learning algorithms use historical bug reports to build a supervised or unsupervised machine learning classifier, which is then used to choose appropriate developers for new bug reports. Naive Bayes is the most widely used classifier in machine learning-based approaches according to prior studies [56,8,6,30,47], and it has been extensively tested [18] in the bug reports of open-source projects. Most studies use Eclipse [8,5,3,51,2,39] and Mozilla [8,51,39,23] projects to validate their proposals. Machine learning models in most approaches [5,36,51] use only summary and description as textual features of the issues. Jonsson et al. [25] use the combined title and description as textual features and version, type, priority, and submitter columns as nominal features. Sharma et al. [46] consider bug attributes, namely, severity, priority, component, operating system, and the bug assignee.

To estimate the value of terms, most of the approaches [51,25,47,42] in the literature employ term-weighting techniques like Tf-Idf. Jonsson et al. [25] represent textual parts in the bug reports as the 100 words with the highest Tf-Idf. Shokripour et al. [47] use time metadata in Tf-Idf (Time-Tf-Idf). To determine the value of terms in a document and corpus, the Tf-Idf technique only considers their frequency. However, in determining the weight, time-based Tf-Idf considers the time spent using the term in the project. The developer’s recent use of the term is taken into account when determining the value of the developer’s expertise. They rank the developers according to their calculated term expertise, and the first developer on the list is assigned to fix the new bug.

However, prior studies focused on open-source projects only but rarely [25,36] attempted in industrial environments like our study. Jonsson et al. [25] use ensemble learner Stacked Generalization, which is our best method also, that combines several machine learning classifiers on data from the automation and telecommunication company. In their approach, the different classes correspond to the development teams. Oliveira et al. [36] also use the data set of a large electronic company. They create a model that can distribute new issues according to the responsibilities of the teams using a variety of machine learning techniques and the WEKA [21] tool.

To improve prediction accuracy, some researchers use incremental learning methods. Bhattacharya et al. [8] use various machine learning algorithms and achieve the best results using the NB classifier in combination with the product-component features, tossing graphs, and incremental learning in mostly used large projects: Mozilla and Eclipse. Xia et al. [55] offer the multi-feature topic model (MTM), a specialized topic modeling approach that extends Latent Dirichlet Allocation (LDA) for the bug assignment. To map the term space to the subject space, their approach takes into account product and component information from issue reports. Then, they suggest an incremental learning mechanism that uses the topic distribution of a new bug report to assign an appropriate developer based on the reports that the developer has previously fixed.

The deep learning algorithms are attempted first in 2017 [16] for bug report assignment recommendation, to the best of our knowledge. Gupta et al. [19] describe the popular deep learning approaches applied to the domain of bug reports and Recurrent Neural Networks (RNN) and Long Short Term Memory (LSTM) are a few famous approaches being used for the deep learning-based approaches [16,31]. Mani et al. [31] use title and description parts and Florea et al. [16] use the component id, product id, and bug severity fields as one-hot-encoded categorical variables in addition to title, description, and comments to represent the issues. In 2022, Feng et al. [15] use four transformers models BERT and RoBERTa along with their distilled counterparts DistilBERT, DistilRoBERTa in an ensemble using a resolver team, resolver person, and description columns of the issues.

In research using Turkish issue reports, there are limited studies available in a few fields. The reason may be the agglutinative nature of the Turkish language and the absence of a shared data set for Turkish issues in the literature. Aktas et al. [1] classified the issues they gathered from the banking industry among various software development teams. They use the Jira issue reports for their research like our study. They use SVC, CNN, and LSTM models to solve the classification problem, and they represent the summary and description columns of the issue reports as ordered vectors of Tf-Idf scores. The linear SVC classifier offers the best assignment accuracy for their research with a 0.82 score. Koksal et al. [27] present an automated bug classification approach using a commercial proprietary bug data set. They apply several machine learning algorithms and the SVM classifier is the best algorithm with 0.72 accuracy. In 2022, Tunali [52] prioritizes the software development demands of a private insurance company in Turkey. He proposes several deep-learning architectures and a pre-trained transformer model called distilbert-base-turkish-cased based on DistilBERT to achieve the highest accuracy of 0.82.