MASC: A Tool for Mutation-Based Evaluation of Static Crypto-API Misuse Detectors

Table of Links

Abstract and 1 Introduction

2 Overview of MASC

3 Design Goals

4 Implementation of MASC

4.1 Mutation Operators

4.2 Mutation Scopes

5 Using MASC

6 Future Work and Conclusion, Acknowledgments, and References

ABSTRACT

While software engineers are optimistically adopting crypto-API misuse detectors (or crypto-detectors) in their software development cycles, this momentum must be accompanied by a rigorous understanding of crypto-detectors’ effectiveness at finding cryptoAPI misuses in practice. This demo paper presents the technical details and usage scenarios of our tool, namely Mutation Analysis for evaluating Static Crypto-API misuse detectors (MASC). We developed 12 generalizable, usage based mutation operators and three mutation scopes, namely Main Scope, Similarity Scope, and Exhaustive Scope, which can be used to expressively instantiate compilable variants of the crypto-API misuse cases. Using MASC, we evaluated nine major crypto-detectors, and discovered 19 unique, undocumented flaws. We designed MASC to be configurable and user-friendly; a user can configure the parameters to change the nature of generated mutations. Furthermore, MASC comes with both Command Line Interface and Web-based front-end, making it practical for users of different levels of expertise.

1 INTRODUCTION

Software engineers have been relying on crypto-detectors for decades to ensure the correct use of cryptographic APIs in the software and services they create, develop, and maintain [6]. Such cryptodetectors are ubiquitous in software engineering, as they are integrated into IDEs (e.g., CogniCrypt plugin for Eclipse IDE [8]), testing suite of organizations such as Oracle Corporation [9, 17], and for Continuous Integration/Continuous Deployment (CI/CD) pipelines [12, 18]. In addition, hosting providers such as GitHub are formally provisioning such crypto-detectors e.g., GitHub Code Scan Initiative [10]. In other words, the security of software and services are increasingly becoming more reliant on crypto-detectors. However, we have been relying on manually-curated benchmarks for evaluating the performance of crypto-detectors, such benchmarks are known to be incomplete, incorrect, and impractical to maintain [16]. Therefore, determining the effectiveness of cryptodetectors from a security-focused perspective requires a reliable and evolving evaluation technique that can scale with the volume and diversity of crypto-API and the different patterns of misuse.

We contextualized mutation testing techniques to create the Mutation Analysis for evaluating Static Crypto-API misuse detectors (MASC) framework. In our original, prototype implementation of MASC [3], it internally leveraged 12 generalizable, usage-based mutation operators to instantiate mutations of crypto-API misuse cases for Java. The mutation operators were designed based on the design principles of Java Cryptographic Architecture (JCA) [11] and a threat model that consisted of users of varying skills and intentions (Section 4.1). MASC injects these mutated misuse cases in Java or Android-based apps at three mutation scopes (injection sites), namely Similarity Scope (extended from MDroid+ [13, 14]), Exhaustive Scope (extended from 𝜇SE [4, 5, 7]), and its independently developed Main Scope, thus creating mutated applications that contain crypto-API misuse. We demonstrated the practicality of prototype implementation of MASC by evaluating nine cryptodetectors from industry and academia, and discovered 19 previously undocumented, unknown flaws that compromise the within-scope soundness of crypto-detectors. The full details of MASC’s methodology, design considerations, evaluation of crypto-detectors leading to finding novel flaws, practical impact of found flaws in open source applications (therefore, the applicability of the mutation operators), and discussion of the findings are available in the original research paper [3].

In this paper, we present a mature implementation of MASC framework with focus on extensibility, ease of use, and maintainability to the stakeholders of crypto-detectors, such as security researchers, developers, and users. To elaborate, because of the newly developed plug-in architecture, MASC users can now create their own mutation operators that can be easily plugged into MASC, without diving deep into the existing code base (11𝐾+ source lines of code). Moreover, whereas the original prototype implementation of MASC involved semi-automated evaluation of crypto-detectors, we made MASC’s workflow automated by leveraging the de-facto SARIF [15] formatted output of crypto-detectors. Furthermore, we have created a web-based front-end of MASC’s implementation for the users to reduce the barrier to entry. Finally, we restructured and refactored the open-source codebase of MASC to increase maintainability and extensibility of MASC, which will make future contributions and enhancements easier for both developers and open-source enthusiasts of MASC. With these additions and enhancements, we hope that the current, open-source implementation of MASC will be used in finding flaws in, and thus helping to improve, existing crypto-detectors.

Contribution: We present MASC, a user-friendly framework that leverages mutation-testing techniques for evaluating crypto-detectors, with details of underlying techniques, design considerations, and improvements. The new, key features of MASC are as follows: Automated Evaluation of Crypto-detectors: MASC can be used to evaluate crypto-detectors in an end-to-end automated workflow within the Main Scope.

Customizable Evaluation of Crypto-detectors: A user can customize the evaluation of crypto-detectors by specifying the mutation operators for creating crypto-API misuse instances.

Plug-in Architecture for Custom Operators: MASC helps security researchers, developers and users, jump right into evaluating crypto-detectors by creating their own, custom mutation operators that can be directly plugged-in the Main Scope, without requiring them to learn and understand about the internal details of MASC.

User-friendly Front-end for End-users: In addition to enhancing the command line interface of the original prototype implementation, we create and introduce an open-sourced, web-based front-end for end-users that can be run locally. The front-end contains an additional play-test-learn interface, MASC Lab, where stakeholders can interact with mutation operators and can learn about mutating crypto-API misuse.

Tool and Data Availability: The prototype implementation of the MASC framework, scripts and results of evaluating crypto-detectors,

as described in the original paper [3], are available in the MASC Artifact [1]. Furthermore, the codebase of actively maintained, mature implementation of MASC is available separately with extensive documentation and examples [2].