Table of Links
IV. Systematic Security Vulnerability Discovery of Code Generation Models
VII. Conclusion, Acknowledgments, and References
Appendix
A. Details of Code Language Models
B. Finding Security Vulnerabilities in GitHub Copilot
C. Other Baselines Using ChatGPT
D. Effect of Different Number of Few-shot Examples
E. Effectiveness in Generating Specific Vulnerabilities for C Codes
F. Security Vulnerability Results after Fuzzy Code Deduplication
G. Detailed Results of Transferability of the Generated Nonsecure Prompts
H. Details of Generating non-secure prompts Dataset
I. Detailed Results of Evaluating CodeLMs using Non-secure Dataset
J. Effect of Sampling Temperature
K. Effectiveness of the Model Inversion Scheme in Reconstructing the Vulnerable Codes
L. Qualitative Examples Generated by CodeGen and ChatGPT
M. Qualitative Examples Generated by GitHub Copilot
L. Qualitative Examples Generated by CodeGen and ChatGPT
Listing 8 and Listing 9 provide two examples of vulnerable Python codes generated by ChatGPT. Listing 8 shows a Python code example that contains a security vulnerability of type CWE-022 (Path traversal). Listing 9 provides a Python code example with a vulnerability of type CWE-089 (SQL injection). In Listing 8, the first eight lines are the non-secure prompt, and the rest of the code example is the completion for the given non-secure prompt. The code contains a path traversal vulnerability in line 23. In Listing 9, the first eight lines are the non-secure prompt, and the rest of the code example is the completion for the given non-secure prompt. The code in Listing 9 contains an SQL injection vulnerability in line 22.
Listing 10 and Listing 11 provide two examples of vulnerable C codes generated by CodeGen. Listing 10 and Listing 11 provide C code with multiple vulnerabilities of type CWE787 (out-of-bounds write). In Listing 10, lines 1 to 7 are the non-secure prompt, and the rest of the code example is the completion for the given non-secure prompt. The code contains a vulnerability of type CWE-787 in line 25. In Listing 11, the first nine lines are the non-secure prompt, and the rest of the code example is the completion for the given non-secure prompt. The code in Listing 11 contains several out-of-bounds write vulnerabilities in lines 10, 11 and 17.
Authors:
(1) Hossein Hajipour, CISPA Helmholtz Center for Information Security ([email protected]);
(2) Keno Hassler, CISPA Helmholtz Center for Information Security ([email protected]);
(3) Thorsten Holz, CISPA Helmholtz Center for Information Security ([email protected]);
(4) Lea Schonherr, CISPA Helmholtz Center for Information Security ([email protected]);
(5) Mario Fritz, CISPA Helmholtz Center for Information Security ([email protected]).
This paper is
