Effect of Different Number of Few-shot Examples

Table of Links

Abstract and I. Introduction

II. Related Work

III. Technical Background

IV. Systematic Security Vulnerability Discovery of Code Generation Models

V. Experiments

VI. Discussion

VII. Conclusion, Acknowledgments, and References

Appendix

A. Details of Code Language Models

B. Finding Security Vulnerabilities in GitHub Copilot

C. Other Baselines Using ChatGPT

D. Effect of Different Number of Few-shot Examples

E. Effectiveness in Generating Specific Vulnerabilities for C Codes

F. Security Vulnerability Results after Fuzzy Code Deduplication

G. Detailed Results of Transferability of the Generated Nonsecure Prompts

H. Details of Generating non-secure prompts Dataset

I. Detailed Results of Evaluating CodeLMs using Non-secure Dataset

J. Effect of Sampling Temperature

K. Effectiveness of the Model Inversion Scheme in Reconstructing the Vulnerable Codes

L. Qualitative Examples Generated by CodeGen and ChatGPT

M. Qualitative Examples Generated by GitHub Copilot

D. Effect of Different Number of Few-shot Examples

Here we investigate the effect of using a different number of few-shot examples in our FS-Code method. Figure 5 shows the results of the number of generated vulnerable Python codes by ChatGPT using the different number of few-shot examples. In Figure 5, we provide the total number of generated vulnerable Python codes with four different CWEs (CWE-020, CWE-022, CWE-078, and CWE-079) and 125 code samples for each CWE. The result in Figure 5 shows that using more few-shot examples in our FS-Code method leads the model to generate more vulnerable codes. This shows that providing more context of the targeted vulnerability helps our approach to finding more

vulnerable codes in the code generation models. Note that in our experiment in Section V-B, we also used three examples as demonstration examples in the few-shot prompts.