Step-by-Step Vulnerability Detection Using Large Language Models

Step-by-Step Vulnerability Detection using Large Language Models Motivation Vulnerability detection is a very critical task for systems security. Current analysis techniques suffer from the trade-off between coverage and accuracy. ML-based* analysis tools are non-robust, black-box and unreliable to use in real-world [1]. LLMs* demonstrate revolutionizing capabilities for programming language-related tasks but they are also studied in a black-box fashion for both vulnerability detection and its repair. Security experts follow a step-by-step approach for vulnerability detection. Can using the same approach help LLMs performing better at the vulnerability detection task? Visualizing the Process of Vulnerability Detection In this experiment, we study the behavior of an LLM when it is asked to detect a vulnerability in two different scenarios. First, when it is asked to give a direct answer (Figure 3a); and second, when it is first asked to perform human-expert like reasoning and then make a decision (Figure 3b). Figure 3 shows that step-by-step reasoning guides the LLM to detect the (CWE-787) vulnerability. We choose GPT-3.5 as an LLM and a code snippet (shown in Figure 2) containing an out-ofbound write vulnerability as a running example. Figure 2. Code Snippet from the MITRE Out-of-Bound We use the ‘gpt-3.5-turbo-16k’ chat API to compare our approach with SoTA tools (Table 1). To systematically evaluate this approach, we create our own diverse synthetic dataset based on a subset of the MITRE 2022 top 25 most dangerous vulnerabilities. For each vulnerability we create vulnerable examples and their patches with varying levels of complexity. Write Vulnerability (CWE-787). Objective Design a framework to emulate step-by-step reasoning process of a human security expert using LLMs, to efficiently detect vulnerabilities in source code. Methodology Our approach uses few-shot in-context learning to guide LLMs to follow a step-by-step human-like reasoning model for vulnerability detection. Evaluation (a) Standard Prompting. Tool/Model Description cppcheck, infer, Combination of SoTA static analysis (SA) tools flawfinder for C/C++ RoBERTa-based model fine-tuned for defect UniXcoder detection in C/C++ LLM specifically pre-trained for progamming CodeT5+ langauges-related tasks, including C/C++ GPT-3.5 GPT-3.5 without reasoning Our approach GPT-3.5 with step-by-step reasoning with GPT-3.5 Size F1 Precision - 0.49 0.53 126M 0.33 0.25 16B 0.46 0.54 175B 0.48 0.50 175B 0.70 0.72 Table 1. Evaluation of different vulnerability analysis techniques on our dataset. Takeaways We make sure that the model first generates chain-of-thought reasoning [5] and then makes a decision based on that reasoning (as shown in Figure 1 and 3b). Following a human-like step-by-step reasoning approach helps LLMs to efficiently analyze code and detect vulnerabilities. Our approach provides an explanation for the detected vulnerabilities, which helps user to better contextualize them and to find their root cause. Systematic evaluation of this approach on real-world datasets is still required to determine its reliability in real-world use cases. Figure 1. Overview of our few-shot in-context learning approach for vulnerability detection using LLMs. * ML = Machine Learning LLM = Large Language Model (b) Step-by-Step Reason Prompting. Figure 3. Vulnerability Analysis using GPT-3.5