17 Aug 2024 | Wei Ma1, Daoyuan Wu2*, Yuqiang Sun1, Tianwen Wang3, Shangqing Liu1, Jian Zhang1, Yue Xue4, Yang Liu1
The paper "Combining Fine-tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justifications" addresses the challenge of auditing smart contracts using large language models (LLMs). The authors propose iAudit, a framework that combines fine-tuning and LLM-based agents to enhance the accuracy and intuition of smart contract auditing. iAudit employs a two-stage fine-tuning approach: first, a Detector model is fine-tuned to make decisions about whether code is vulnerable, and then a Reasoner model is fine-tuned to generate explanations for the identified vulnerabilities. To improve the accuracy of identifying the optimal cause of a vulnerability, the authors introduce two LLM-based agents, Ranker and Critic, which iteratively select and debate the most suitable cause based on the output of the Reasoner model.
The evaluation of iAudit involves a balanced dataset of 1,734 positive and 1,810 negative samples, collected from real smart contract auditing reports. iAudit is compared with traditional fine-tuned models (CodeBERT, GraphCodeBERT, CodeTS, and UnixCoder) and prompt learning-based LLMs (GPT-14, GPT-3.5, and CodeLlama-13b/24b). The results show that iAudit achieves an F1 score of 91.21% and an accuracy of 91.11%, significantly outperforming other models. The consistency of the generated explanations with the ground truth causes is also high, reaching 37.99%.
The paper also includes ablation studies to validate the effectiveness of the two-stage fine-tuning approach, majority voting, and the impact of additional call graph information. The results confirm that iAudit's two-stage approach, majority voting, and the integration of additional information have positive effects on the model's performance. However, the inclusion of call graph information can sometimes confuse the model, leading to reduced performance.
Overall, the paper demonstrates the potential of iAudit in improving the accuracy and intuition of smart contract auditing, making it a valuable tool for developers and auditors.The paper "Combining Fine-tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justifications" addresses the challenge of auditing smart contracts using large language models (LLMs). The authors propose iAudit, a framework that combines fine-tuning and LLM-based agents to enhance the accuracy and intuition of smart contract auditing. iAudit employs a two-stage fine-tuning approach: first, a Detector model is fine-tuned to make decisions about whether code is vulnerable, and then a Reasoner model is fine-tuned to generate explanations for the identified vulnerabilities. To improve the accuracy of identifying the optimal cause of a vulnerability, the authors introduce two LLM-based agents, Ranker and Critic, which iteratively select and debate the most suitable cause based on the output of the Reasoner model.
The evaluation of iAudit involves a balanced dataset of 1,734 positive and 1,810 negative samples, collected from real smart contract auditing reports. iAudit is compared with traditional fine-tuned models (CodeBERT, GraphCodeBERT, CodeTS, and UnixCoder) and prompt learning-based LLMs (GPT-14, GPT-3.5, and CodeLlama-13b/24b). The results show that iAudit achieves an F1 score of 91.21% and an accuracy of 91.11%, significantly outperforming other models. The consistency of the generated explanations with the ground truth causes is also high, reaching 37.99%.
The paper also includes ablation studies to validate the effectiveness of the two-stage fine-tuning approach, majority voting, and the impact of additional call graph information. The results confirm that iAudit's two-stage approach, majority voting, and the integration of additional information have positive effects on the model's performance. However, the inclusion of call graph information can sometimes confuse the model, leading to reduced performance.
Overall, the paper demonstrates the potential of iAudit in improving the accuracy and intuition of smart contract auditing, making it a valuable tool for developers and auditors.