2024 | Zilong Lin, Jian Cui, Xiaojing Liao, XiaoFeng Wang
This paper presents the first systematic study of real-world malicious services (Malla) that exploit large language models (LLMs) for cybercrime. The study analyzes 212 Malla samples, revealing their proliferation in underground marketplaces and exposing their operational modalities. The research uncovers eight backend LLMs used by Mallas, along with 182 prompts that circumvent the protective measures of public LLM APIs. The study demystifies the tactics employed by Mallas, including the abuse of uncensored LLMs and the exploitation of public LLM APIs through jailbreak prompts. The findings highlight the significant growth and impact of Malla on public LLM services, as well as the potential for LLMs to be misused for malicious purposes. The study also reveals that certain Mallas, like DarkGPT and EscapeGPT, excel in generating high-quality malicious code that is both compilable and capable of evading Virus-Total detection. The research provides insights into the strategies used by cybercriminals to exploit LLMs and offers a deeper understanding of the economic and technical aspects of Malla. The study also identifies two predominant techniques used by miscreants: exploitation of uncensored LLMs and jailbreaking of public LLM APIs. The findings contribute to the understanding of the real-world exploitation of LLMs by cybercriminals and offer insights into strategies to counteract this cybercrime. The study also highlights the ethical concerns posed by publicly accessible, uncensored LLMs when they fall into the hands of adversaries. The research provides a detailed examination of the Malla ecosystem, revealing its significant growth and impact on today’s public LLM services. The study also identifies the key players within the Malla ecosystem, the methods used to orchestrate and monetize Malla, and the techniques used to exploit LLMs and build up Mallas. The study also reveals the economic factors at play, including the pricing of Malla services and the revenue generated by specific Malla services. The research also highlights the potential for LLMs to be misused for other prohibited purposes, such as generating harmful content. The study also discusses the ethical implications of data collection and the responsible disclosure of findings to affected LLM vendors and LLMA hosting platforms. The research provides a comprehensive analysis of the Malla ecosystem, including the scope and magnitude of Malla services, the stakeholders involved, the price strategy and revenue generated by Malla services, and the quality of Malla-generated content. The study also assesses the effectiveness of the generated exploit payloads and the susceptibility of users to Malla-created phishing content. The findings demonstrate the significant threat posed by Malla to cybersecurity and the need for further research and action to address this issue.This paper presents the first systematic study of real-world malicious services (Malla) that exploit large language models (LLMs) for cybercrime. The study analyzes 212 Malla samples, revealing their proliferation in underground marketplaces and exposing their operational modalities. The research uncovers eight backend LLMs used by Mallas, along with 182 prompts that circumvent the protective measures of public LLM APIs. The study demystifies the tactics employed by Mallas, including the abuse of uncensored LLMs and the exploitation of public LLM APIs through jailbreak prompts. The findings highlight the significant growth and impact of Malla on public LLM services, as well as the potential for LLMs to be misused for malicious purposes. The study also reveals that certain Mallas, like DarkGPT and EscapeGPT, excel in generating high-quality malicious code that is both compilable and capable of evading Virus-Total detection. The research provides insights into the strategies used by cybercriminals to exploit LLMs and offers a deeper understanding of the economic and technical aspects of Malla. The study also identifies two predominant techniques used by miscreants: exploitation of uncensored LLMs and jailbreaking of public LLM APIs. The findings contribute to the understanding of the real-world exploitation of LLMs by cybercriminals and offer insights into strategies to counteract this cybercrime. The study also highlights the ethical concerns posed by publicly accessible, uncensored LLMs when they fall into the hands of adversaries. The research provides a detailed examination of the Malla ecosystem, revealing its significant growth and impact on today’s public LLM services. The study also identifies the key players within the Malla ecosystem, the methods used to orchestrate and monetize Malla, and the techniques used to exploit LLMs and build up Mallas. The study also reveals the economic factors at play, including the pricing of Malla services and the revenue generated by specific Malla services. The research also highlights the potential for LLMs to be misused for other prohibited purposes, such as generating harmful content. The study also discusses the ethical implications of data collection and the responsible disclosure of findings to affected LLM vendors and LLMA hosting platforms. The research provides a comprehensive analysis of the Malla ecosystem, including the scope and magnitude of Malla services, the stakeholders involved, the price strategy and revenue generated by Malla services, and the quality of Malla-generated content. The study also assesses the effectiveness of the generated exploit payloads and the susceptibility of users to Malla-created phishing content. The findings demonstrate the significant threat posed by Malla to cybersecurity and the need for further research and action to address this issue.