This paper introduces a unified framework for understanding prompt injection attacks against large language models (LLMs) and proposes an automated gradient-based method for generating highly effective and universal prompt injection data. The authors address two major challenges in prompt injection research: unclear attack objectives and the reliance on manually crafted prompts. They define three distinct prompt injection objectives—static, semi-dynamic, and dynamic—to cover a wide range of attack scenarios. The static objective aims for a consistent response regardless of user instructions, the semi-dynamic objective requires the model to produce consistent content before responding to user input, and the dynamic objective involves generating responses relevant to user instructions while incorporating malicious content. The authors introduce a momentum-enhanced gradient search-based algorithm that utilizes gradient information from victim LLMs to automatically generate prompt injection data. This method achieves high attack success rates across diverse text datasets, even in the presence of defensive measures. The approach is evaluated on seven natural language tasks, demonstrating its effectiveness and universality. The results show that the proposed method outperforms existing baselines, achieving an average attack success rate of 50% with only five training samples. The method is also effective against various defense mechanisms, highlighting the need for gradient-based testing in assessing prompt injection robustness. The study emphasizes the importance of automatic and universal prompt injection attacks in evaluating the security of LLM applications. It underscores the risks posed by prompt injection attacks, which can manipulate LLM-integrated applications to produce responses aligned with the attacker's goals, deviating from user requests. The research contributes to the field by providing a comprehensive framework for understanding and evaluating prompt injection attacks, as well as an effective method for generating attack data. The findings highlight the need for robust defense mechanisms and the importance of testing LLMs against such attacks to ensure their security and reliability.This paper introduces a unified framework for understanding prompt injection attacks against large language models (LLMs) and proposes an automated gradient-based method for generating highly effective and universal prompt injection data. The authors address two major challenges in prompt injection research: unclear attack objectives and the reliance on manually crafted prompts. They define three distinct prompt injection objectives—static, semi-dynamic, and dynamic—to cover a wide range of attack scenarios. The static objective aims for a consistent response regardless of user instructions, the semi-dynamic objective requires the model to produce consistent content before responding to user input, and the dynamic objective involves generating responses relevant to user instructions while incorporating malicious content. The authors introduce a momentum-enhanced gradient search-based algorithm that utilizes gradient information from victim LLMs to automatically generate prompt injection data. This method achieves high attack success rates across diverse text datasets, even in the presence of defensive measures. The approach is evaluated on seven natural language tasks, demonstrating its effectiveness and universality. The results show that the proposed method outperforms existing baselines, achieving an average attack success rate of 50% with only five training samples. The method is also effective against various defense mechanisms, highlighting the need for gradient-based testing in assessing prompt injection robustness. The study emphasizes the importance of automatic and universal prompt injection attacks in evaluating the security of LLM applications. It underscores the risks posed by prompt injection attacks, which can manipulate LLM-integrated applications to produce responses aligned with the attacker's goals, deviating from user requests. The research contributes to the field by providing a comprehensive framework for understanding and evaluating prompt injection attacks, as well as an effective method for generating attack data. The findings highlight the need for robust defense mechanisms and the importance of testing LLMs against such attacks to ensure their security and reliability.