AttackEval introduces a novel framework to evaluate the effectiveness of jailbreak attacks on large language models (LLMs). As LLMs become more prevalent in critical domains, ensuring their security against sophisticated threats like jailbreak attacks is essential. Traditional binary evaluations focus on whether an LLM responds with harmful content, but AttackEval provides a more nuanced assessment by evaluating both the overall effectiveness of attack prompts across multiple LLMs and the intricacies of each prompt on individual models. The framework includes two evaluation methods: a coarse-grained approach that assesses effectiveness across various LLMs and a fine-grained approach that analyzes the effectiveness of individual prompts. A comprehensive ground truth dataset is developed to benchmark these evaluations, enabling researchers to systematically assess LLM responses under different jailbreak conditions. The coarse-grained evaluation measures the effectiveness of attack prompts across multiple LLMs, considering their varying defense robustness. It uses a weighted scoring system based on the performance of each LLM in defending against attacks. The fine-grained evaluation, with and without ground truth, provides a more detailed analysis by comparing the responses of LLMs to attack prompts. The ground truth version uses similarity scores between model responses and predefined answers, while the no-ground truth version classifies responses into four categories based on the degree of compliance with the attack prompt. The study compares these evaluation methods with traditional metrics like Attack Success Rate (ASR) and finds that AttackEval provides a more accurate and detailed assessment of attack prompt effectiveness. It identifies attack prompts that may appear harmless in traditional evaluations but are actually effective in bypassing LLMs. The results show that AttackEval aligns with baseline metrics but offers a more refined analysis, highlighting the importance of a multi-faceted approach to evaluating jailbreak attacks. The framework also helps identify potential vulnerabilities in LLMs and supports the development of more robust defense strategies. Overall, AttackEval establishes a solid foundation for assessing the effectiveness of attack prompts and contributes to the broader goal of enhancing the security of large language models.AttackEval introduces a novel framework to evaluate the effectiveness of jailbreak attacks on large language models (LLMs). As LLMs become more prevalent in critical domains, ensuring their security against sophisticated threats like jailbreak attacks is essential. Traditional binary evaluations focus on whether an LLM responds with harmful content, but AttackEval provides a more nuanced assessment by evaluating both the overall effectiveness of attack prompts across multiple LLMs and the intricacies of each prompt on individual models. The framework includes two evaluation methods: a coarse-grained approach that assesses effectiveness across various LLMs and a fine-grained approach that analyzes the effectiveness of individual prompts. A comprehensive ground truth dataset is developed to benchmark these evaluations, enabling researchers to systematically assess LLM responses under different jailbreak conditions. The coarse-grained evaluation measures the effectiveness of attack prompts across multiple LLMs, considering their varying defense robustness. It uses a weighted scoring system based on the performance of each LLM in defending against attacks. The fine-grained evaluation, with and without ground truth, provides a more detailed analysis by comparing the responses of LLMs to attack prompts. The ground truth version uses similarity scores between model responses and predefined answers, while the no-ground truth version classifies responses into four categories based on the degree of compliance with the attack prompt. The study compares these evaluation methods with traditional metrics like Attack Success Rate (ASR) and finds that AttackEval provides a more accurate and detailed assessment of attack prompt effectiveness. It identifies attack prompts that may appear harmless in traditional evaluations but are actually effective in bypassing LLMs. The results show that AttackEval aligns with baseline metrics but offers a more refined analysis, highlighting the importance of a multi-faceted approach to evaluating jailbreak attacks. The framework also helps identify potential vulnerabilities in LLMs and supports the development of more robust defense strategies. Overall, AttackEval establishes a solid foundation for assessing the effectiveness of attack prompts and contributes to the broader goal of enhancing the security of large language models.