This paper introduces SPML, a domain-specific language (DSL) for refining prompts and monitoring inputs to large language model (LLM)-based chatbots. SPML actively checks attack prompts to ensure user inputs align with chatbot definitions, preventing malicious execution on the LLM backbone and optimizing costs. It also streamlines chatbot definition crafting with programming language capabilities, overcoming natural language design challenges. Additionally, the paper introduces a benchmark with 1.8k system prompts and 20k user inputs, offering the first language and benchmark for chatbot definition evaluation. Experiments across datasets demonstrate SPML's proficiency in understanding attacker prompts, surpassing models like GPT-4, GPT-3.5, and LLAMA. SPML's key contributions include a novel DSL for crafting secure prompts, a benchmark for evaluating chatbot definitions, and a method for detecting prompt injection attacks. SPML outperforms state-of-the-art LLMs in identifying attacks and handles multi-layered attacks. The paper also presents a comprehensive dataset of system prompts and user inputs for evaluating SPML's effectiveness. The results show that SPML is effective in detecting prompt injection attacks and can be used to monitor and secure LLM-based chatbots.This paper introduces SPML, a domain-specific language (DSL) for refining prompts and monitoring inputs to large language model (LLM)-based chatbots. SPML actively checks attack prompts to ensure user inputs align with chatbot definitions, preventing malicious execution on the LLM backbone and optimizing costs. It also streamlines chatbot definition crafting with programming language capabilities, overcoming natural language design challenges. Additionally, the paper introduces a benchmark with 1.8k system prompts and 20k user inputs, offering the first language and benchmark for chatbot definition evaluation. Experiments across datasets demonstrate SPML's proficiency in understanding attacker prompts, surpassing models like GPT-4, GPT-3.5, and LLAMA. SPML's key contributions include a novel DSL for crafting secure prompts, a benchmark for evaluating chatbot definitions, and a method for detecting prompt injection attacks. SPML outperforms state-of-the-art LLMs in identifying attacks and handles multi-layered attacks. The paper also presents a comprehensive dataset of system prompts and user inputs for evaluating SPML's effectiveness. The results show that SPML is effective in detecting prompt injection attacks and can be used to monitor and secure LLM-based chatbots.