NetConfEval: Can LLMs Facilitate Network Configuration? This paper explores the potential of Large Language Models (LLMs) to simplify network configuration and reduce errors in network device setup and routing algorithm development. The authors design a benchmark called NetConfEval to evaluate how well different models can facilitate and automate network configuration. The benchmark focuses on four tasks: (i) converting high-level requirements into formal specifications, (ii) generating API/function calls from high-level requirements, (iii) developing routing algorithms based on high-level descriptions, and (iv) generating low-level configurations for existing and new protocols based on input documentation. The authors propose principles for designing LLM-based systems for network configuration and present two GPT-4-based prototypes: one to automatically configure P4-enabled devices from high-level requirements and another to integrate LLMs into existing network synthesizers. LLMs offer new opportunities for network configuration by enabling the rapid acquisition of vast knowledge, including protocol standards and best practices. However, challenges remain, such as the unreliability of LLMs, the need for effective prompt engineering, and the high cost of training and using LLMs. The authors evaluate various LLMs, including GPT-4, and find that GPT-4 exhibits high accuracy in translating human-language intents into formal specifications and generating basic routing algorithms. Smaller models also show good accuracy when fine-tuned for specific tasks. The paper discusses the potential of LLMs in network configuration, including the ability to detect conflicts, translate high-level requirements into function/API calls, and generate routing algorithms. The authors also highlight the importance of task-specific verifiers to ensure error-free outputs and the need for human oversight in the configuration process. The results show that GPT-4 performs well in these tasks, but challenges remain, such as the need for detailed instructions and the impact of context window size on performance. The authors propose three principles for designing LLM-based systems for network configuration: (1) splitting complex tasks into smaller subtasks, (2) supporting task-specific verifiers, and (3) keeping humans in the loop. The paper presents two prototypes using LLMs to configure P4-enabled networks and integrate LLMs into existing network synthesizers. The results demonstrate the feasibility of using LLMs for network configuration, but further research is needed to address the challenges and improve the performance of LLMs in this domain.NetConfEval: Can LLMs Facilitate Network Configuration? This paper explores the potential of Large Language Models (LLMs) to simplify network configuration and reduce errors in network device setup and routing algorithm development. The authors design a benchmark called NetConfEval to evaluate how well different models can facilitate and automate network configuration. The benchmark focuses on four tasks: (i) converting high-level requirements into formal specifications, (ii) generating API/function calls from high-level requirements, (iii) developing routing algorithms based on high-level descriptions, and (iv) generating low-level configurations for existing and new protocols based on input documentation. The authors propose principles for designing LLM-based systems for network configuration and present two GPT-4-based prototypes: one to automatically configure P4-enabled devices from high-level requirements and another to integrate LLMs into existing network synthesizers. LLMs offer new opportunities for network configuration by enabling the rapid acquisition of vast knowledge, including protocol standards and best practices. However, challenges remain, such as the unreliability of LLMs, the need for effective prompt engineering, and the high cost of training and using LLMs. The authors evaluate various LLMs, including GPT-4, and find that GPT-4 exhibits high accuracy in translating human-language intents into formal specifications and generating basic routing algorithms. Smaller models also show good accuracy when fine-tuned for specific tasks. The paper discusses the potential of LLMs in network configuration, including the ability to detect conflicts, translate high-level requirements into function/API calls, and generate routing algorithms. The authors also highlight the importance of task-specific verifiers to ensure error-free outputs and the need for human oversight in the configuration process. The results show that GPT-4 performs well in these tasks, but challenges remain, such as the need for detailed instructions and the impact of context window size on performance. The authors propose three principles for designing LLM-based systems for network configuration: (1) splitting complex tasks into smaller subtasks, (2) supporting task-specific verifiers, and (3) keeping humans in the loop. The paper presents two prototypes using LLMs to configure P4-enabled networks and integrate LLMs into existing network synthesizers. The results demonstrate the feasibility of using LLMs for network configuration, but further research is needed to address the challenges and improve the performance of LLMs in this domain.