**Abstract:** As the range of applications for Large Language Models (LLMs) continues to expand, the demand for effective serving solutions becomes increasingly critical. Despite the versatility of LLMs, no single model can optimally address all tasks and applications, particularly when balancing performance with cost. This limitation has led to the development of LLM routing systems, which combine the strengths of various models to overcome the constraints of individual LLMs. However, the absence of a standardized benchmark for evaluating the performance of LLM routers hinders progress in this area. To address this gap, we present ROUTERBENCH, a novel evaluation framework designed to systematically assess the efficacy of LLM routing systems. Alongside ROUTERBENCH, we provide a comprehensive dataset comprising over 405k inference outcomes from representative LLMs to support the development of routing strategies. We also propose a theoretical framework for LLM routing and deliver a comparative analysis of various routing approaches through ROUTERBENCH, highlighting their potentials and limitations within our evaluation framework. This work not only formalizes and advances the development of LLM routing systems but also sets a standard for their assessment, paving the way for more accessible and economically viable LLM deployments. The code and data are available at https://github.com/withmartian/routerbench. **Introduction:** Large Language Models (LLMs) have demonstrated remarkable capabilities in addressing a wide range of tasks across academic and industrial scenarios. However, the proliferation of LLMs presents a challenge for application builders to identify the most suitable model for their applications. While some proprietary models, such as GPT-4, offer superior performance, they often incur high economic costs due to their high API prices. Single-LLM enhancements, such as fine-tuning, prompting, and quantization, can improve performance but may not remain feasible or scalable in the long term. Routing, which selects the optimal LLM for each input without performing inference on every candidate model, offers several advantages over single-LLM optimization. It is lightweight, flexible, and can benefit from the diversity of LLMs. ROUTERBENCH is designed to evaluate routing systems in terms of inference cost and performance, covering a broad spectrum of tasks and domains. Our experiments reveal that while some previous routing mechanisms struggle with complex tasks and up-to-date models, there are promising fields where even simple routing demonstrates outstanding performance. **Math Formulation for Router Evaluation:** We develop a framework that captures the multi-faceted nature of router performance through a single metric. This framework includes mathematical formulations for expected cost and quality, and introduces concepts such as linear interpolation and non-decreasing convex hull to facilitate the comparison of different routers. **Benchmark Construction - ROUTERBENCH:** ROUTERBENCH is constructed by leveraging existing datasets widely recognized and utilized in the evaluation of leading LLMs, ensuring a diverse and representative set of tasks. The benchmark consists of 8 representative datasets from multiple different tasks, including commons**Abstract:** As the range of applications for Large Language Models (LLMs) continues to expand, the demand for effective serving solutions becomes increasingly critical. Despite the versatility of LLMs, no single model can optimally address all tasks and applications, particularly when balancing performance with cost. This limitation has led to the development of LLM routing systems, which combine the strengths of various models to overcome the constraints of individual LLMs. However, the absence of a standardized benchmark for evaluating the performance of LLM routers hinders progress in this area. To address this gap, we present ROUTERBENCH, a novel evaluation framework designed to systematically assess the efficacy of LLM routing systems. Alongside ROUTERBENCH, we provide a comprehensive dataset comprising over 405k inference outcomes from representative LLMs to support the development of routing strategies. We also propose a theoretical framework for LLM routing and deliver a comparative analysis of various routing approaches through ROUTERBENCH, highlighting their potentials and limitations within our evaluation framework. This work not only formalizes and advances the development of LLM routing systems but also sets a standard for their assessment, paving the way for more accessible and economically viable LLM deployments. The code and data are available at https://github.com/withmartian/routerbench. **Introduction:** Large Language Models (LLMs) have demonstrated remarkable capabilities in addressing a wide range of tasks across academic and industrial scenarios. However, the proliferation of LLMs presents a challenge for application builders to identify the most suitable model for their applications. While some proprietary models, such as GPT-4, offer superior performance, they often incur high economic costs due to their high API prices. Single-LLM enhancements, such as fine-tuning, prompting, and quantization, can improve performance but may not remain feasible or scalable in the long term. Routing, which selects the optimal LLM for each input without performing inference on every candidate model, offers several advantages over single-LLM optimization. It is lightweight, flexible, and can benefit from the diversity of LLMs. ROUTERBENCH is designed to evaluate routing systems in terms of inference cost and performance, covering a broad spectrum of tasks and domains. Our experiments reveal that while some previous routing mechanisms struggle with complex tasks and up-to-date models, there are promising fields where even simple routing demonstrates outstanding performance. **Math Formulation for Router Evaluation:** We develop a framework that captures the multi-faceted nature of router performance through a single metric. This framework includes mathematical formulations for expected cost and quality, and introduces concepts such as linear interpolation and non-decreasing convex hull to facilitate the comparison of different routers. **Benchmark Construction - ROUTERBENCH:** ROUTERBENCH is constructed by leveraging existing datasets widely recognized and utilized in the evaluation of leading LLMs, ensuring a diverse and representative set of tasks. The benchmark consists of 8 representative datasets from multiple different tasks, including commons