The paper introduces DIALECTBENCH, a comprehensive benchmark for evaluating natural language processing (NLP) systems on non-standard dialects and language varieties. The benchmark aims to address the gap in NLP research and evaluation by focusing on language variation, which is often overlooked. DIALECTBENCH includes 10 text-level tasks covering 281 varieties across 40 language clusters, providing a broad and detailed assessment of NLP system performance on different language varieties. Key aspects of DIALECTBENCH include: - **Variety Selection**: Languages and varieties are selected based on well-established resources and high resource varieties. - **Cluster-Variety Mapping**: Varieties are grouped into clusters based on mutual intelligibility, phylogenetic similarity, and geographic proximity. - **Task and Dataset Selection**: Tasks are chosen to promote diversity and require diverse levels of textual understanding. - **Evaluation Principles**: Performance is evaluated using standard metrics and assessed for linguistic and demographic utility. The paper reports on the creation and evaluation of baselines for all tasks and varieties in DIALECTBENCH, including the introduction of a *dialect performance gap* metric to quantify performance disparities. The results highlight significant performance gaps between standard and non-standard varieties, particularly in low-resource varieties. The study also discusses the impact of model hyperparameters and the effectiveness of zero-shot and fine-tuning approaches. The authors conclude that DIALECTBENCH is a valuable resource for advancing NLP research on language varieties and non-standard dialects, and they outline future directions for improving the benchmark, such as expanding task coverage and addressing data quality and quantity issues.The paper introduces DIALECTBENCH, a comprehensive benchmark for evaluating natural language processing (NLP) systems on non-standard dialects and language varieties. The benchmark aims to address the gap in NLP research and evaluation by focusing on language variation, which is often overlooked. DIALECTBENCH includes 10 text-level tasks covering 281 varieties across 40 language clusters, providing a broad and detailed assessment of NLP system performance on different language varieties. Key aspects of DIALECTBENCH include: - **Variety Selection**: Languages and varieties are selected based on well-established resources and high resource varieties. - **Cluster-Variety Mapping**: Varieties are grouped into clusters based on mutual intelligibility, phylogenetic similarity, and geographic proximity. - **Task and Dataset Selection**: Tasks are chosen to promote diversity and require diverse levels of textual understanding. - **Evaluation Principles**: Performance is evaluated using standard metrics and assessed for linguistic and demographic utility. The paper reports on the creation and evaluation of baselines for all tasks and varieties in DIALECTBENCH, including the introduction of a *dialect performance gap* metric to quantify performance disparities. The results highlight significant performance gaps between standard and non-standard varieties, particularly in low-resource varieties. The study also discusses the impact of model hyperparameters and the effectiveness of zero-shot and fine-tuning approaches. The authors conclude that DIALECTBENCH is a valuable resource for advancing NLP research on language varieties and non-standard dialects, and they outline future directions for improving the benchmark, such as expanding task coverage and addressing data quality and quantity issues.