OLMES is a new open standard for evaluating language models, designed to improve the transparency and reproducibility of evaluations. The paper highlights the challenges in evaluating language models due to varying practices in prompt formatting, in-context examples, probability normalization, and task formulation. These variations can lead to inconsistent results and make it difficult to compare models across different studies. OLMES addresses these issues by providing a standardized, documented, and practical evaluation framework. OLMES includes detailed guidelines for formatting dataset instances, selecting few-shot examples, normalizing probabilities for completion formulations, and choosing between multiple-choice (MCF) and completion (CF) task formulations. The standard ensures that evaluations are reproducible by specifying all evaluation details, from dataset processing to output interpretation. It also provides practical recommendations for resource usage and is open-source, allowing researchers to build upon it for new tasks and models. The paper presents experimental results showing that different normalization techniques can significantly affect evaluation outcomes. OLMES recommends specific normalization methods for different tasks, such as "pmi" for ARC-CHALLENGE, COMMONSENSEQA, and OPENBOOKQA, and "character" for ARC-EASY, HELLASWAG, PIQA, SOCIAL IQA, and MMLU. These recommendations are based on empirical analysis and are designed to provide fair and meaningful comparisons between models. OLMES also standardizes the use of both MCF and CF formulations for evaluating models, using the best result from each. This allows for a more accurate assessment of model performance, especially for models that have not yet mastered the MCF format. The paper demonstrates that using both formulations can lead to more representative and meaningful comparisons, particularly for smaller models that may struggle with MCF. The paper concludes that OLMES provides a valuable tool for improving the evaluation of language models, promoting consistency, transparency, and reproducibility in the field. It is designed to be adopted by researchers and developers, facilitating robust comparisons of model performances across a wide range of tasks and models.OLMES is a new open standard for evaluating language models, designed to improve the transparency and reproducibility of evaluations. The paper highlights the challenges in evaluating language models due to varying practices in prompt formatting, in-context examples, probability normalization, and task formulation. These variations can lead to inconsistent results and make it difficult to compare models across different studies. OLMES addresses these issues by providing a standardized, documented, and practical evaluation framework. OLMES includes detailed guidelines for formatting dataset instances, selecting few-shot examples, normalizing probabilities for completion formulations, and choosing between multiple-choice (MCF) and completion (CF) task formulations. The standard ensures that evaluations are reproducible by specifying all evaluation details, from dataset processing to output interpretation. It also provides practical recommendations for resource usage and is open-source, allowing researchers to build upon it for new tasks and models. The paper presents experimental results showing that different normalization techniques can significantly affect evaluation outcomes. OLMES recommends specific normalization methods for different tasks, such as "pmi" for ARC-CHALLENGE, COMMONSENSEQA, and OPENBOOKQA, and "character" for ARC-EASY, HELLASWAG, PIQA, SOCIAL IQA, and MMLU. These recommendations are based on empirical analysis and are designed to provide fair and meaningful comparisons between models. OLMES also standardizes the use of both MCF and CF formulations for evaluating models, using the best result from each. This allows for a more accurate assessment of model performance, especially for models that have not yet mastered the MCF format. The paper demonstrates that using both formulations can lead to more representative and meaningful comparisons, particularly for smaller models that may struggle with MCF. The paper concludes that OLMES provides a valuable tool for improving the evaluation of language models, promoting consistency, transparency, and reproducibility in the field. It is designed to be adopted by researchers and developers, facilitating robust comparisons of model performances across a wide range of tasks and models.