This paper proposes a novel automated evaluation method for Retrieval-Augmented Language Models (RAG) using task-specific synthetic exams. The method generates multiple-choice questions based on the corpus of documents associated with the task, allowing for automated, cost-efficient, and interpretable evaluation of RAG systems. Item Response Theory (IRT) is used to estimate exam quality and informativeness, enabling iterative improvement by eliminating low-informative questions. The approach is demonstrated on four open-ended question-answering tasks based on ArXiv abstracts, StackExchange questions, AWS DevOps troubleshooting guides, and SEC filings. The results reveal that optimizing retrieval mechanisms often leads to greater performance gains than simply increasing model size. The method provides a comprehensive framework for evaluating RAG pipelines, including benchmark datasets and an open-source implementation. The framework allows for the automated generation and iterative refinement of exams to maximize their informativeness. The paper also discusses the impact of factors such as model size, retrieval mechanism, prompting, and fine-tuning on RAG performance. The proposed method offers predictive and prescriptive evaluation, enabling continuous feedback-driven improvements to the exam corpus. The work contributes to the field of automated evaluation by providing a scalable, interpretable, and robust approach to assessing RAG systems.This paper proposes a novel automated evaluation method for Retrieval-Augmented Language Models (RAG) using task-specific synthetic exams. The method generates multiple-choice questions based on the corpus of documents associated with the task, allowing for automated, cost-efficient, and interpretable evaluation of RAG systems. Item Response Theory (IRT) is used to estimate exam quality and informativeness, enabling iterative improvement by eliminating low-informative questions. The approach is demonstrated on four open-ended question-answering tasks based on ArXiv abstracts, StackExchange questions, AWS DevOps troubleshooting guides, and SEC filings. The results reveal that optimizing retrieval mechanisms often leads to greater performance gains than simply increasing model size. The method provides a comprehensive framework for evaluating RAG pipelines, including benchmark datasets and an open-source implementation. The framework allows for the automated generation and iterative refinement of exams to maximize their informativeness. The paper also discusses the impact of factors such as model size, retrieval mechanism, prompting, and fine-tuning on RAG performance. The proposed method offers predictive and prescriptive evaluation, enabling continuous feedback-driven improvements to the exam corpus. The work contributes to the field of automated evaluation by providing a scalable, interpretable, and robust approach to assessing RAG systems.