PEDANTS (Precise Evaluations of Diverse Answer Nominee Text for Skinflints) is a new method for evaluating open-domain question answering (QA) that improves upon existing metrics like Exact Match (EM) and token F1. The paper addresses the limitations of current evaluation methods, which often fail to align with human judgments, especially for complex or long-form answers. The authors propose a more accurate, interpretable, and efficient evaluation method that incorporates guidelines from human QA competitions and uses a learned classifier to assess answer correctness. The paper introduces PEDANTS, a multi-level evaluation system that goes beyond EM and provides a more fine-grained assessment of answer correctness based on human judgment rules. PEDANTS uses a combination of rule-based and neural methods to evaluate answers, taking into account the type of question and the specific rules applicable to the answer. It is trained on a diverse set of QA examples, including those from the Jeopardy! community, and is tested on seven QA benchmarks and an expert human dataset. The authors argue that current evaluation methods are too simplistic and fail to capture the nuances of human judgment. PEDANTS is designed to be more robust and stable than existing methods, and it can serve as a proxy for GPT-4 evaluations on short-form QA. The paper also highlights the importance of diverse evaluation data and the need for more transparent and interpretable evaluation methods in QA research. The authors evaluate PEDANTS on various datasets and compare its performance with existing methods like EM, BERTScore, and LERC. They find that PEDANTS shows better correlation with human judgments and is more efficient than EM. The paper also discusses the challenges of QA evaluation, including the difficulty of assessing answers that require commonsense reasoning or fact-checking, and the need for more comprehensive and diverse evaluation metrics. Overall, the paper presents a new approach to QA evaluation that improves upon existing methods by incorporating human judgment rules and using a learned classifier to assess answer correctness. The authors believe that PEDANTS can help improve the accuracy and fairness of QA evaluations and provide a more reliable benchmark for comparing different QA models.PEDANTS (Precise Evaluations of Diverse Answer Nominee Text for Skinflints) is a new method for evaluating open-domain question answering (QA) that improves upon existing metrics like Exact Match (EM) and token F1. The paper addresses the limitations of current evaluation methods, which often fail to align with human judgments, especially for complex or long-form answers. The authors propose a more accurate, interpretable, and efficient evaluation method that incorporates guidelines from human QA competitions and uses a learned classifier to assess answer correctness. The paper introduces PEDANTS, a multi-level evaluation system that goes beyond EM and provides a more fine-grained assessment of answer correctness based on human judgment rules. PEDANTS uses a combination of rule-based and neural methods to evaluate answers, taking into account the type of question and the specific rules applicable to the answer. It is trained on a diverse set of QA examples, including those from the Jeopardy! community, and is tested on seven QA benchmarks and an expert human dataset. The authors argue that current evaluation methods are too simplistic and fail to capture the nuances of human judgment. PEDANTS is designed to be more robust and stable than existing methods, and it can serve as a proxy for GPT-4 evaluations on short-form QA. The paper also highlights the importance of diverse evaluation data and the need for more transparent and interpretable evaluation methods in QA research. The authors evaluate PEDANTS on various datasets and compare its performance with existing methods like EM, BERTScore, and LERC. They find that PEDANTS shows better correlation with human judgments and is more efficient than EM. The paper also discusses the challenges of QA evaluation, including the difficulty of assessing answers that require commonsense reasoning or fact-checking, and the need for more comprehensive and diverse evaluation metrics. Overall, the paper presents a new approach to QA evaluation that improves upon existing methods by incorporating human judgment rules and using a learned classifier to assess answer correctness. The authors believe that PEDANTS can help improve the accuracy and fairness of QA evaluations and provide a more reliable benchmark for comparing different QA models.