| Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, Lynette Hirschman
This paper presents a model-theoretic coreference scoring scheme for the MUC6 task. The scheme improves upon the original approach by grounding the scoring in a model, producing more intuitive recall and precision scores, and avoiding the need for explicit computation of the transitive closure of coreference. The key difference is moving from a syntactic scoring model based on following coreference links to one defined by the model theory of those links.
The scheme compares equivalence classes defined by links in the key and response, rather than the links themselves. These classes are the models of the IDENT equivalence relation. Recall and precision scores are calculated by determining the minimal perturbations needed to align the response's equivalence classes with the key's. For recall, the score is the number of missing links divided by the minimal number of correct links. For precision, the score is the number of correct links divided by the minimal number of correct links.
The paper presents a problematic case where the key links generate an equivalence class {A B C D}, while the response links generate two equivalence classes {A B} and {C D}. The recall score is 2/3, and the precision score is 2/2 = 1. This is because the response provides two correct links, while the key requires three to fully coreferentialize the entities.
The paper also presents more complex examples where the key and response do not neatly overlap. In one example, the key has seven entities and the response has three equivalence classes. The recall score is 50%, and the precision score is also 50%. In another example, the key has two equivalence classes and the response has three. The recall score is 40%, and the precision score is 50%.
The paper concludes that the model-theoretic scoring scheme is computationally efficient and produces intuitive scores. It is based on the model theory of coreference and avoids the need for explicit computation of the transitive closure of coreference.This paper presents a model-theoretic coreference scoring scheme for the MUC6 task. The scheme improves upon the original approach by grounding the scoring in a model, producing more intuitive recall and precision scores, and avoiding the need for explicit computation of the transitive closure of coreference. The key difference is moving from a syntactic scoring model based on following coreference links to one defined by the model theory of those links.
The scheme compares equivalence classes defined by links in the key and response, rather than the links themselves. These classes are the models of the IDENT equivalence relation. Recall and precision scores are calculated by determining the minimal perturbations needed to align the response's equivalence classes with the key's. For recall, the score is the number of missing links divided by the minimal number of correct links. For precision, the score is the number of correct links divided by the minimal number of correct links.
The paper presents a problematic case where the key links generate an equivalence class {A B C D}, while the response links generate two equivalence classes {A B} and {C D}. The recall score is 2/3, and the precision score is 2/2 = 1. This is because the response provides two correct links, while the key requires three to fully coreferentialize the entities.
The paper also presents more complex examples where the key and response do not neatly overlap. In one example, the key has seven entities and the response has three equivalence classes. The recall score is 50%, and the precision score is also 50%. In another example, the key has two equivalence classes and the response has three. The recall score is 40%, and the precision score is 50%.
The paper concludes that the model-theoretic scoring scheme is computationally efficient and produces intuitive scores. It is based on the model theory of coreference and avoids the need for explicit computation of the transitive closure of coreference.