Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue!

Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue!

May 21, 2024 | Dean Allemang, Juan F. Sequeda
This technical report presents an approach to improve the accuracy of Large Language Model (LLM)-powered question-answering systems by leveraging ontologies and incorporating LLM-based query repair. The study shows that using a knowledge graph representation of an enterprise SQL database significantly increases accuracy compared to direct SQL-based question answering. Previous research demonstrated that using a knowledge graph improved accuracy from 16% to 54%. The current study further enhances this accuracy to 72% by introducing two components: Ontology-based Query Check (OBQC) and LLM Repair. OBQC detects errors in LLM-generated SPARQL queries by comparing them against the semantics of the ontology. It checks both the body (WHERE clause) and head (SELECT clause) of the query. The OBQC is deterministic and does not use an LLM, relying solely on the ontology's logical structure. If an error is detected, the query is either not returned or passed to the LLM Repair component. LLM Repair uses the error explanations generated by OBQC to prompt the LLM to repair the query. This process is repeated until the query passes the OBQC check or a maximum number of iterations is reached. The results show that this approach significantly improves accuracy, with an overall error rate of 20%, compared to 83.3% without using a knowledge graph. The study also identifies common errors in LLM-generated SPARQL queries, such as incorrect property direction or domain/range mismatches. The OBQC primarily addresses domain-related errors, which account for 70% of the repairs, while SELECT clause checks account for 30%. The results demonstrate that ontologies can enhance the accuracy of LLM-powered question-answering systems, supporting the conclusion that investing in metadata, semantics, ontologies, and knowledge graphs is essential for achieving higher accuracy in such systems.This technical report presents an approach to improve the accuracy of Large Language Model (LLM)-powered question-answering systems by leveraging ontologies and incorporating LLM-based query repair. The study shows that using a knowledge graph representation of an enterprise SQL database significantly increases accuracy compared to direct SQL-based question answering. Previous research demonstrated that using a knowledge graph improved accuracy from 16% to 54%. The current study further enhances this accuracy to 72% by introducing two components: Ontology-based Query Check (OBQC) and LLM Repair. OBQC detects errors in LLM-generated SPARQL queries by comparing them against the semantics of the ontology. It checks both the body (WHERE clause) and head (SELECT clause) of the query. The OBQC is deterministic and does not use an LLM, relying solely on the ontology's logical structure. If an error is detected, the query is either not returned or passed to the LLM Repair component. LLM Repair uses the error explanations generated by OBQC to prompt the LLM to repair the query. This process is repeated until the query passes the OBQC check or a maximum number of iterations is reached. The results show that this approach significantly improves accuracy, with an overall error rate of 20%, compared to 83.3% without using a knowledge graph. The study also identifies common errors in LLM-generated SPARQL queries, such as incorrect property direction or domain/range mismatches. The OBQC primarily addresses domain-related errors, which account for 70% of the repairs, while SELECT clause checks account for 30%. The results demonstrate that ontologies can enhance the accuracy of LLM-powered question-answering systems, supporting the conclusion that investing in metadata, semantics, ontologies, and knowledge graphs is essential for achieving higher accuracy in such systems.
Reach us at info@study.space
Understanding Increasing the LLM Accuracy for Question Answering%3A Ontologies to the Rescue!