Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering

Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering

July 14-18, 2024 | Zhintao Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang, Zheng Li
This paper presents a novel customer service question-answering method that integrates retrieval-augmented generation (RAG) with a knowledge graph (KG). The method constructs a KG from historical customer service issue tickets, preserving the intra-issue structure and inter-issue relations. During the question-answering phase, the system parses consumer queries and retrieves related sub-graphs from the KG to generate answers. This integration improves retrieval accuracy by preserving customer service structure information and enhances answering quality by mitigating the effects of text segmentation. Empirical assessments on benchmark datasets show that the method outperforms the baseline by 77.6% in MRR and by 0.32 in BLEU. The method has been deployed within LinkedIn's customer service team for approximately six months and has reduced the median per-issue resolution time by 28.6%. The system comprises two phases: KG construction and question-answering. In the KG construction phase, the system builds a comprehensive knowledge graph from historical customer service issue tickets, integrating a tree-structured representation of each issue and interlinking them based on relational context. In the question-answering phase, the system parses consumer queries to identify named entities and intents, then navigates within the KG to identify related sub-graphs for generating answers. The method uses embeddings for graph node values generated using pre-trained text-embedding models like BERT and E5. The system also employs embedding-based retrieval to extract pertinent sub-graphs from the knowledge graph, aligned with user-provided specifics and user intentions. The answers are synthesized by correlating retrieved data with the initial query, with the LLM serving as a decoder to formulate responses to user inquiries. The system has been tested on a curated "golden" dataset, showing consistent improvements across all metrics, with the method surpassing the baseline by 77.6% in MRR and by 0.32 in BLEU. The system has been deployed within LinkedIn's customer service team, significantly reducing the median resolution time per issue. The research advances automated question answering systems for customer service by integrating RAG with a KG, improving retrieval and answering metrics, and overall service effectiveness. Future work will focus on developing an automated mechanism for extracting graph templates, enhancing system adaptability, investigating dynamic updates to the knowledge graph based on user queries, and exploring the system's applicability in other contexts beyond customer service.This paper presents a novel customer service question-answering method that integrates retrieval-augmented generation (RAG) with a knowledge graph (KG). The method constructs a KG from historical customer service issue tickets, preserving the intra-issue structure and inter-issue relations. During the question-answering phase, the system parses consumer queries and retrieves related sub-graphs from the KG to generate answers. This integration improves retrieval accuracy by preserving customer service structure information and enhances answering quality by mitigating the effects of text segmentation. Empirical assessments on benchmark datasets show that the method outperforms the baseline by 77.6% in MRR and by 0.32 in BLEU. The method has been deployed within LinkedIn's customer service team for approximately six months and has reduced the median per-issue resolution time by 28.6%. The system comprises two phases: KG construction and question-answering. In the KG construction phase, the system builds a comprehensive knowledge graph from historical customer service issue tickets, integrating a tree-structured representation of each issue and interlinking them based on relational context. In the question-answering phase, the system parses consumer queries to identify named entities and intents, then navigates within the KG to identify related sub-graphs for generating answers. The method uses embeddings for graph node values generated using pre-trained text-embedding models like BERT and E5. The system also employs embedding-based retrieval to extract pertinent sub-graphs from the knowledge graph, aligned with user-provided specifics and user intentions. The answers are synthesized by correlating retrieved data with the initial query, with the LLM serving as a decoder to formulate responses to user inquiries. The system has been tested on a curated "golden" dataset, showing consistent improvements across all metrics, with the method surpassing the baseline by 77.6% in MRR and by 0.32 in BLEU. The system has been deployed within LinkedIn's customer service team, significantly reducing the median resolution time per issue. The research advances automated question answering systems for customer service by integrating RAG with a KG, improving retrieval and answering metrics, and overall service effectiveness. Future work will focus on developing an automated mechanism for extracting graph templates, enhancing system adaptability, investigating dynamic updates to the knowledge graph based on user queries, and exploring the system's applicability in other contexts beyond customer service.
Reach us at info@study.space
[slides and audio] Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering