5 Mar 2024 | Yutong Li, Lu Chen, Aiwei Liu, Kai Yu, Lijie Wen
**ChatCite: LLM Agent with Human Workflow Guidance for Comparative Literature Summary**
This paper introduces *ChatCite*, an LLM-based agent designed to generate comprehensive and comparative literature summaries. The agent is guided by human workflow to address the challenges of information omission, lack of comparative analysis, and organizational deficiencies in literature reviews. *ChatCite* consists of two main modules: the Key Element Extractor and the Reflective Incremental Generator. The Key Element Extractor processes the proposed work description and reference papers to extract key elements, while the Reflective Incremental Generator iteratively generates and evaluates literature summaries, ensuring consistency, coherence, comparative analysis, integrity, fluency, and cite accuracy.
The authors propose a new evaluation metric, G-Score, which is based on human evaluation criteria and assesses the quality of generated summaries across multiple dimensions. Experimental results show that *ChatCite* outperforms other LLM-based literature summarization methods in various quality dimensions. The generated summaries can be directly used for drafting literature reviews, demonstrating the effectiveness of *ChatCite* in handling complex inferential summarization tasks.
The paper also includes a detailed analysis of the effectiveness of each module through ablation studies and a human study, further validating the contributions of *ChatCite* in improving the quality and stability of literature summaries. The limitations and ethical considerations of the work are discussed, along with plans for future research to enhance the stability and quality of the output.**ChatCite: LLM Agent with Human Workflow Guidance for Comparative Literature Summary**
This paper introduces *ChatCite*, an LLM-based agent designed to generate comprehensive and comparative literature summaries. The agent is guided by human workflow to address the challenges of information omission, lack of comparative analysis, and organizational deficiencies in literature reviews. *ChatCite* consists of two main modules: the Key Element Extractor and the Reflective Incremental Generator. The Key Element Extractor processes the proposed work description and reference papers to extract key elements, while the Reflective Incremental Generator iteratively generates and evaluates literature summaries, ensuring consistency, coherence, comparative analysis, integrity, fluency, and cite accuracy.
The authors propose a new evaluation metric, G-Score, which is based on human evaluation criteria and assesses the quality of generated summaries across multiple dimensions. Experimental results show that *ChatCite* outperforms other LLM-based literature summarization methods in various quality dimensions. The generated summaries can be directly used for drafting literature reviews, demonstrating the effectiveness of *ChatCite* in handling complex inferential summarization tasks.
The paper also includes a detailed analysis of the effectiveness of each module through ablation studies and a human study, further validating the contributions of *ChatCite* in improving the quality and stability of literature summaries. The limitations and ethical considerations of the work are discussed, along with plans for future research to enhance the stability and quality of the output.