LONGAGENT: Scaling Language Models to 128k Context through Multi-Agent Collaboration

LONGAGENT: Scaling Language Models to 128k Context through Multi-Agent Collaboration

2024 | Jun Zhao, Can Zu, Hao Xu, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, Xuanjing Huang
LONGAGENT is a multi-agent collaboration method that scales large language models (LLMs) to handle long texts with up to 128k tokens. The method involves a leader and multiple members working together to process long texts. The leader understands user intent, organizes discussions, and resolves conflicts among members. Members process text chunks and respond to the leader's instructions. To address hallucinations, an inter-member communication mechanism is used to resolve conflicts and ensure accurate information. The method is evaluated on tasks such as long-text retrieval and multi-hop question answering, showing potential superiority over GPT-4. The leader coordinates members to gather information, resolve conflicts, and deduce answers. The method is efficient, with linear time complexity for processing long texts, and demonstrates improved performance in handling long texts compared to other methods. The results show that LONGAGENT outperforms existing models in long-text processing, with significant improvements in accuracy. The method also addresses hallucination issues through inter-member communication and demonstrates potential for handling texts exceeding 128k tokens. The paper introduces a new benchmark, Needle-in-a-Haystack PLUS, to evaluate LLMs' long-text capabilities. The results indicate that LONGAGENT is a promising approach for long-text processing.LONGAGENT is a multi-agent collaboration method that scales large language models (LLMs) to handle long texts with up to 128k tokens. The method involves a leader and multiple members working together to process long texts. The leader understands user intent, organizes discussions, and resolves conflicts among members. Members process text chunks and respond to the leader's instructions. To address hallucinations, an inter-member communication mechanism is used to resolve conflicts and ensure accurate information. The method is evaluated on tasks such as long-text retrieval and multi-hop question answering, showing potential superiority over GPT-4. The leader coordinates members to gather information, resolve conflicts, and deduce answers. The method is efficient, with linear time complexity for processing long texts, and demonstrates improved performance in handling long texts compared to other methods. The results show that LONGAGENT outperforms existing models in long-text processing, with significant improvements in accuracy. The method also addresses hallucination issues through inter-member communication and demonstrates potential for handling texts exceeding 128k tokens. The paper introduces a new benchmark, Needle-in-a-Haystack PLUS, to evaluate LLMs' long-text capabilities. The results indicate that LONGAGENT is a promising approach for long-text processing.
Reach us at info@study.space
Understanding LongAgent%3A Scaling Language Models to 128k Context through Multi-Agent Collaboration