FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generation

FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generation

19 Feb 2024 | SHUAI WANG, The University of Queensland, Australia; EKATERINA KHRAMTSOVA, The University of Queensland, Australia; SHENGYAO ZHUANG, CSIRO, Australia; GUIDO ZUCCON, The University of Queensland, Australia
FeB4RAG is a novel dataset designed to evaluate federated search within Retrieval-Augmented Generation (RAG) frameworks. Federated search aggregates results from multiple search engines to enhance quality and align with user intent, but existing datasets like TREC FedWeb tracks lack representation of modern information retrieval challenges. FeB4RAG addresses this gap by including 790 information requests tailored for chatbot applications, along with top results and LLM-derived relevance judgments. The dataset supports the development and evaluation of new federated search methods, particularly in the context of RAG pipelines. The paper demonstrates the impact of a high-quality federated search system on response generation compared to a naive approach, showing that current practices in federated search are suboptimal. FeB4RAG also provides a framework for resource selection and result merging tasks, with potential for future expansion and adaptation. The evaluation highlights the importance of effective federated search in RAG, emphasizing the need for more sophisticated methods to improve response quality and user satisfaction.FeB4RAG is a novel dataset designed to evaluate federated search within Retrieval-Augmented Generation (RAG) frameworks. Federated search aggregates results from multiple search engines to enhance quality and align with user intent, but existing datasets like TREC FedWeb tracks lack representation of modern information retrieval challenges. FeB4RAG addresses this gap by including 790 information requests tailored for chatbot applications, along with top results and LLM-derived relevance judgments. The dataset supports the development and evaluation of new federated search methods, particularly in the context of RAG pipelines. The paper demonstrates the impact of a high-quality federated search system on response generation compared to a naive approach, showing that current practices in federated search are suboptimal. FeB4RAG also provides a framework for resource selection and result merging tasks, with potential for future expansion and adaptation. The evaluation highlights the importance of effective federated search in RAG, emphasizing the need for more sophisticated methods to improve response quality and user satisfaction.
Reach us at info@study.space