GenQREnsemble: Zero-Shot LLM Ensemble Prompting for Generative Query Reformulation

GenQREnsemble: Zero-Shot LLM Ensemble Prompting for Generative Query Reformulation

4 Apr 2024 | Kaustubh D. Dhole, Eugene Agichtein
GenQREnsemble is a zero-shot ensemble prompting method for generative query reformulation (QR). It leverages multiple paraphrased instructions to generate diverse query reformulations, improving retrieval performance. The method is evaluated on four IR benchmarks, showing significant improvements over previous zero-shot approaches. GenQREnsemble outperforms existing methods by up to 18% in nDCG@10 and 24% in MAP in pre-retrieval settings. Its post-retrieval variant, GenQREnsembleRF, incorporates pseudo-relevance feedback, achieving 5% MRR and 9% nDCG@10 improvements on the MSMarco Passage Ranking task. The approach uses ensemble prompting to generate diverse query reformulations, which enhances the effectiveness of query expansion. The method is evaluated using various retrieval settings, including BM25 and neural reranking. Results show that GenQREnsemble performs better than single-instruction approaches, especially when combined with neural reranking. The study also highlights the benefits of incorporating relevance feedback, demonstrating that GenQREnsembleRF can effectively utilize feedback documents to improve retrieval performance. While generative QR benefits from the ensemble approach, it may introduce increased latency, which is becoming less of a concern with the availability of batch inference for LLMs. The proposed methods could be applied to other settings, such as optimizing different aspects of queries or metrics.GenQREnsemble is a zero-shot ensemble prompting method for generative query reformulation (QR). It leverages multiple paraphrased instructions to generate diverse query reformulations, improving retrieval performance. The method is evaluated on four IR benchmarks, showing significant improvements over previous zero-shot approaches. GenQREnsemble outperforms existing methods by up to 18% in nDCG@10 and 24% in MAP in pre-retrieval settings. Its post-retrieval variant, GenQREnsembleRF, incorporates pseudo-relevance feedback, achieving 5% MRR and 9% nDCG@10 improvements on the MSMarco Passage Ranking task. The approach uses ensemble prompting to generate diverse query reformulations, which enhances the effectiveness of query expansion. The method is evaluated using various retrieval settings, including BM25 and neural reranking. Results show that GenQREnsemble performs better than single-instruction approaches, especially when combined with neural reranking. The study also highlights the benefits of incorporating relevance feedback, demonstrating that GenQREnsembleRF can effectively utilize feedback documents to improve retrieval performance. While generative QR benefits from the ensemble approach, it may introduce increased latency, which is becoming less of a concern with the availability of batch inference for LLMs. The proposed methods could be applied to other settings, such as optimizing different aspects of queries or metrics.
Reach us at info@study.space
[slides and audio] GenQREnsemble%3A Zero-Shot LLM Ensemble Prompting for Generative Query Reformulation