July 14–18, 2024 | Chuan Meng, Negar Arabzadeh, Arian Askari, Mohammad Aliannejadi, Maarten de Rijke
The paper explores the application of Ranked List Truncation (RLT) in the context of re-ranking, particularly for large language model (LLM)-based re-ranking. RLT involves optimizing the re-ranking process by truncating the retrieved list to improve efficiency and effectiveness. The authors conduct a comprehensive study on 8 RLT methods and pipelines involving 3 retrievers and 2 re-rankers, using the TREC 2019 and 2020 deep learning tracks. They find that supervised RLT methods do not consistently outperform unsupervised methods, and potential fixed re-ranking depths can achieve similar effectiveness/efficiency trade-offs. The choice of retriever significantly impacts RLT performance, with effective retrievers like SPLADE++ and RepLLaMA showing better effectiveness/efficiency trade-offs at a fixed depth of 20. The type of re-ranker (LLM or pre-trained LM-based) does not appear to influence the findings. The paper also highlights the need for enhancing RLT methods to improve re-ranking efficiency and effectiveness, particularly in handling query-specific re-ranking cut-offs.The paper explores the application of Ranked List Truncation (RLT) in the context of re-ranking, particularly for large language model (LLM)-based re-ranking. RLT involves optimizing the re-ranking process by truncating the retrieved list to improve efficiency and effectiveness. The authors conduct a comprehensive study on 8 RLT methods and pipelines involving 3 retrievers and 2 re-rankers, using the TREC 2019 and 2020 deep learning tracks. They find that supervised RLT methods do not consistently outperform unsupervised methods, and potential fixed re-ranking depths can achieve similar effectiveness/efficiency trade-offs. The choice of retriever significantly impacts RLT performance, with effective retrievers like SPLADE++ and RepLLaMA showing better effectiveness/efficiency trade-offs at a fixed depth of 20. The type of re-ranker (LLM or pre-trained LM-based) does not appear to influence the findings. The paper also highlights the need for enhancing RLT methods to improve re-ranking efficiency and effectiveness, particularly in handling query-specific re-ranking cut-offs.