11 Mar 2024 | Carlos Lassance, Hervé Déjean, Thibault Formal, Stéphane Clinchant
This technical report introduces SPLADE-v3, a new version of the SPLADE library, which includes several improvements to the training structure and model performance. The authors detail enhancements such as multiple negatives per batch, better distillation scores using an ensemble of cross-encoder re-rankers, and fine-tuning from SPLADE+++SelfDistil. SPLADE-v3 is evaluated using a meta-analysis over 40 query sets, demonstrating statistically significant improvements over BM25 and SPLADE++SelfDistil. It also performs well against cross-encoder re-rankers like MiniLM and DeBERTaV3. The report also releases three variants of SPLADE-v3: SPLADE-v3-DistilBERT, SPLADE-v3-Lexical, and SPLADE-v3-Doc, each with different training starting points and efficiency trade-offs. Overall, SPLADE-v3 shows superior effectiveness, especially in zero-shot settings, and outperforms BM25 and some re-rankers.This technical report introduces SPLADE-v3, a new version of the SPLADE library, which includes several improvements to the training structure and model performance. The authors detail enhancements such as multiple negatives per batch, better distillation scores using an ensemble of cross-encoder re-rankers, and fine-tuning from SPLADE+++SelfDistil. SPLADE-v3 is evaluated using a meta-analysis over 40 query sets, demonstrating statistically significant improvements over BM25 and SPLADE++SelfDistil. It also performs well against cross-encoder re-rankers like MiniLM and DeBERTaV3. The report also releases three variants of SPLADE-v3: SPLADE-v3-DistilBERT, SPLADE-v3-Lexical, and SPLADE-v3-Doc, each with different training starting points and efficiency trade-offs. Overall, SPLADE-v3 shows superior effectiveness, especially in zero-shot settings, and outperforms BM25 and some re-rankers.