The paper introduces Sparse RAG, a novel paradigm designed to address the challenges of increased input length and latency in Retrieval-Augmented Generation (RAG) systems. Sparse RAG aims to reduce computational costs by encoding retrieved documents in parallel and selectively decoding the output by attending only to highly relevant caches. This approach combines the assessment of each individual document with the generation process, enhancing both efficiency and generation quality. The method is evaluated on two datasets, PopQA and Biography, demonstrating its ability to achieve a balanced trade-off between generation quality and computational efficiency across short- and long-form generation tasks. sparse RAG outperforms dense RAG and PCW RAG in terms of both quality and latency, making it a versatile and effective solution for RAG systems.The paper introduces Sparse RAG, a novel paradigm designed to address the challenges of increased input length and latency in Retrieval-Augmented Generation (RAG) systems. Sparse RAG aims to reduce computational costs by encoding retrieved documents in parallel and selectively decoding the output by attending only to highly relevant caches. This approach combines the assessment of each individual document with the generation process, enhancing both efficiency and generation quality. The method is evaluated on two datasets, PopQA and Biography, demonstrating its ability to achieve a balanced trade-off between generation quality and computational efficiency across short- and long-form generation tasks. sparse RAG outperforms dense RAG and PCW RAG in terms of both quality and latency, making it a versatile and effective solution for RAG systems.