C–RAG: Certified Generation Risks for Retrieval-Augmented Language Models This paper introduces C–RAG, a novel framework for certifying generation risks in retrieval-augmented language models (RAG). RAG models enhance the credibility of language models (LLMs) by incorporating external knowledge through retrieval. However, the theoretical understanding of their generation risks remains limited. C–RAG provides conformal risk analysis for RAG models, offering provable guarantees on generation risks. It certifies an upper confidence bound of generation risks, referred to as conformal generation risk. Theoretical guarantees are provided for general bounded risk functions under test-time distribution shifts. The paper proves that RAG achieves a lower conformal generation risk than a single LLM when the quality of the retrieval model and transformer is non-trivial. Empirical results demonstrate the soundness and tightness of the conformal generation risk guarantees across four widely-used NLP datasets on four state-of-the-art retrieval models. The paper also evaluates the conformal generation risk for different SOTA retrieval models, showing that text-embedding-ada-002 and supervised fine-tuned embedding models outperform other baselines in achieving low conformal generation risks. The paper also provides theoretical analysis of C–RAG, showing that RAG achieves a lower conformal generation risk than LLMs without retrieval. The paper further extends the analysis to test-time distribution shifts, providing the first generation risk guarantee under test-time distribution shifts for general bounded risk functions. The paper evaluates C–RAG on four datasets using different retrieval models, finding that RAG reduces the conformal generation risks for different retrieval models, and that the conformal generation risk under distribution shifts is empirically sound and tight. The paper also shows that multi-dimensional RAG configurations maintain sound and tight conformal generation risks. Finally, the paper validates the valid configurations given desired risk levels, showing that the empirical risks of generated configurations are consistently below the given conformal generation risk. The paper concludes that C–RAG provides conformal generation risk guarantees for RAG models, and that RAG reduces conformal generation risks of a single LLM.C–RAG: Certified Generation Risks for Retrieval-Augmented Language Models This paper introduces C–RAG, a novel framework for certifying generation risks in retrieval-augmented language models (RAG). RAG models enhance the credibility of language models (LLMs) by incorporating external knowledge through retrieval. However, the theoretical understanding of their generation risks remains limited. C–RAG provides conformal risk analysis for RAG models, offering provable guarantees on generation risks. It certifies an upper confidence bound of generation risks, referred to as conformal generation risk. Theoretical guarantees are provided for general bounded risk functions under test-time distribution shifts. The paper proves that RAG achieves a lower conformal generation risk than a single LLM when the quality of the retrieval model and transformer is non-trivial. Empirical results demonstrate the soundness and tightness of the conformal generation risk guarantees across four widely-used NLP datasets on four state-of-the-art retrieval models. The paper also evaluates the conformal generation risk for different SOTA retrieval models, showing that text-embedding-ada-002 and supervised fine-tuned embedding models outperform other baselines in achieving low conformal generation risks. The paper also provides theoretical analysis of C–RAG, showing that RAG achieves a lower conformal generation risk than LLMs without retrieval. The paper further extends the analysis to test-time distribution shifts, providing the first generation risk guarantee under test-time distribution shifts for general bounded risk functions. The paper evaluates C–RAG on four datasets using different retrieval models, finding that RAG reduces the conformal generation risks for different retrieval models, and that the conformal generation risk under distribution shifts is empirically sound and tight. The paper also shows that multi-dimensional RAG configurations maintain sound and tight conformal generation risks. Finally, the paper validates the valid configurations given desired risk levels, showing that the empirical risks of generated configurations are consistently below the given conformal generation risk. The paper concludes that C–RAG provides conformal generation risk guarantees for RAG models, and that RAG reduces conformal generation risks of a single LLM.