This paper addresses the issue of trustworthiness in large language models (LLMs), particularly their hallucinations and misalignments. Retrieval-augmented language models (RAG) are proposed to enhance credibility by grounding external knowledge, but the theoretical understanding of their generation risks remains unexplored. The authors introduce C-RAG, a novel framework to certify generation risks for RAG models. They provide conformal risk analysis for RAG models and certify an upper confidence bound of generation risks, referred to as *conformal generation risk*. The paper also offers theoretical guarantees on conformal generation risks for general bounded risk functions under test distribution shifts. Empirical results on four widely-used NLP datasets with four state-of-the-art retrieval models demonstrate the soundness and tightness of the conformal generation risk guarantees. The study shows that RAG consistently achieves lower conformal generation risks than vanilla LLMs, both in scenarios with and without distribution shifts.This paper addresses the issue of trustworthiness in large language models (LLMs), particularly their hallucinations and misalignments. Retrieval-augmented language models (RAG) are proposed to enhance credibility by grounding external knowledge, but the theoretical understanding of their generation risks remains unexplored. The authors introduce C-RAG, a novel framework to certify generation risks for RAG models. They provide conformal risk analysis for RAG models and certify an upper confidence bound of generation risks, referred to as *conformal generation risk*. The paper also offers theoretical guarantees on conformal generation risks for general bounded risk functions under test distribution shifts. Empirical results on four widely-used NLP datasets with four state-of-the-art retrieval models demonstrate the soundness and tightness of the conformal generation risk guarantees. The study shows that RAG consistently achieves lower conformal generation risks than vanilla LLMs, both in scenarios with and without distribution shifts.