This paper addresses the challenge of Conversational Question Answering (CQA) by proposing a Conversation-level Retrieval-Augmented Generation (ConvRAG) approach. ConvRAG integrates fine-grained retrieval-augmentation and self-check mechanisms to enhance the performance of large language models (LLMs) in CQA. The approach consists of three main components: a Conversational Question Refiner, a Fine-Grained Retriever, and a Self-Check based Response Generator. The Conversational Question Refiner refines questions to better understand their context, the Fine-Grained Retriever retrieves relevant information from the web, and the Self-Check based Response Generator filters out irrelevant information to generate more accurate responses. Extensive experiments on benchmark datasets and a newly constructed Chinese CQA dataset demonstrate the effectiveness of ConvRAG, showing significant improvements over state-of-the-art baselines. The paper also introduces a Chinese CQA dataset with new features, including reformulated questions, extracted keywords, retrieved paragraphs, and their helpfulness, which facilitates further research in RAG-enhanced CQA.This paper addresses the challenge of Conversational Question Answering (CQA) by proposing a Conversation-level Retrieval-Augmented Generation (ConvRAG) approach. ConvRAG integrates fine-grained retrieval-augmentation and self-check mechanisms to enhance the performance of large language models (LLMs) in CQA. The approach consists of three main components: a Conversational Question Refiner, a Fine-Grained Retriever, and a Self-Check based Response Generator. The Conversational Question Refiner refines questions to better understand their context, the Fine-Grained Retriever retrieves relevant information from the web, and the Self-Check based Response Generator filters out irrelevant information to generate more accurate responses. Extensive experiments on benchmark datasets and a newly constructed Chinese CQA dataset demonstrate the effectiveness of ConvRAG, showing significant improvements over state-of-the-art baselines. The paper also introduces a Chinese CQA dataset with new features, including reformulated questions, extracted keywords, retrieved paragraphs, and their helpfulness, which facilitates further research in RAG-enhanced CQA.