This paper proposes a conversation-level Retrieval-Augmented Generation (ConvRAG) approach to enhance conversational question answering (CQA). ConvRAG incorporates fine-grained retrieval augmentation and self-check mechanisms to address the challenges of question representation and knowledge acquisition in conversational settings. The approach consists of three components: a conversational question refiner, a fine-grained retriever, and a self-check based response generator. The question refiner improves understanding by reformulating questions and extracting keywords. The fine-grained retriever retrieves relevant information from the web, while the self-check mechanism filters out unhelpful information to improve response generation. The method is evaluated on two datasets: a benchmark dataset and a newly constructed Chinese CQA dataset with diverse topics and features. Experimental results show that ConvRAG outperforms state-of-the-art baselines and industry production systems in most cases. The paper also introduces a Chinese CQA dataset with reformulated questions, extracted keywords, retrieved paragraphs, and their helpfulness, which facilitates further research in RAG-enhanced CQA. The approach demonstrates the effectiveness of integrating LLMs with RAG at the conversation level, and the three components of ConvRAG effectively address the challenges of question representation and knowledge acquisition in CQA.This paper proposes a conversation-level Retrieval-Augmented Generation (ConvRAG) approach to enhance conversational question answering (CQA). ConvRAG incorporates fine-grained retrieval augmentation and self-check mechanisms to address the challenges of question representation and knowledge acquisition in conversational settings. The approach consists of three components: a conversational question refiner, a fine-grained retriever, and a self-check based response generator. The question refiner improves understanding by reformulating questions and extracting keywords. The fine-grained retriever retrieves relevant information from the web, while the self-check mechanism filters out unhelpful information to improve response generation. The method is evaluated on two datasets: a benchmark dataset and a newly constructed Chinese CQA dataset with diverse topics and features. Experimental results show that ConvRAG outperforms state-of-the-art baselines and industry production systems in most cases. The paper also introduces a Chinese CQA dataset with reformulated questions, extracted keywords, retrieved paragraphs, and their helpfulness, which facilitates further research in RAG-enhanced CQA. The approach demonstrates the effectiveness of integrating LLMs with RAG at the conversation level, and the three components of ConvRAG effectively address the challenges of question representation and knowledge acquisition in CQA.