This paper presents seven failure points when engineering a Retrieval Augmented Generation (RAG) system, based on three case studies from research, education, and biomedical domains. RAG systems combine information retrieval with large language models (LLMs) to generate accurate and contextually relevant answers. However, they face limitations inherent to information retrieval systems and reliance on LLMs. The study identifies seven key failure points: 1) Missing content, 2) Missed top-ranked documents, 3) Not in context - consolidation strategy limitations, 4) Not extracted, 5) Wrong format, 6) Incorrect specificity, and 7) Incomplete. The paper also highlights that validation of a RAG system is only feasible during operation, and the robustness of a RAG system evolves rather than being designed in at the start. The study emphasizes the importance of chunking and embeddings, RAG versus finetuning, and testing and monitoring RAG systems. The paper concludes with a list of potential research directions for the software engineering community. The findings provide a guide for practitioners and highlight the challenges faced when implementing RAG systems. The study also includes future research directions for RAG systems related to chunking and embeddings, RAG versus finetuning, and testing and monitoring. The paper is an important step towards building robust RAG systems.This paper presents seven failure points when engineering a Retrieval Augmented Generation (RAG) system, based on three case studies from research, education, and biomedical domains. RAG systems combine information retrieval with large language models (LLMs) to generate accurate and contextually relevant answers. However, they face limitations inherent to information retrieval systems and reliance on LLMs. The study identifies seven key failure points: 1) Missing content, 2) Missed top-ranked documents, 3) Not in context - consolidation strategy limitations, 4) Not extracted, 5) Wrong format, 6) Incorrect specificity, and 7) Incomplete. The paper also highlights that validation of a RAG system is only feasible during operation, and the robustness of a RAG system evolves rather than being designed in at the start. The study emphasizes the importance of chunking and embeddings, RAG versus finetuning, and testing and monitoring RAG systems. The paper concludes with a list of potential research directions for the software engineering community. The findings provide a guide for practitioners and highlight the challenges faced when implementing RAG systems. The study also includes future research directions for RAG systems related to chunking and embeddings, RAG versus finetuning, and testing and monitoring. The paper is an important step towards building robust RAG systems.