This paper advocates for the adoption of retrieval-augmented language models (RAGs) as the next generation of language models, addressing the limitations of parametric language models (LMs). Parametric LMs, while showing remarkable flexibility and capability, suffer from issues such as hallucinations, difficulty in adapting to new data distributions, and a lack of verifiability. RAGs, by incorporating large-scale datastores during inference, can be more reliable, adaptable, and attributable. Despite their potential, RAGs have not been widely adopted due to several challenges, including limited interaction between retrieval and LM components, lack of infrastructure for scaling, and difficulties in leveraging helpful text beyond knowledge-intensive tasks. The paper proposes a roadmap for developing general-purpose RAGs, involving rethinking datastores and retrievers, improving retriever-LM interaction, and investing in efficient training and inference infrastructure. It highlights the need for interdisciplinary efforts to advance RAGs, including better understanding of relevance, improving datastore quality, enhancing interaction between retrieval and LM components, and developing efficient training and inference methods. The ultimate goal is to unlock the full capabilities of RAGs and expand their applications across diverse domains.This paper advocates for the adoption of retrieval-augmented language models (RAGs) as the next generation of language models, addressing the limitations of parametric language models (LMs). Parametric LMs, while showing remarkable flexibility and capability, suffer from issues such as hallucinations, difficulty in adapting to new data distributions, and a lack of verifiability. RAGs, by incorporating large-scale datastores during inference, can be more reliable, adaptable, and attributable. Despite their potential, RAGs have not been widely adopted due to several challenges, including limited interaction between retrieval and LM components, lack of infrastructure for scaling, and difficulties in leveraging helpful text beyond knowledge-intensive tasks. The paper proposes a roadmap for developing general-purpose RAGs, involving rethinking datastores and retrievers, improving retriever-LM interaction, and investing in efficient training and inference infrastructure. It highlights the need for interdisciplinary efforts to advance RAGs, including better understanding of relevance, improving datastore quality, enhancing interaction between retrieval and LM components, and developing efficient training and inference methods. The ultimate goal is to unlock the full capabilities of RAGs and expand their applications across diverse domains.