The paper "Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" by Jinhyuk Lee et al. from Google DeepMind introduces the Long-Context Frontiers (LOFT) benchmark to evaluate the capabilities of long-context language models (LCLMs) on real-world tasks. LOFT consists of six tasks across text, visual, and audio modalities, designed to push LCLMs to their limits and assess their performance in in-context retrieval, reasoning, and many-shot learning on corpora up to millions of tokens. The authors find that LCLMs can rival state-of-the-art retrieval and RAG systems despite never being explicitly trained for these tasks, but they still face challenges in complex compositional reasoning tasks like SQL-like queries. Prompting strategies significantly influence performance, highlighting the need for further research. The paper also introduces Corpus-in-Context (CiC) prompting, a novel approach that leverages LCLMs' ability to process entire corpora within their context window, and evaluates three state-of-the-art LCLMs—Gemini 1.5 Pro, GPT-4o, and Claude 3 Opus—on LOFT. Initial results show that LCLMs can match or surpass specialized models in certain tasks, but there is still room for improvement in long-context reasoning. The paper concludes by emphasizing the potential of LCLMs to revolutionize tasks traditionally relying on external tools and the need for continued research to enhance their robustness and efficiency.The paper "Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" by Jinhyuk Lee et al. from Google DeepMind introduces the Long-Context Frontiers (LOFT) benchmark to evaluate the capabilities of long-context language models (LCLMs) on real-world tasks. LOFT consists of six tasks across text, visual, and audio modalities, designed to push LCLMs to their limits and assess their performance in in-context retrieval, reasoning, and many-shot learning on corpora up to millions of tokens. The authors find that LCLMs can rival state-of-the-art retrieval and RAG systems despite never being explicitly trained for these tasks, but they still face challenges in complex compositional reasoning tasks like SQL-like queries. Prompting strategies significantly influence performance, highlighting the need for further research. The paper also introduces Corpus-in-Context (CiC) prompting, a novel approach that leverages LCLMs' ability to process entire corpora within their context window, and evaluates three state-of-the-art LCLMs—Gemini 1.5 Pro, GPT-4o, and Claude 3 Opus—on LOFT. Initial results show that LCLMs can match or surpass specialized models in certain tasks, but there is still room for improvement in long-context reasoning. The paper concludes by emphasizing the potential of LCLMs to revolutionize tasks traditionally relying on external tools and the need for continued research to enhance their robustness and efficiency.