Do Language Models Plan Ahead for Future Tokens? Language models may "think ahead" during inference by preparing information in hidden states that is useful for future predictions. Two hypotheses explain this phenomenon: pre-caching, where models compute features irrelevant to the current task but useful for the future, and breadcrumbs, where features relevant to the current task are also beneficial for future predictions. Experiments with myopic training, which prevents gradients from past steps, suggest pre-caching occurs in synthetic data but is less prominent in smaller models like GPT-2, supporting the breadcrumbs hypothesis. However, pre-caching becomes more significant with larger models, indicating they may "plan for the future." The myopia gap, measuring the difference in performance between myopic and standard models, highlights the extent of pre-caching. Synthetic experiments show that transformers learn to pre-cache in tasks requiring it, while natural language models show mixed results, with smaller models relying more on breadcrumbs. Larger models exhibit greater pre-caching, suggesting they can anticipate future needs. This research contributes to understanding how transformers process information and their potential for future planning.Do Language Models Plan Ahead for Future Tokens? Language models may "think ahead" during inference by preparing information in hidden states that is useful for future predictions. Two hypotheses explain this phenomenon: pre-caching, where models compute features irrelevant to the current task but useful for the future, and breadcrumbs, where features relevant to the current task are also beneficial for future predictions. Experiments with myopic training, which prevents gradients from past steps, suggest pre-caching occurs in synthetic data but is less prominent in smaller models like GPT-2, supporting the breadcrumbs hypothesis. However, pre-caching becomes more significant with larger models, indicating they may "plan for the future." The myopia gap, measuring the difference in performance between myopic and standard models, highlights the extent of pre-caching. Synthetic experiments show that transformers learn to pre-cache in tasks requiring it, while natural language models show mixed results, with smaller models relying more on breadcrumbs. Larger models exhibit greater pre-caching, suggesting they can anticipate future needs. This research contributes to understanding how transformers process information and their potential for future planning.