Linear transformers are versatile in-context learners capable of discovering sophisticated optimization algorithms. This paper demonstrates that linear transformers maintain an implicit linear model and can be interpreted as performing a variant of preconditioned gradient descent. When trained on noisy linear regression problems, linear transformers outperform or match the performance of baselines, revealing an optimization algorithm that incorporates momentum and adaptive rescaling based on noise levels. The study shows that even simple linear transformers can learn complex optimization strategies, highlighting their potential in in-context learning. Experiments with different noise variance distributions demonstrate the flexibility and effectiveness of linear transformers, showing that they can adapt to varying noise levels and outperform traditional methods. The findings contribute to the growing body of research on discovering novel algorithms through transformer weight analysis, emphasizing the remarkable versatility of linear transformers in learning and optimization tasks.Linear transformers are versatile in-context learners capable of discovering sophisticated optimization algorithms. This paper demonstrates that linear transformers maintain an implicit linear model and can be interpreted as performing a variant of preconditioned gradient descent. When trained on noisy linear regression problems, linear transformers outperform or match the performance of baselines, revealing an optimization algorithm that incorporates momentum and adaptive rescaling based on noise levels. The study shows that even simple linear transformers can learn complex optimization strategies, highlighting their potential in in-context learning. Experiments with different noise variance distributions demonstrate the flexibility and effectiveness of linear transformers, showing that they can adapt to varying noise levels and outperform traditional methods. The findings contribute to the growing body of research on discovering novel algorithms through transformer weight analysis, emphasizing the remarkable versatility of linear transformers in learning and optimization tasks.