This paper investigates the impact of fine-tuning data on the downstream factuality of large language models (LLMs). The authors find that fine-tuning on lesser-known facts, which are poorly stored during pretraining, significantly worsens factuality compared to fine-tuning on well-known facts, even when all facts are seen during pretraining. They provide theoretical proofs and demonstrate this phenomenon across three question-answering benchmarks (PopQA, Entity Questions, and MMLU) and two LLMs (Llama-2.7B and Mistral-7B). The study reveals that fine-tuning on less salient facts can lead the model to ignore subject entity names and output generic plausible responses, even when relevant factual knowledge is encoded in the model. The findings highlight the importance of considering how facts are stored in the pre-trained model when fine-tuning for knowledge-intensive tasks. The paper also introduces the concept of *factual salience*, which measures how well a fact is learned by the model, and shows that fine-tuning on less salient facts can contribute to attention imbalance, leading to incorrect responses. The results suggest that focusing on a smaller subset of well-known facts can significantly improve downstream factuality, even when the entire dataset is used for fine-tuning.This paper investigates the impact of fine-tuning data on the downstream factuality of large language models (LLMs). The authors find that fine-tuning on lesser-known facts, which are poorly stored during pretraining, significantly worsens factuality compared to fine-tuning on well-known facts, even when all facts are seen during pretraining. They provide theoretical proofs and demonstrate this phenomenon across three question-answering benchmarks (PopQA, Entity Questions, and MMLU) and two LLMs (Llama-2.7B and Mistral-7B). The study reveals that fine-tuning on less salient facts can lead the model to ignore subject entity names and output generic plausible responses, even when relevant factual knowledge is encoded in the model. The findings highlight the importance of considering how facts are stored in the pre-trained model when fine-tuning for knowledge-intensive tasks. The paper also introduces the concept of *factual salience*, which measures how well a fact is learned by the model, and shows that fine-tuning on less salient facts can contribute to attention imbalance, leading to incorrect responses. The results suggest that focusing on a smaller subset of well-known facts can significantly improve downstream factuality, even when the entire dataset is used for fine-tuning.