Instruction-tuned language models (LLMs) are more effective at learning new knowledge compared to standard instruction-tuning. This study investigates how to improve LLMs' ability to absorb knowledge from new documents. The key finding is that pre-instruction-tuning (PIT), which involves training LLMs on question-answer (QA) pairs before continuing pre-training on documents, significantly enhances their ability to learn from new documents. This approach outperforms standard instruction-tuning by 17.8% on Llama-2 7B and 16.3% on Llama-2 70B. The effectiveness of PIT is attributed to its ability to prioritize learning how to access knowledge over learning to encode knowledge from documents. The study also demonstrates that PIT improves performance across different domains, indicating its potential for broader generalization. The research highlights the importance of aligning knowledge access with knowledge encoding during training. The proposed method, PIT, is shown to be more effective than standard instruction-tuning in eliciting knowledge from LLMs. The study also addresses the "perplexity curse," where minimizing document perplexity does not necessarily lead to better knowledge acquisition. The results suggest that training LLMs on QA pairs before documents helps them better understand how to access knowledge, leading to improved performance in answering questions. The study uses a dataset called Wiki2023, which includes documents and QA pairs related to 2023. The experiments show that PIT significantly improves the ability of LLMs to absorb knowledge from new documents, outperforming standard instruction-tuning. The study also explores different training strategies and ablation studies to identify the key factors contributing to the effectiveness of PIT. Overall, the findings suggest that pre-instruction-tuning is a promising approach for improving LLMs' ability to learn from new documents.Instruction-tuned language models (LLMs) are more effective at learning new knowledge compared to standard instruction-tuning. This study investigates how to improve LLMs' ability to absorb knowledge from new documents. The key finding is that pre-instruction-tuning (PIT), which involves training LLMs on question-answer (QA) pairs before continuing pre-training on documents, significantly enhances their ability to learn from new documents. This approach outperforms standard instruction-tuning by 17.8% on Llama-2 7B and 16.3% on Llama-2 70B. The effectiveness of PIT is attributed to its ability to prioritize learning how to access knowledge over learning to encode knowledge from documents. The study also demonstrates that PIT improves performance across different domains, indicating its potential for broader generalization. The research highlights the importance of aligning knowledge access with knowledge encoding during training. The proposed method, PIT, is shown to be more effective than standard instruction-tuning in eliciting knowledge from LLMs. The study also addresses the "perplexity curse," where minimizing document perplexity does not necessarily lead to better knowledge acquisition. The results suggest that training LLMs on QA pairs before documents helps them better understand how to access knowledge, leading to improved performance in answering questions. The study uses a dataset called Wiki2023, which includes documents and QA pairs related to 2023. The experiments show that PIT significantly improves the ability of LLMs to absorb knowledge from new documents, outperforming standard instruction-tuning. The study also explores different training strategies and ablation studies to identify the key factors contributing to the effectiveness of PIT. Overall, the findings suggest that pre-instruction-tuning is a promising approach for improving LLMs' ability to learn from new documents.