The paper "Instruction-tuned Language Models are Better Knowledge Learners" by Zhengbao Jiang et al. explores how to enhance large language models (LLMs) in absorbing and answering questions about new documents. The authors find that while LLMs can answer questions after continued pre-training on new documents, their performance is limited, even when the perplexity of the documents is minimized. They hypothesize that exposing LLMs to question-answer (QA) pairs before continued pre-training on documents can improve their ability to encode knowledge from complex documents. To test this hypothesis, the authors propose a method called Pre-instruction-tuning (PIT), which involves instruction-tuning on QA pairs before training on documents. Extensive experiments using the Llama-2 model demonstrate that PIT significantly enhances the LLMs' ability to absorb knowledge from new documents, outperforming standard instruction-tuning by 17.8% on the Llama-2 7B model and 16.3% on the Llama-2 70B model. The paper also includes detailed ablation studies to identify the key contributors to the improved performance of PIT. Additionally, the authors evaluate PIT's effectiveness across different domains and real-world scenarios, showing that it can generalize well to new domains and answer questions from real users. Overall, the study highlights the importance of prioritizing understanding how knowledge is accessed through QA pairs before encoding it from complex documents, providing a practical approach to improving LLMs' knowledge acquisition capabilities.The paper "Instruction-tuned Language Models are Better Knowledge Learners" by Zhengbao Jiang et al. explores how to enhance large language models (LLMs) in absorbing and answering questions about new documents. The authors find that while LLMs can answer questions after continued pre-training on new documents, their performance is limited, even when the perplexity of the documents is minimized. They hypothesize that exposing LLMs to question-answer (QA) pairs before continued pre-training on documents can improve their ability to encode knowledge from complex documents. To test this hypothesis, the authors propose a method called Pre-instruction-tuning (PIT), which involves instruction-tuning on QA pairs before training on documents. Extensive experiments using the Llama-2 model demonstrate that PIT significantly enhances the LLMs' ability to absorb knowledge from new documents, outperforming standard instruction-tuning by 17.8% on the Llama-2 7B model and 16.3% on the Llama-2 70B model. The paper also includes detailed ablation studies to identify the key contributors to the improved performance of PIT. Additionally, the authors evaluate PIT's effectiveness across different domains and real-world scenarios, showing that it can generalize well to new domains and answer questions from real users. Overall, the study highlights the importance of prioritizing understanding how knowledge is accessed through QA pairs before encoding it from complex documents, providing a practical approach to improving LLMs' knowledge acquisition capabilities.