SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature

SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature

20 Aug 2024 | David Wadden, Kejian Shi, Jacob Morrison, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, Doug Downey, Hannaneh Hajishirzi, Arman Cohan
**SciRIFF (Scientific Resource for Instruction-Following and Finetuning)** is a dataset designed to enhance the instruction-following capabilities of large language models (LLMs) in scientific literature understanding. The dataset includes 137K demonstrations for 54 tasks across five scientific domains: information extraction, summarization, question answering, claim verification, and classification. Each task features long input contexts and requires structured outputs, making it suitable for real-world scientific use cases. The authors developed a sample-efficient strategy to adapt a general instruction-following model, SciFiuU, by performing additional finetuning on a mix of general-domain and SciRIFF demonstrations. Evaluations on nine held-out scientific tasks show that SciFiuU improves over a strong LLM baseline by 28.1% at the 7B scale and 6.5% at the 70B scale, while maintaining general instruction-following performance within 2% of the baseline. The dataset, model checkpoints, and evaluation code are publicly released to facilitate further research and development of LLMs for scientific literature understanding. The work highlights the importance of structured outputs and long input contexts in scientific literature understanding tasks, and demonstrates the potential of SciRIFF in advancing scientific progress through LLMs.**SciRIFF (Scientific Resource for Instruction-Following and Finetuning)** is a dataset designed to enhance the instruction-following capabilities of large language models (LLMs) in scientific literature understanding. The dataset includes 137K demonstrations for 54 tasks across five scientific domains: information extraction, summarization, question answering, claim verification, and classification. Each task features long input contexts and requires structured outputs, making it suitable for real-world scientific use cases. The authors developed a sample-efficient strategy to adapt a general instruction-following model, SciFiuU, by performing additional finetuning on a mix of general-domain and SciRIFF demonstrations. Evaluations on nine held-out scientific tasks show that SciFiuU improves over a strong LLM baseline by 28.1% at the 7B scale and 6.5% at the 70B scale, while maintaining general instruction-following performance within 2% of the baseline. The dataset, model checkpoints, and evaluation code are publicly released to facilitate further research and development of LLMs for scientific literature understanding. The work highlights the importance of structured outputs and long input contexts in scientific literature understanding tasks, and demonstrates the potential of SciRIFF in advancing scientific progress through LLMs.
Reach us at info@study.space