Scaling AstroLLaMA with Conversational and Diverse Datasets

Scaling AstroLLaMA with Conversational and Diverse Datasets

January 5, 2024 | Ernest Perkowski, Rui Pan, Tuan Dung Nguyen, Yuan-Sen Ting, Sandor Kruk, Tong Zhang, Charlie O'Neill, Maja Jablonska, Zechang Sun, Michael J. Smith, Huiing Liu, Kevin Schawinski, Kartheik Iyer, Ioana Ciucă
The paper introduces AstroLLaMA-Chat, an enhanced version of the AstroLLaMA model, which is trained on a curated set of astronomy corpora, including abstracts, introductions, and conclusions of arXiv papers. This model is designed to improve performance in specialized astronomy question-answering tasks. The model is fine-tuned on a domain-specific conversational dataset, and the training process involves a diverse mix of datasets, including the LIMA dataset, Open Orca, and UltraChat. The training is conducted using the LMFlow framework with advanced techniques like Flash Attention and ZeRO Optimization, leading to significant efficiency gains. The model is available at https://huggingface.co/universeTBD and is the first open-source conversational AI tool tailored for the astronomy community. AstroLLaMA-Chat demonstrates superior performance in highly specialized topics, such as the dimensionality of elemental abundance space and cosmological parity violation, compared to general-purpose models like GPT-4 and LLaMA-2. While it may not consistently outperform these models in general astronomy-related Q&A, it excels in niche areas. The model is also capable of completing abstracts and introductions of astronomy articles more effectively than LLaMA-2. The paper highlights the benefits of continual pre-training on a dedicated astronomy corpus, even with a relatively modest model like the 7B parameter AstroLLaMA. The research also emphasizes the importance of domain-specific training and alignment techniques to improve the model's performance in multi-turn conversations. The authors hope this research will inspire more astronomers to explore the fine-tuning of smaller models with modest computational resources. The models are available on the Hugging Face demo playground, allowing users to rate responses and provide feedback, which is crucial for advancing this field of study.The paper introduces AstroLLaMA-Chat, an enhanced version of the AstroLLaMA model, which is trained on a curated set of astronomy corpora, including abstracts, introductions, and conclusions of arXiv papers. This model is designed to improve performance in specialized astronomy question-answering tasks. The model is fine-tuned on a domain-specific conversational dataset, and the training process involves a diverse mix of datasets, including the LIMA dataset, Open Orca, and UltraChat. The training is conducted using the LMFlow framework with advanced techniques like Flash Attention and ZeRO Optimization, leading to significant efficiency gains. The model is available at https://huggingface.co/universeTBD and is the first open-source conversational AI tool tailored for the astronomy community. AstroLLaMA-Chat demonstrates superior performance in highly specialized topics, such as the dimensionality of elemental abundance space and cosmological parity violation, compared to general-purpose models like GPT-4 and LLaMA-2. While it may not consistently outperform these models in general astronomy-related Q&A, it excels in niche areas. The model is also capable of completing abstracts and introductions of astronomy articles more effectively than LLaMA-2. The paper highlights the benefits of continual pre-training on a dedicated astronomy corpus, even with a relatively modest model like the 7B parameter AstroLLaMA. The research also emphasizes the importance of domain-specific training and alignment techniques to improve the model's performance in multi-turn conversations. The authors hope this research will inspire more astronomers to explore the fine-tuning of smaller models with modest computational resources. The models are available on the Hugging Face demo playground, allowing users to rate responses and provide feedback, which is crucial for advancing this field of study.
Reach us at info@study.space
[slides and audio] AstroLLaMA-Chat%3A Scaling AstroLLaMA with Conversational and Diverse Datasets