MOSS is an open-source conversational large language model (LLM) with 16 billion parameters, developed by Fudan University. It is designed to perform various instructions in multi-turn interactions with humans. The base model is pre-trained on large-scale unlabeled English, Chinese, and code data. To optimize the model for dialogue, 1.1 million synthetic conversations are generated based on user prompts collected through earlier versions of the model API. Preference-aware training is then performed on preference data annotated from AI feedback. Evaluation results on real-world use cases and academic benchmarks demonstrate the effectiveness of the proposed approaches. Additionally, an effective practice is presented to augment MOSS with several external tools. Through the development of MOSS, a complete technical roadmap for large language models from pre-training, supervised fine-tuning to alignment is established, verifying the feasibility of chatGPT under resource-limited conditions and providing a reference for both the academic and industrial communities. Model weights and code are publicly available at https://github.com/OpenMOSS/MOSS. MOSS is developed in three stages: cross-lingual pre-training, supervised fine-tuning, and preference-aware training. The cross-lingual pre-training involves training the MOSS-base model on a diverse dataset comprising 360B English tokens, 100B Chinese tokens, and 220B code tokens. Supervised fine-tuning is performed with synthetic conversational data and real-world data. Preference-aware training is performed using a preference model to tag model responses with their overall quality. MOSS is also designed to be honest and harmless, with data collected and extended for supervised fine-tuning. Additionally, MOSS is augmented with several external tools, such as a search engine, calculator, equation solver, and text-to-image generator. Automatic evaluations demonstrate significant improvement over its base model and concurrent chat models in terms of model capabilities.MOSS is an open-source conversational large language model (LLM) with 16 billion parameters, developed by Fudan University. It is designed to perform various instructions in multi-turn interactions with humans. The base model is pre-trained on large-scale unlabeled English, Chinese, and code data. To optimize the model for dialogue, 1.1 million synthetic conversations are generated based on user prompts collected through earlier versions of the model API. Preference-aware training is then performed on preference data annotated from AI feedback. Evaluation results on real-world use cases and academic benchmarks demonstrate the effectiveness of the proposed approaches. Additionally, an effective practice is presented to augment MOSS with several external tools. Through the development of MOSS, a complete technical roadmap for large language models from pre-training, supervised fine-tuning to alignment is established, verifying the feasibility of chatGPT under resource-limited conditions and providing a reference for both the academic and industrial communities. Model weights and code are publicly available at https://github.com/OpenMOSS/MOSS. MOSS is developed in three stages: cross-lingual pre-training, supervised fine-tuning, and preference-aware training. The cross-lingual pre-training involves training the MOSS-base model on a diverse dataset comprising 360B English tokens, 100B Chinese tokens, and 220B code tokens. Supervised fine-tuning is performed with synthetic conversational data and real-world data. Preference-aware training is performed using a preference model to tag model responses with their overall quality. MOSS is also designed to be honest and harmless, with data collected and extended for supervised fine-tuning. Additionally, MOSS is augmented with several external tools, such as a search engine, calculator, equation solver, and text-to-image generator. Automatic evaluations demonstrate significant improvement over its base model and concurrent chat models in terms of model capabilities.