BLOOM is a 176 billion parameter open-access multilingual language model developed by a collaboration of hundreds of researchers through the BigScience initiative. It was trained on the ROOTS corpus, which includes 46 natural and 13 programming languages. BLOOM demonstrates competitive performance on various benchmarks, with enhanced results after multitask prompted finetuning. The model's architecture, training dataset, and engineering strategy are detailed, emphasizing the importance of distributed learning and ethical considerations. The paper also discusses the challenges and solutions in data curation, tokenization, and model architecture, aiming to democratize access to large-scale language models and promote reproducibility and interpretability.BLOOM is a 176 billion parameter open-access multilingual language model developed by a collaboration of hundreds of researchers through the BigScience initiative. It was trained on the ROOTS corpus, which includes 46 natural and 13 programming languages. BLOOM demonstrates competitive performance on various benchmarks, with enhanced results after multitask prompted finetuning. The model's architecture, training dataset, and engineering strategy are detailed, emphasizing the importance of distributed learning and ethical considerations. The paper also discusses the challenges and solutions in data curation, tokenization, and model architecture, aiming to democratize access to large-scale language models and promote reproducibility and interpretability.