**Summary:** This paper investigates the phenomenon of model collapse in language models, which occurs when models are trained on synthetic data generated from previously trained models. The study introduces a statistical model to analyze the impact of recursive training scenarios on language models. It demonstrates that model collapse is inevitable when training solely on synthetic data, but can be mitigated by mixing synthetic and real data. The paper provides theoretical analysis and empirical validation to support these findings. Key contributions include: 1. **Model Collapse Definition**: Model collapse is defined as the convergence of the model's distribution to a Dirac mass, indicating a loss of diversity in the model's output. 2. **Theoretical Analysis**: The paper analyzes two recursive training scenarios: - **Fully Synthetic**: Training on synthetic data generated from the previous generation model. - **Partially Synthetic**: Training on a mixture of synthetic and real data. Theoretical results show that model collapse is inevitable in the fully synthetic case, while the partially synthetic case allows for the mitigation of collapse by including a sufficient amount of real data. 3. **Empirical Validation**: Experiments with transformer-based models and statistical models confirm the theoretical findings. The results show that training on synthetic data alone leads to model collapse, while mixing synthetic and real data helps maintain diversity in the model's output. 4. **Implications**: The study highlights the importance of balancing synthetic and real data in training language models to avoid model collapse. It also suggests that the amount of synthetic data should be significantly smaller than the original data to prevent collapse. The paper concludes that understanding and mitigating model collapse is crucial for the development of robust and diverse language models. The findings provide a theoretical foundation for future research on the dynamics of next-generation language models.**Summary:** This paper investigates the phenomenon of model collapse in language models, which occurs when models are trained on synthetic data generated from previously trained models. The study introduces a statistical model to analyze the impact of recursive training scenarios on language models. It demonstrates that model collapse is inevitable when training solely on synthetic data, but can be mitigated by mixing synthetic and real data. The paper provides theoretical analysis and empirical validation to support these findings. Key contributions include: 1. **Model Collapse Definition**: Model collapse is defined as the convergence of the model's distribution to a Dirac mass, indicating a loss of diversity in the model's output. 2. **Theoretical Analysis**: The paper analyzes two recursive training scenarios: - **Fully Synthetic**: Training on synthetic data generated from the previous generation model. - **Partially Synthetic**: Training on a mixture of synthetic and real data. Theoretical results show that model collapse is inevitable in the fully synthetic case, while the partially synthetic case allows for the mitigation of collapse by including a sufficient amount of real data. 3. **Empirical Validation**: Experiments with transformer-based models and statistical models confirm the theoretical findings. The results show that training on synthetic data alone leads to model collapse, while mixing synthetic and real data helps maintain diversity in the model's output. 4. **Implications**: The study highlights the importance of balancing synthetic and real data in training language models to avoid model collapse. It also suggests that the amount of synthetic data should be significantly smaller than the original data to prevent collapse. The paper concludes that understanding and mitigating model collapse is crucial for the development of robust and diverse language models. The findings provide a theoretical foundation for future research on the dynamics of next-generation language models.