HEAT DEATH OF GENERATIVE MODELS IN CLOSED-LOOP LEARNING

HEAT DEATH OF GENERATIVE MODELS IN CLOSED-LOOP LEARNING

2 Apr 2024 | MATTEO MARCHI, STEFANO SOATTO, PRATIK CHAUDHARI, AND PAULO TABUADA
This paper investigates the phenomenon of "generative closed-loop learning," where a generative model is trained on data it generates itself. The study focuses on the stability of the training process and the long-term behavior of the model's probability distribution. The key finding is that when a generative model is trained on its own generated data, it can lead to a degradation of the model's performance, with the probability distribution collapsing to a small set of outputs or becoming uniform over a large set of outputs. This degradation is exacerbated by the use of a "temperature" parameter, which controls the randomness of the model's outputs. The paper shows that unless a sufficient amount of external data is introduced at each training iteration, any non-trivial temperature leads to asymptotic degeneration of the learning process. The analysis is based on dynamical systems theory and shows that the model's probability distribution converges to a neighborhood of the uniform probability vector or to a region where most outputs have very low probability mass. The results highlight the risks of data self-ingestion in generative models, particularly in the context of large-scale deep learning models trained on internet data. The paper also discusses the implications of these findings for the long-term stability and effectiveness of generative models.This paper investigates the phenomenon of "generative closed-loop learning," where a generative model is trained on data it generates itself. The study focuses on the stability of the training process and the long-term behavior of the model's probability distribution. The key finding is that when a generative model is trained on its own generated data, it can lead to a degradation of the model's performance, with the probability distribution collapsing to a small set of outputs or becoming uniform over a large set of outputs. This degradation is exacerbated by the use of a "temperature" parameter, which controls the randomness of the model's outputs. The paper shows that unless a sufficient amount of external data is introduced at each training iteration, any non-trivial temperature leads to asymptotic degeneration of the learning process. The analysis is based on dynamical systems theory and shows that the model's probability distribution converges to a neighborhood of the uniform probability vector or to a region where most outputs have very low probability mass. The results highlight the risks of data self-ingestion in generative models, particularly in the context of large-scale deep learning models trained on internet data. The paper also discusses the implications of these findings for the long-term stability and effectiveness of generative models.
Reach us at info@study.space