This paper addresses the problem of entropy-regularized fine-tuning in continuous-time diffusion models, a method recently proposed by Uehara et al. (2024). The approach uses stochastic control to generate samples, incorporating an entropy regularizer to mitigate reward collapse. The paper provides a rigorous treatment of this theory and extends it to fine-tuning with general $f$-divergence regularizers. Key contributions include: 1. **Entropy-regularized Fine-tuning**: The paper develops a stochastic control framework to emulate the fine-tuned distribution, which is an exponential tilting of the pretrained model. This approach decouples the control and initial distribution, allowing for more flexible sampling. 2. **Stochastic Control Problem**: The problem is formulated as a stochastic control problem, where both the control and initial distribution are decision variables. The optimal control and initial distribution are derived, and their properties are analyzed. 3. **$f$-divergence Regularization**: The paper generalizes the entropy-regularized fine-tuning to $f$-divergence regularization, providing a unified framework for various divergence measures. This extension maintains the benefits of entropy regularization while allowing for more flexible divergence choices. 4. **Theoretical Results**: The paper includes theoretical results on the performance of the fine-tuned distribution, including bounds on the total variation distance to the target distribution. These bounds help understand the trade-offs between reward performance and diversity. 5. **Practical Implications**: The paper discusses the practical implications of the proposed methods, including the use of neural ODEs/SDEs for solving the control problem and the potential for real-world applications such as image synthesis and protein sequence generation. The paper concludes with directions for future research, including the extension of the theory to viscosity solutions and the exploration of direct $f$-divergence regularization.This paper addresses the problem of entropy-regularized fine-tuning in continuous-time diffusion models, a method recently proposed by Uehara et al. (2024). The approach uses stochastic control to generate samples, incorporating an entropy regularizer to mitigate reward collapse. The paper provides a rigorous treatment of this theory and extends it to fine-tuning with general $f$-divergence regularizers. Key contributions include: 1. **Entropy-regularized Fine-tuning**: The paper develops a stochastic control framework to emulate the fine-tuned distribution, which is an exponential tilting of the pretrained model. This approach decouples the control and initial distribution, allowing for more flexible sampling. 2. **Stochastic Control Problem**: The problem is formulated as a stochastic control problem, where both the control and initial distribution are decision variables. The optimal control and initial distribution are derived, and their properties are analyzed. 3. **$f$-divergence Regularization**: The paper generalizes the entropy-regularized fine-tuning to $f$-divergence regularization, providing a unified framework for various divergence measures. This extension maintains the benefits of entropy regularization while allowing for more flexible divergence choices. 4. **Theoretical Results**: The paper includes theoretical results on the performance of the fine-tuned distribution, including bounds on the total variation distance to the target distribution. These bounds help understand the trade-offs between reward performance and diversity. 5. **Practical Implications**: The paper discusses the practical implications of the proposed methods, including the use of neural ODEs/SDEs for solving the control problem and the potential for real-world applications such as image synthesis and protein sequence generation. The paper concludes with directions for future research, including the extension of the theory to viscosity solutions and the exploration of direct $f$-divergence regularization.