A Survey on Personalized Content Synthesis with Diffusion Models

A Survey on Personalized Content Synthesis with Diffusion Models

9 May 2024 | Xulu Zhang, Xiao-Yong Wei, Wengyu Zhang, Jinlin Wu, Zhaoxiang Zhang, Zhen Lei, Qing Li
This paper provides a comprehensive survey of Personalized Content Synthesis (PCS) with diffusion models. PCS aims to generate images aligned with user-defined prompts using a small set of user-provided examples. Over the past two years, more than 150 methods have been proposed, but existing surveys mainly focus on text-to-image generation. This paper focuses on diffusion models and introduces two generic frameworks: optimization-based and learning-based. Optimization-based methods fine-tune a distinct generative model for each personalization request, while learning-based methods aim to train a unified model that can handle any SoI generation. The paper also discusses specialized tasks within PCS, such as personalized object generation, face synthesis, and style personalization, highlighting their unique challenges and innovations. Challenges such as overfitting and the trade-off between subject fidelity and text alignment are analyzed. The paper also explores the use of diffusion models in various modalities, including video, 3D representations, and speech. It discusses the importance of robust evaluation metrics, standardized test datasets, and faster processing times. The paper concludes with future directions for research in PCS.This paper provides a comprehensive survey of Personalized Content Synthesis (PCS) with diffusion models. PCS aims to generate images aligned with user-defined prompts using a small set of user-provided examples. Over the past two years, more than 150 methods have been proposed, but existing surveys mainly focus on text-to-image generation. This paper focuses on diffusion models and introduces two generic frameworks: optimization-based and learning-based. Optimization-based methods fine-tune a distinct generative model for each personalization request, while learning-based methods aim to train a unified model that can handle any SoI generation. The paper also discusses specialized tasks within PCS, such as personalized object generation, face synthesis, and style personalization, highlighting their unique challenges and innovations. Challenges such as overfitting and the trade-off between subject fidelity and text alignment are analyzed. The paper also explores the use of diffusion models in various modalities, including video, 3D representations, and speech. It discusses the importance of robust evaluation metrics, standardized test datasets, and faster processing times. The paper concludes with future directions for research in PCS.
Reach us at info@study.space
Understanding A Survey on Personalized Content Synthesis with Diffusion Models