PALP: Prompt Aligned Personalization of Text-to-Image Models

PALP: Prompt Aligned Personalization of Text-to-Image Models

11 Jan 2024 | Moab Arar, Andrey Voynov, Amir Hertz, Omri Avrahami, Shlomi Fruchter, Yael Pritch, Daniel Cohen-Or, Ariel Shamir
The paper "PALP: Prompt Aligned Personalization of Text-to-Image Models" addresses the challenge of generating personalized images that align well with complex textual prompts. Traditional text-to-image models often struggle with both personalization and prompt alignment, leading to suboptimal results. The authors propose a novel approach called Prompt-Aligned Personalization (PALP) to improve this balance. PALP focuses on personalizing a single prompt while maintaining alignment with the target prompt. This approach leverages pre-trained diffusion models and score distillation sampling to ensure that the personalized model remains aligned with the target prompt. The method involves two main components: personalization and prompt alignment. Personalization teaches the model to recognize and generate images of a specific subject, while prompt alignment ensures that the model's predictions are aligned with the target prompt. The authors demonstrate the effectiveness of PALP through various experiments, including multi- and single-shot settings. They show that PALP can generate images with rich and complex scenes, including multiple subjects and reference images from artworks. The method outperforms existing baselines in terms of both prompt alignment and subject fidelity, achieving superior results without requiring large-scale pre-training data. The paper also includes a user study to evaluate the performance of PALP, confirming its ability to produce images that closely match the desired prompts. Additionally, the authors discuss the computational complexity of PALP and provide additional results for different experimental setups, further validating the effectiveness of their approach.The paper "PALP: Prompt Aligned Personalization of Text-to-Image Models" addresses the challenge of generating personalized images that align well with complex textual prompts. Traditional text-to-image models often struggle with both personalization and prompt alignment, leading to suboptimal results. The authors propose a novel approach called Prompt-Aligned Personalization (PALP) to improve this balance. PALP focuses on personalizing a single prompt while maintaining alignment with the target prompt. This approach leverages pre-trained diffusion models and score distillation sampling to ensure that the personalized model remains aligned with the target prompt. The method involves two main components: personalization and prompt alignment. Personalization teaches the model to recognize and generate images of a specific subject, while prompt alignment ensures that the model's predictions are aligned with the target prompt. The authors demonstrate the effectiveness of PALP through various experiments, including multi- and single-shot settings. They show that PALP can generate images with rich and complex scenes, including multiple subjects and reference images from artworks. The method outperforms existing baselines in terms of both prompt alignment and subject fidelity, achieving superior results without requiring large-scale pre-training data. The paper also includes a user study to evaluate the performance of PALP, confirming its ability to produce images that closely match the desired prompts. Additionally, the authors discuss the computational complexity of PALP and provide additional results for different experimental setups, further validating the effectiveness of their approach.
Reach us at info@study.space
[slides and audio] PALP%3A Prompt Aligned Personalization of Text-to-Image Models