Jul 2024 | Xiaohan Peng, Janin Koch, Wendy E Mackay
DesignPrompt is a digital moodboard tool that enables designers to create multimodal prompts for generative AI (GenAI) to explore and express their design intentions. The tool allows designers to combine images, colors, and semantic metadata into a single prompt, enabling them to fine-tune their intentions and generate more effective results. A comparative study with 12 professional designers found that multimodal input encouraged more effective exploration and expression. Designers developed innovative uses of DesignPrompt, including creating elaborate multimodal prompts and establishing a pattern for maximizing novelty while ensuring consistency. The study identified four key design implications: (1) supporting different levels of abstraction and semantics in prompts, (2) helping users translate abstract intentions into richer prompts, (3) enabling users to understand and control the impact of prompts on outputs, and (4) allowing users to control and manipulate images in an engaging way. DesignPrompt was evaluated to address research questions about the effectiveness of multimodal input in prompting, the alignment of results with user expectations, and the perception of system transparency and usefulness. The study found that multimodal input improved system expressivity, user understanding of prompts and images, and output understanding, but was less effective at producing images that aligned with expectations. Overall, the study highlights the potential of multimodal interaction in design exploration with GenAI.DesignPrompt is a digital moodboard tool that enables designers to create multimodal prompts for generative AI (GenAI) to explore and express their design intentions. The tool allows designers to combine images, colors, and semantic metadata into a single prompt, enabling them to fine-tune their intentions and generate more effective results. A comparative study with 12 professional designers found that multimodal input encouraged more effective exploration and expression. Designers developed innovative uses of DesignPrompt, including creating elaborate multimodal prompts and establishing a pattern for maximizing novelty while ensuring consistency. The study identified four key design implications: (1) supporting different levels of abstraction and semantics in prompts, (2) helping users translate abstract intentions into richer prompts, (3) enabling users to understand and control the impact of prompts on outputs, and (4) allowing users to control and manipulate images in an engaging way. DesignPrompt was evaluated to address research questions about the effectiveness of multimodal input in prompting, the alignment of results with user expectations, and the perception of system transparency and usefulness. The study found that multimodal input improved system expressivity, user understanding of prompts and images, and output understanding, but was less effective at producing images that aligned with expectations. Overall, the study highlights the potential of multimodal interaction in design exploration with GenAI.