**RoboDreamer: Learning Compositional World Models for Robot Imagination** **Abstract:** Text-to-video models have shown significant potential in robotic decision-making, enabling the generation of realistic plans and accurate environment simulations. However, these models often struggle with generalization, as they are limited to synthesizing videos based on language instructions similar to those seen during training. This limitation is particularly problematic in robotics, where the goal is to synthesize plans for unseen combinations of objects and actions in new environments. To address this issue, RoboDreamer introduces an innovative approach to learn a compositional world model by factorizing video generation. By leveraging the natural compositionality of language, RoboDreamer parses instructions into lower-level primitives, which are then conditioned on a set of models to generate videos. This factorization enables compositional generalization, allowing the model to handle new combinations of language and multimodal input. The approach also supports the specification of videos using both natural language instructions and goal images, enhancing its flexibility and applicability in robotics tasks. **Contributions:** 1. **RoboDreamer:** A compositional world model that factorizes video generation, enabling compositional generalization. 2. **Multimodal Composition:** The ability to combine multimodal information, such as goal images and sketches, with natural language instructions. 3. **Empirical Results:** Strong alignment with tasks under multimodal instructions and promising performance in robot manipulation tasks. **Background:** The paper discusses the background of text-conditioned video generation and its application in robotics, including planning and execution of video plans. **Experiments:** - **Video Generation:** RoboDreamer demonstrates zero-shot generalization and improved spatial reasoning with multi-modal inputs. - **Robot Planning:** RoboDreamer outperforms baselines in robotic planning tasks, achieving higher success rates. **Related Work:** The paper reviews related work in diffusion models for decision-making and compositional generation, highlighting the novelty of RoboDreamer's approach. **Conclusion:** RoboDreamer significantly advances the field of machine learning by enabling more accurate and generalizable video generation for robotics. However, limitations such as single-camera view support and limited real-world image generalization are noted for future research.**RoboDreamer: Learning Compositional World Models for Robot Imagination** **Abstract:** Text-to-video models have shown significant potential in robotic decision-making, enabling the generation of realistic plans and accurate environment simulations. However, these models often struggle with generalization, as they are limited to synthesizing videos based on language instructions similar to those seen during training. This limitation is particularly problematic in robotics, where the goal is to synthesize plans for unseen combinations of objects and actions in new environments. To address this issue, RoboDreamer introduces an innovative approach to learn a compositional world model by factorizing video generation. By leveraging the natural compositionality of language, RoboDreamer parses instructions into lower-level primitives, which are then conditioned on a set of models to generate videos. This factorization enables compositional generalization, allowing the model to handle new combinations of language and multimodal input. The approach also supports the specification of videos using both natural language instructions and goal images, enhancing its flexibility and applicability in robotics tasks. **Contributions:** 1. **RoboDreamer:** A compositional world model that factorizes video generation, enabling compositional generalization. 2. **Multimodal Composition:** The ability to combine multimodal information, such as goal images and sketches, with natural language instructions. 3. **Empirical Results:** Strong alignment with tasks under multimodal instructions and promising performance in robot manipulation tasks. **Background:** The paper discusses the background of text-conditioned video generation and its application in robotics, including planning and execution of video plans. **Experiments:** - **Video Generation:** RoboDreamer demonstrates zero-shot generalization and improved spatial reasoning with multi-modal inputs. - **Robot Planning:** RoboDreamer outperforms baselines in robotic planning tasks, achieving higher success rates. **Related Work:** The paper reviews related work in diffusion models for decision-making and compositional generation, highlighting the novelty of RoboDreamer's approach. **Conclusion:** RoboDreamer significantly advances the field of machine learning by enabling more accurate and generalizable video generation for robotics. However, limitations such as single-camera view support and limited real-world image generalization are noted for future research.