The paper introduces SAM-E, a novel architecture for robot manipulation that leverages a vision-foundation model and sequence imitation for long-term action reasoning. The key contributions of SAM-E include: 1. **Vision-Foundation Model**: Utilizes the Segment Anything Model (SAM) as the foundation model for extracting task-relevant features from images and masks. SAM is pre-trained on a large dataset and is prompt-conditioned, making it suitable for language-instructed manipulation tasks. 2. **Parameter-Efficient Fine-Tuning**: Fine-tunes the SAM encoder with robot data to enhance understanding of embodied scenarios, using parameter-efficient methods like Low-Rank Adaptation (LoRA). 3. **Multi-View Transformer**: Integrates multi-view visual observations, depth information, and language instructions using attention mechanisms to achieve comprehensive fusion of input modalities. 4. **Action-Sequence Prediction**: Develops a novel multi-channel heatmap prediction head to predict coherent action sequences in a single pass, enhancing execution efficiency and long-horizon reasoning. 5. **Generalization and Efficiency**: Experimental results on various 3D instruction-following tasks from RLbench demonstrate superior performance, higher execution efficiency, and better generalization in few-shot adaptation to new tasks compared to baselines. - **Scene Understanding**: SAM-E leverages SAM to extract task-relevant visual features, enhancing scene understanding and generalization in various manipulation scenarios. - **Action Sequence Prediction**: The multi-channel heatmap prediction head enables efficient and coherent planning of action sequences, improving execution efficiency. - **Generalization**: SAM-E shows superior performance and generalization in few-shot adaptation to new tasks, highlighting its robustness and adaptability. - **Multi-Task Learning**: SAM-E outperforms state-of-the-art methods in most tasks on the RLbench benchmark, achieving higher success rates and better execution efficiency. - **Few-Shot Adaptation**: SAM-E demonstrates strong generalization capabilities by adapting to new tasks with significantly fewer demonstrations and update steps. - **Real-World Experiment**: Successful performance in real-world scenarios with a Franka Panda robot arm, validating the effectiveness of SAM-E in practical applications. - **SAM-E** is a novel architecture for robot manipulation that combines a vision-foundation model and sequence imitation for enhanced scene understanding, action prediction, and execution efficiency. It demonstrates superior performance and generalization in both simulated and real-world environments.The paper introduces SAM-E, a novel architecture for robot manipulation that leverages a vision-foundation model and sequence imitation for long-term action reasoning. The key contributions of SAM-E include: 1. **Vision-Foundation Model**: Utilizes the Segment Anything Model (SAM) as the foundation model for extracting task-relevant features from images and masks. SAM is pre-trained on a large dataset and is prompt-conditioned, making it suitable for language-instructed manipulation tasks. 2. **Parameter-Efficient Fine-Tuning**: Fine-tunes the SAM encoder with robot data to enhance understanding of embodied scenarios, using parameter-efficient methods like Low-Rank Adaptation (LoRA). 3. **Multi-View Transformer**: Integrates multi-view visual observations, depth information, and language instructions using attention mechanisms to achieve comprehensive fusion of input modalities. 4. **Action-Sequence Prediction**: Develops a novel multi-channel heatmap prediction head to predict coherent action sequences in a single pass, enhancing execution efficiency and long-horizon reasoning. 5. **Generalization and Efficiency**: Experimental results on various 3D instruction-following tasks from RLbench demonstrate superior performance, higher execution efficiency, and better generalization in few-shot adaptation to new tasks compared to baselines. - **Scene Understanding**: SAM-E leverages SAM to extract task-relevant visual features, enhancing scene understanding and generalization in various manipulation scenarios. - **Action Sequence Prediction**: The multi-channel heatmap prediction head enables efficient and coherent planning of action sequences, improving execution efficiency. - **Generalization**: SAM-E shows superior performance and generalization in few-shot adaptation to new tasks, highlighting its robustness and adaptability. - **Multi-Task Learning**: SAM-E outperforms state-of-the-art methods in most tasks on the RLbench benchmark, achieving higher success rates and better execution efficiency. - **Few-Shot Adaptation**: SAM-E demonstrates strong generalization capabilities by adapting to new tasks with significantly fewer demonstrations and update steps. - **Real-World Experiment**: Successful performance in real-world scenarios with a Franka Panda robot arm, validating the effectiveness of SAM-E in practical applications. - **SAM-E** is a novel architecture for robot manipulation that combines a vision-foundation model and sequence imitation for enhanced scene understanding, action prediction, and execution efficiency. It demonstrates superior performance and generalization in both simulated and real-world environments.