Augmented Object Intelligence with XR-Objects

Augmented Object Intelligence with XR-Objects

October 13–16, 2024, Pittsburgh, PA, USA | Mustafa Doga Dogan, Eric J. Gonzalez, Karan Ahuja, Ruofei Du, Andrea Colaço, Johnny Lee, Mar Gonzalez-Franco, David Kim
The paper explores the concept of Augmented Object Intelligence (AOI) in the context of Extended Reality (XR), aiming to integrate physical objects with digital functionalities seamlessly. The authors introduce XR-Objects, an open-source prototype system that enables users to interact with real-world objects as if they were digital entities. This is achieved through real-time object segmentation and classification, combined with Multimodal Large Language Models (MLLMs). XR-Objects allows objects to provide context-specific information and perform digital actions, such as setting timers or filtering products. The system is designed to be object-centric, using a world-space UI, and includes features like semi-transparent bubbles to indicate interactive objects. A user study comparing XR-Objects to a state-of-the-art MLLM assistant interface (Gemini app) showed that XR-Objects significantly reduced task completion time and improved user satisfaction. The paper also discusses potential applications in cooking, shopping, discovery, productivity, learning, and IoT connectivity, highlighting the system's versatility and potential for enhancing everyday interactions.The paper explores the concept of Augmented Object Intelligence (AOI) in the context of Extended Reality (XR), aiming to integrate physical objects with digital functionalities seamlessly. The authors introduce XR-Objects, an open-source prototype system that enables users to interact with real-world objects as if they were digital entities. This is achieved through real-time object segmentation and classification, combined with Multimodal Large Language Models (MLLMs). XR-Objects allows objects to provide context-specific information and perform digital actions, such as setting timers or filtering products. The system is designed to be object-centric, using a world-space UI, and includes features like semi-transparent bubbles to indicate interactive objects. A user study comparing XR-Objects to a state-of-the-art MLLM assistant interface (Gemini app) showed that XR-Objects significantly reduced task completion time and improved user satisfaction. The paper also discusses potential applications in cooking, shopping, discovery, productivity, learning, and IoT connectivity, highlighting the system's versatility and potential for enhancing everyday interactions.
Reach us at info@study.space