**PANDALens: Towards AI-Assisted In-Context Writing on OHMD During Travels** **Authors:** Runze Cai, Nuwan Janaka, Yang Chen, Lucia Wang, Shengdong Zhao, and Can Liu **Abstract:** Traditional in-context writing tools are passive and lack intelligence, reducing the enjoyment of primary tasks like travel and hindering high-quality writing. PANDALens, a Proactive AI Narrative Documentation Assistant, addresses these issues by leveraging an Optical See-Through Head Mounted Display (OHMD) to support personalized documentation in everyday activities. PANDALens observes multimodal contextual information from user behaviors and the environment to confirm interests and elicit contemplation, using Large Language Models (LLMs) to transform this information into coherent narratives with minimal user effort. A real-world travel scenario comparing PANDALens with a smartphone alternative confirmed its effectiveness in improving writing quality and travel enjoyment while minimizing user effort. The study proposes design guidelines for AI-assisted in-context writing, highlighting the potential of transforming tools into intelligent companions. **Keywords:** HMD, smart glasses, AI, large language model, multimodal information, Human-AI collaborative writing, in-context writing, travel blog **Design Space:** - **Interaction:** Utilizes various contextual cues to guide AI interactions, including multimodal context analysis for interest detection, question generation, and optimal interaction methods. - **Content Generation:** leverages contextual data to reduce writing workload, enhancing narrative depth and personalization. **System Overview:** - **Usage Scenarios:** PANDALens captures interesting moments, generates context-related questions, and creates final narratives. - **Key Features:** - **Mixed Initiation for Moment Capture:** Combines AI and human initiation for seamless interaction. - **Multimodal Analyzer for User Interest Detection:** Detects situational and personal interests using gaze patterns, movement changes, speaking tone, gestures, and facial cues. - **Interaction Design:** Minimizes distraction through subtle visual and audio notifications, peripheral vision placement, and limited notification frequency. **Iterative Design:** - **Human Initiation:** Subtle ring interaction for menu manipulation, allowing users to remain engaged in primary activities. - **Processing of Interested Moments:** Transcribes user comments, enriches with multimodal context, and generates context-specific questions and final narratives using LLMs. - **Context-Related Questions for Inspiration:** Poses relevant questions tailored to user preferences, balancing inspiration and potential distraction. **Conclusion:** PANDALens effectively supports high-quality, personalized documentation in travel scenarios, enhancing travel enjoyment and writing quality while minimizing user effort. The system's design guidelines and empirical validation offer valuable insights for future AI-assisted in-context writing tools.**PANDALens: Towards AI-Assisted In-Context Writing on OHMD During Travels** **Authors:** Runze Cai, Nuwan Janaka, Yang Chen, Lucia Wang, Shengdong Zhao, and Can Liu **Abstract:** Traditional in-context writing tools are passive and lack intelligence, reducing the enjoyment of primary tasks like travel and hindering high-quality writing. PANDALens, a Proactive AI Narrative Documentation Assistant, addresses these issues by leveraging an Optical See-Through Head Mounted Display (OHMD) to support personalized documentation in everyday activities. PANDALens observes multimodal contextual information from user behaviors and the environment to confirm interests and elicit contemplation, using Large Language Models (LLMs) to transform this information into coherent narratives with minimal user effort. A real-world travel scenario comparing PANDALens with a smartphone alternative confirmed its effectiveness in improving writing quality and travel enjoyment while minimizing user effort. The study proposes design guidelines for AI-assisted in-context writing, highlighting the potential of transforming tools into intelligent companions. **Keywords:** HMD, smart glasses, AI, large language model, multimodal information, Human-AI collaborative writing, in-context writing, travel blog **Design Space:** - **Interaction:** Utilizes various contextual cues to guide AI interactions, including multimodal context analysis for interest detection, question generation, and optimal interaction methods. - **Content Generation:** leverages contextual data to reduce writing workload, enhancing narrative depth and personalization. **System Overview:** - **Usage Scenarios:** PANDALens captures interesting moments, generates context-related questions, and creates final narratives. - **Key Features:** - **Mixed Initiation for Moment Capture:** Combines AI and human initiation for seamless interaction. - **Multimodal Analyzer for User Interest Detection:** Detects situational and personal interests using gaze patterns, movement changes, speaking tone, gestures, and facial cues. - **Interaction Design:** Minimizes distraction through subtle visual and audio notifications, peripheral vision placement, and limited notification frequency. **Iterative Design:** - **Human Initiation:** Subtle ring interaction for menu manipulation, allowing users to remain engaged in primary activities. - **Processing of Interested Moments:** Transcribes user comments, enriches with multimodal context, and generates context-specific questions and final narratives using LLMs. - **Context-Related Questions for Inspiration:** Poses relevant questions tailored to user preferences, balancing inspiration and potential distraction. **Conclusion:** PANDALens effectively supports high-quality, personalized documentation in travel scenarios, enhancing travel enjoyment and writing quality while minimizing user effort. The system's design guidelines and empirical validation offer valuable insights for future AI-assisted in-context writing tools.