May 11–16, 2024, Honolulu, HI, USA | Runze Cai, Nuwan Janaka, Yang Chen, Lucia Wang, Shengdong Zhao, Can Liu
PANDALens is an AI-assisted in-context writing system designed for Optical See-Through Head-Mounted Displays (OHMDs) to support personalized documentation during travel. It leverages multimodal context information from user behaviors and environments to identify interests and elicit contemplation, then uses Large Language Models (LLMs) to transform this information into coherent narratives with reduced user effort. A real-world travel scenario comparing PANDALens with a smartphone alternative confirmed its effectiveness in improving writing quality and travel enjoyment while minimizing user effort. The system proactively captures interesting moments, asks context-related questions, and generates travel blogs post-travel. It supports seamless interaction during primary tasks and minimizes post-editing efforts. PANDALens introduces a design space for multimodal context information naturally occurring in travel scenarios, demonstrating how this information can enhance interaction and writing. It presents an AI-assisted writing approach that transforms passive mobile tools into proactive wearable assistants. It also provides an empirical study validating this approach in realistic scenarios, offering further design implications. The system uses mixed-initiative interactions to reduce interference and utilizes LLMs for document co-creation. It employs a multimodal context analyzer to detect user interests and initiate AI interactions when users' attention isn't occupied. PANDALens offers non-intrusive suggestions and facilitates more natural and personalized co-created documentation. It also asks context-related in-situ questions to enrich the documentation, a departure from traditional systems offering fixed questions. The system was compared with LiveSnippets in real-world travel settings, showing its effectiveness in generating high-quality narratives with high user experience during travel. The study found that PANDALens could effectively improve travel enjoyment, evoke more profound reflections, and produce high-quality narrative documentation with reduced effort. The system's design space includes interaction and content generation, with interaction involving detecting situational and personal interests and content generation involving using context information to reduce writing workload. PANDALens was tested with eight users, and iterative design refined the system to support user preferences and minimize distraction. The system uses LLMs to generate context-related questions and final narratives, with prompts designed to ensure comprehensive understanding of user travel experiences. The system also addresses challenges in LLM data processing, such as irrelevant questions and unsatisfactory final narratives, by refining prompts and using a summary approach for each distinct moment. PANDALens demonstrates the potential of transforming AI-assisted in-context writing from tools to intelligent companions, enhancing the quality and personalization of travel documentation.PANDALens is an AI-assisted in-context writing system designed for Optical See-Through Head-Mounted Displays (OHMDs) to support personalized documentation during travel. It leverages multimodal context information from user behaviors and environments to identify interests and elicit contemplation, then uses Large Language Models (LLMs) to transform this information into coherent narratives with reduced user effort. A real-world travel scenario comparing PANDALens with a smartphone alternative confirmed its effectiveness in improving writing quality and travel enjoyment while minimizing user effort. The system proactively captures interesting moments, asks context-related questions, and generates travel blogs post-travel. It supports seamless interaction during primary tasks and minimizes post-editing efforts. PANDALens introduces a design space for multimodal context information naturally occurring in travel scenarios, demonstrating how this information can enhance interaction and writing. It presents an AI-assisted writing approach that transforms passive mobile tools into proactive wearable assistants. It also provides an empirical study validating this approach in realistic scenarios, offering further design implications. The system uses mixed-initiative interactions to reduce interference and utilizes LLMs for document co-creation. It employs a multimodal context analyzer to detect user interests and initiate AI interactions when users' attention isn't occupied. PANDALens offers non-intrusive suggestions and facilitates more natural and personalized co-created documentation. It also asks context-related in-situ questions to enrich the documentation, a departure from traditional systems offering fixed questions. The system was compared with LiveSnippets in real-world travel settings, showing its effectiveness in generating high-quality narratives with high user experience during travel. The study found that PANDALens could effectively improve travel enjoyment, evoke more profound reflections, and produce high-quality narrative documentation with reduced effort. The system's design space includes interaction and content generation, with interaction involving detecting situational and personal interests and content generation involving using context information to reduce writing workload. PANDALens was tested with eight users, and iterative design refined the system to support user preferences and minimize distraction. The system uses LLMs to generate context-related questions and final narratives, with prompts designed to ensure comprehensive understanding of user travel experiences. The system also addresses challenges in LLM data processing, such as irrelevant questions and unsatisfactory final narratives, by refining prompts and using a summary approach for each distinct moment. PANDALens demonstrates the potential of transforming AI-assisted in-context writing from tools to intelligent companions, enhancing the quality and personalization of travel documentation.