The paper "LLMSense: Harnessing LLMs for High-level Reasoning Over Spatiotemporal Sensor Traces" by Xiaomin Ouyang and Mani Srivastava addresses the challenge of high-level reasoning tasks in sensing systems, which require understanding and interpreting long-term sensor data. Traditional machine learning approaches struggle with generalization due to limited training samples and high-dimensional sensor traces. The authors propose LLMSense, a system that leverages the reasoning capabilities and world knowledge of Large Language Models (LLMs) to handle these tasks.
LLMSense includes an effective prompting framework for high-level reasoning tasks, which can process both raw sensor data and low-level perception results. Two strategies are designed to enhance performance with long sensor traces: summarization before reasoning and selective inclusion of historical traces. The framework can be implemented in an edge-cloud setup, where small LLMs run on the edge for data summarization and high-level reasoning on the cloud to preserve privacy.
The paper evaluates LLMSense on two high-level reasoning tasks: dementia diagnosis with behavior traces and occupancy tracking with environmental sensor traces. Results show that LLMSense achieves over 80% accuracy, demonstrating the effectiveness of the proposed approach. The authors also provide insights and guidelines for leveraging LLMs for high-level reasoning on sensor traces and suggest future directions, such as processing longer or infinite traces, improving LLM performance through verifications, and joint optimization of low-level perception and high-level reasoning tasks.The paper "LLMSense: Harnessing LLMs for High-level Reasoning Over Spatiotemporal Sensor Traces" by Xiaomin Ouyang and Mani Srivastava addresses the challenge of high-level reasoning tasks in sensing systems, which require understanding and interpreting long-term sensor data. Traditional machine learning approaches struggle with generalization due to limited training samples and high-dimensional sensor traces. The authors propose LLMSense, a system that leverages the reasoning capabilities and world knowledge of Large Language Models (LLMs) to handle these tasks.
LLMSense includes an effective prompting framework for high-level reasoning tasks, which can process both raw sensor data and low-level perception results. Two strategies are designed to enhance performance with long sensor traces: summarization before reasoning and selective inclusion of historical traces. The framework can be implemented in an edge-cloud setup, where small LLMs run on the edge for data summarization and high-level reasoning on the cloud to preserve privacy.
The paper evaluates LLMSense on two high-level reasoning tasks: dementia diagnosis with behavior traces and occupancy tracking with environmental sensor traces. Results show that LLMSense achieves over 80% accuracy, demonstrating the effectiveness of the proposed approach. The authors also provide insights and guidelines for leveraging LLMs for high-level reasoning on sensor traces and suggest future directions, such as processing longer or infinite traces, improving LLM performance through verifications, and joint optimization of low-level perception and high-level reasoning tasks.