Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition

Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition

October 13-16, 2024 | Majeed Kazemitabaar, Jack Williams, Ian Drosos, Tovi Grossman, Austin Z. Henley, Carina Negreanu, Advait Sarkar
This paper presents a study on improving steering and verification in AI-assisted data analysis through interactive task decomposition. The authors developed two systems, STEPWISE and PHASEWISE, to address the limitations of conversational AI tools in data analysis. These systems allow users to decompose tasks into editable components, enabling more control over the AI's output and verification of results. A controlled, within-subjects experiment with 18 participants compared these systems against a conversational baseline. Users reported significantly greater control with the STEPWISE and PHASEWISE systems, finding intervention, correction, and verification easier compared to the baseline. The results suggest design guidelines and trade-offs for AI-assisted data analysis tools. The study highlights the need for new affordances that decompose and display the AI's reasoning as structured and interactive assumptions, enabling users to modify them at any time. The authors also discuss related work on AI-assisted data analysis, verifying LLM outputs, and steering LLMs. The study contributes a formative study identifying limitations of conversational AI tools, a novel approach to improve steering and verification using editable assumptions, and a within-subjects experiment comparing the two systems with a conversational baseline. The results show that while there was no difference in task success or completion time, participants felt significantly more in control with the PHASEWISE and STEPWISE systems compared to the baseline. The paper also discusses the design of the systems, including task input, decomposition, and code execution, as well as the implementation of the systems as a web application and Python server stack. The study concludes that interactive task decomposition improves steering and verification in AI-assisted data analysis.This paper presents a study on improving steering and verification in AI-assisted data analysis through interactive task decomposition. The authors developed two systems, STEPWISE and PHASEWISE, to address the limitations of conversational AI tools in data analysis. These systems allow users to decompose tasks into editable components, enabling more control over the AI's output and verification of results. A controlled, within-subjects experiment with 18 participants compared these systems against a conversational baseline. Users reported significantly greater control with the STEPWISE and PHASEWISE systems, finding intervention, correction, and verification easier compared to the baseline. The results suggest design guidelines and trade-offs for AI-assisted data analysis tools. The study highlights the need for new affordances that decompose and display the AI's reasoning as structured and interactive assumptions, enabling users to modify them at any time. The authors also discuss related work on AI-assisted data analysis, verifying LLM outputs, and steering LLMs. The study contributes a formative study identifying limitations of conversational AI tools, a novel approach to improve steering and verification using editable assumptions, and a within-subjects experiment comparing the two systems with a conversational baseline. The results show that while there was no difference in task success or completion time, participants felt significantly more in control with the PHASEWISE and STEPWISE systems compared to the baseline. The paper also discusses the design of the systems, including task input, decomposition, and code execution, as well as the implementation of the systems as a web application and Python server stack. The study concludes that interactive task decomposition improves steering and verification in AI-assisted data analysis.
Reach us at info@study.space
[slides] Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition | StudySpace