Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models

Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models

May 11-16, 2024, Honolulu, HI, USA | Paramveer S. Dhillon, Somayeh Molaiei, Jiaqi Li, Maximilian Golub, Shaocun Zheng, Lionel P. Robert
This paper explores how varying levels of scaffolding from large language models (LLMs) influence the co-writing process. Using a within-subjects field experiment with a Latin square design, researchers tested three conditions: no AI assistance (control), next-sentence suggestions (low scaffolding), and next-paragraph suggestions (high scaffolding). Participants (N=131) responded to argumentative writing prompts under these conditions. The findings reveal a U-shaped relationship between scaffolding and writing quality and productivity (words/time). Low scaffolding did not significantly improve writing quality or productivity, while high scaffolding led to significant improvements, especially benefiting non-regular writers and less tech-savvy users. No significant cognitive burden was observed, but a moderate decrease in text ownership and satisfaction was noted. The study highlights the need for personalized scaffolding mechanisms in AI-powered writing tools. The results suggest that high-level scaffolding provides a more structured framework, reducing cognitive load and enhancing writing quality. However, it also reduces perceived text ownership and satisfaction. The study emphasizes the importance of tailoring AI assistance to the writer's expertise and needs, ensuring a balance between AI support and human agency. The findings have implications for the design of AI-assisted writing tools, emphasizing the need for adaptive, personalized scaffolding to enhance user satisfaction and writing outcomes.This paper explores how varying levels of scaffolding from large language models (LLMs) influence the co-writing process. Using a within-subjects field experiment with a Latin square design, researchers tested three conditions: no AI assistance (control), next-sentence suggestions (low scaffolding), and next-paragraph suggestions (high scaffolding). Participants (N=131) responded to argumentative writing prompts under these conditions. The findings reveal a U-shaped relationship between scaffolding and writing quality and productivity (words/time). Low scaffolding did not significantly improve writing quality or productivity, while high scaffolding led to significant improvements, especially benefiting non-regular writers and less tech-savvy users. No significant cognitive burden was observed, but a moderate decrease in text ownership and satisfaction was noted. The study highlights the need for personalized scaffolding mechanisms in AI-powered writing tools. The results suggest that high-level scaffolding provides a more structured framework, reducing cognitive load and enhancing writing quality. However, it also reduces perceived text ownership and satisfaction. The study emphasizes the importance of tailoring AI assistance to the writer's expertise and needs, ensuring a balance between AI support and human agency. The findings have implications for the design of AI-assisted writing tools, emphasizing the need for adaptive, personalized scaffolding to enhance user satisfaction and writing outcomes.
Reach us at info@study.space