RAMBLER: Supporting Writing With Speech via LLM-Assisted Gist Manipulation

RAMBLER: Supporting Writing With Speech via LLM-Assisted Gist Manipulation

May 11–16, 2024, Honolulu, HI, USA | Susan Lin, Jeremy Warner, J.D. Zamfirescu-Pereira, Matthew G. Lee, Sauhard Jain, Michael Xuelin Huang, Piyawat Lertvittayakumjorn, Shangqing Cai, Shumin Zhai, Björn Hartmann, Can Liu
RAMBLER is an LLM-powered graphical user interface that supports writing with speech by enabling gist-level manipulation of dictated text. It offers two main functions: gist extraction and macro revision. Gist extraction generates keywords and summaries to support review and interaction with spoken text. LLM-assisted macro revisions allow users to respeak, split, merge, and transform dictated text without specifying precise editing locations. RAMBLER helps bridge the gap between spontaneous spoken words and well-structured writing by enabling interactive dictation and revision. In a comparative study with 12 participants, RAMBLER outperformed a speech-to-text editor + ChatGPT, as it better facilitates iterative revisions with enhanced user control over content while supporting diverse user strategies. RAMBLER is designed to address the challenges of speech production and memory retention, providing a gist-based interface tailored to dictated text. It allows users to dictate their thoughts and place each segment into a Ramble. RAMBLER provides Semantic Zoom to visualize multiple summarization levels in the form of gists for each Ramble, which users can review on demand. Users then focus on iterating on higher-level ideas by manipulating the Rambles through respeaking, splitting, merging, and reorganizing them, activities collectively called macro revision. Behind the scenes, RAMBLER uses an LLM to automatically clean up disfluencies and punctuation errors from the raw transcript, as well as to complete broken sentences and smooth transitions, activities called micro revision. RAMBLER also extracts keywords from text to aid visual skimming across Rambles and Semantic Zoom levels. Individual keywords can be activated and deactivated by the user to indicate importance; these keywords then serve as parameters to customize summary (gist) generation and other LLM-based word processing. RAMBLER aims to aid the review and revision of spoken text and ease interactions with text on mobile devices. To evaluate RAMBLER's effectiveness, a comparative study with 12 participants using a long-form verbal text composition task was conducted. The baseline is a standard speech-to-text editor interface with ChatGPT provided on the side. While participants demonstrated varied strategies and feature usage, most expressed a preference for RAMBLER over the baseline and acceptance of using it for their own tasks. Their feedback shows RAMBLER provides better support for developing and reviewing spoken content, supporting iteration, and improving user control over content revision. Our work makes the following contributions: First, RAMBLER: a novel tool that implements the concept of a gist-based interface to support writing with speech on mobile devices. We motivate our design goals and provide implementation details for RAMBLER. Second, we contribute empirical findings from a user study revealing the potential of RAMBLER in helping close the gap between rambling and writing. Our study also reveals several advantages of our specialized LLM-backed GUI over a one-size-fits-all dialogue-based chatbot UI for LLMs; we thus also offer insights, based on our findings,RAMBLER is an LLM-powered graphical user interface that supports writing with speech by enabling gist-level manipulation of dictated text. It offers two main functions: gist extraction and macro revision. Gist extraction generates keywords and summaries to support review and interaction with spoken text. LLM-assisted macro revisions allow users to respeak, split, merge, and transform dictated text without specifying precise editing locations. RAMBLER helps bridge the gap between spontaneous spoken words and well-structured writing by enabling interactive dictation and revision. In a comparative study with 12 participants, RAMBLER outperformed a speech-to-text editor + ChatGPT, as it better facilitates iterative revisions with enhanced user control over content while supporting diverse user strategies. RAMBLER is designed to address the challenges of speech production and memory retention, providing a gist-based interface tailored to dictated text. It allows users to dictate their thoughts and place each segment into a Ramble. RAMBLER provides Semantic Zoom to visualize multiple summarization levels in the form of gists for each Ramble, which users can review on demand. Users then focus on iterating on higher-level ideas by manipulating the Rambles through respeaking, splitting, merging, and reorganizing them, activities collectively called macro revision. Behind the scenes, RAMBLER uses an LLM to automatically clean up disfluencies and punctuation errors from the raw transcript, as well as to complete broken sentences and smooth transitions, activities called micro revision. RAMBLER also extracts keywords from text to aid visual skimming across Rambles and Semantic Zoom levels. Individual keywords can be activated and deactivated by the user to indicate importance; these keywords then serve as parameters to customize summary (gist) generation and other LLM-based word processing. RAMBLER aims to aid the review and revision of spoken text and ease interactions with text on mobile devices. To evaluate RAMBLER's effectiveness, a comparative study with 12 participants using a long-form verbal text composition task was conducted. The baseline is a standard speech-to-text editor interface with ChatGPT provided on the side. While participants demonstrated varied strategies and feature usage, most expressed a preference for RAMBLER over the baseline and acceptance of using it for their own tasks. Their feedback shows RAMBLER provides better support for developing and reviewing spoken content, supporting iteration, and improving user control over content revision. Our work makes the following contributions: First, RAMBLER: a novel tool that implements the concept of a gist-based interface to support writing with speech on mobile devices. We motivate our design goals and provide implementation details for RAMBLER. Second, we contribute empirical findings from a user study revealing the potential of RAMBLER in helping close the gap between rambling and writing. Our study also reveals several advantages of our specialized LLM-backed GUI over a one-size-fits-all dialogue-based chatbot UI for LLMs; we thus also offer insights, based on our findings,
Reach us at info@study.space
[slides and audio] Rambler%3A Supporting Writing With Speech via LLM-Assisted Gist Manipulation