FIGURAl1y: AI Assistance for Writing Scientific Alt Text

FIGURAl1y: AI Assistance for Writing Scientific Alt Text

March 18-21, 2024 | Nikhil Singh, Lucy Lu Wang, Jonathan Bragg
FIGURA11Y is an AI-assisted system for generating high-quality alt text for scientific figures. The system helps authors create descriptive alt text by providing draft descriptions and suggestions for revision. It uses a pipeline driven by extracted figure and paper metadata to generate initial drafts and offer interactive suggestions. Two versions of the system were tested: Draft+Revise, which provides automatically generated drafts and metadata for revision, and Interactive Assistance, which adds contextualized suggestions such as text snippets for iterative description and hypothetical user questions to reveal ambiguities. A study with 14 authors found that the system helped them efficiently produce descriptive alt text, with interactive features enabling more iteration and highlighting aspects for authors to consider without increasing cognitive load or manual effort. The system was developed to address the challenges of creating accurate and complete alt text for complex scientific figures, which is often difficult for authors due to the intricate visual information and varied informational needs of readers. Guidelines for alt text writing are often narrow in scope, focusing on specific figure types, and fully automated approaches are limited by the diversity of figure types and domain-specific details. FIGURA11Y combines human and AI capabilities to provide tailored guidance within interactive alt text drafting workflows. It extracts figures, captions, and metadata from papers, and uses this information to generate prompts for large language models. The system also includes features such as Generate at Cursor, which allows authors to expand descriptions at user-directed points, and Potential User Questions, which prompt authors to address ambiguous elements. The system was evaluated through a within-subjects study where authors described their own figures across a diverse set of figures and fields of study. The study found that the system assisted in rapid drafting and editing of descriptive alt text through different strategies based on author needs. Interactive features enhanced the experience without increasing cognitive load or effort on average, and enabled greater deviation from generated drafts by supporting iterative refinement. The system's design incorporates structured prompts, metadata extraction, and a user interface that provides AI-assisted support throughout the alt text drafting process. The system was implemented using TypeScript and Python, with a frontend built using ReactJS, Next.js, Zustand, and Mantine, and a backend using Flask and PostgreSQL. The study aimed to evaluate the usefulness of the system for assisting authors in producing alt text, examining whether authors perceived benefit from the system's scaffolding and pre-generated drafts, whether interactive features supported further enhancement of descriptions, and whether added features incurred additional cognitive load. The results showed that the system was effective in helping authors produce high-quality alt text for scientific figures.FIGURA11Y is an AI-assisted system for generating high-quality alt text for scientific figures. The system helps authors create descriptive alt text by providing draft descriptions and suggestions for revision. It uses a pipeline driven by extracted figure and paper metadata to generate initial drafts and offer interactive suggestions. Two versions of the system were tested: Draft+Revise, which provides automatically generated drafts and metadata for revision, and Interactive Assistance, which adds contextualized suggestions such as text snippets for iterative description and hypothetical user questions to reveal ambiguities. A study with 14 authors found that the system helped them efficiently produce descriptive alt text, with interactive features enabling more iteration and highlighting aspects for authors to consider without increasing cognitive load or manual effort. The system was developed to address the challenges of creating accurate and complete alt text for complex scientific figures, which is often difficult for authors due to the intricate visual information and varied informational needs of readers. Guidelines for alt text writing are often narrow in scope, focusing on specific figure types, and fully automated approaches are limited by the diversity of figure types and domain-specific details. FIGURA11Y combines human and AI capabilities to provide tailored guidance within interactive alt text drafting workflows. It extracts figures, captions, and metadata from papers, and uses this information to generate prompts for large language models. The system also includes features such as Generate at Cursor, which allows authors to expand descriptions at user-directed points, and Potential User Questions, which prompt authors to address ambiguous elements. The system was evaluated through a within-subjects study where authors described their own figures across a diverse set of figures and fields of study. The study found that the system assisted in rapid drafting and editing of descriptive alt text through different strategies based on author needs. Interactive features enhanced the experience without increasing cognitive load or effort on average, and enabled greater deviation from generated drafts by supporting iterative refinement. The system's design incorporates structured prompts, metadata extraction, and a user interface that provides AI-assisted support throughout the alt text drafting process. The system was implemented using TypeScript and Python, with a frontend built using ReactJS, Next.js, Zustand, and Mantine, and a backend using Flask and PostgreSQL. The study aimed to evaluate the usefulness of the system for assisting authors in producing alt text, examining whether authors perceived benefit from the system's scaffolding and pre-generated drafts, whether interactive features supported further enhancement of descriptions, and whether added features incurred additional cognitive load. The results showed that the system was effective in helping authors produce high-quality alt text for scientific figures.
Reach us at info@study.space