May 11–16, 2024 | Michelle S. Lam, Janice Teoh, James A. Landay, Jeffrey Heer, Michael S. Bernstein
LLooM is a concept induction algorithm that extracts high-level concepts from unstructured text using large language models (LLMs). Unlike traditional topic modeling, which focuses on low-level keywords, LLooM produces human-interpretable concepts defined by explicit inclusion criteria. The algorithm is implemented in the LLooM Workbench, a mixed-initiative text analysis tool that enables analysts to visualize datasets in terms of high-level concepts. The Workbench allows analysts to interact with data, refine concepts, and explore patterns in a theory-driven manner.
LLooM leverages LLMs to iteratively synthesize sampled text and propose concepts of increasing generality. The algorithm includes a Distill operator to condense data into a compact representation, a Cluster operator to group related items, and a Synthesize operator to generate high-level concepts. The Synthesize operator uses LLMs to identify unifying patterns among examples and generate criteria for evaluating those patterns. The algorithm also includes a Score operator to assess how well each example matches a concept, and a Loop operator to refine concepts through multiple iterations.
The LLooM Workbench provides an interactive interface for analysts to explore data in terms of high-level concepts. It includes a Matrix view that displays the prevalence of concepts across user-defined data slices, a Concept Detail view that shows the criteria and examples associated with a concept, and a Slice Detail view that displays the examples within a specific data slice. Analysts can add, edit, merge, or split concepts to refine their analysis.
LLooM has been evaluated on four real-world datasets: toxic online content, political social media data, academic paper abstracts, and AI research impact statements. In these scenarios, LLooM produced concepts that were more specific and nuanced than those generated by traditional topic models. For example, in the toxic content dataset, LLooM identified concepts such as "Empowerment of women" and "Gender inequality," while BERTopic produced topics like "women, power, female." LLooM also covered more examples than BERTopic, with an average coverage of 93% compared to 77.7% for cluster-based models.
Expert case studies demonstrated that LLooM helped researchers uncover new insights even from familiar datasets. For example, in a political social media dataset, LLooM suggested a previously unnoticed concept of attacks on out-party stances. The algorithm's ability to generate concepts defined by explicit criteria allowed analysts to explore data in a more meaningful and interpretable way. LLooM's concepts improved upon the quality and coverage of topic models, enabling analysts to shift their focus from interpreting topics to engaging in theory-driven analysis.LLooM is a concept induction algorithm that extracts high-level concepts from unstructured text using large language models (LLMs). Unlike traditional topic modeling, which focuses on low-level keywords, LLooM produces human-interpretable concepts defined by explicit inclusion criteria. The algorithm is implemented in the LLooM Workbench, a mixed-initiative text analysis tool that enables analysts to visualize datasets in terms of high-level concepts. The Workbench allows analysts to interact with data, refine concepts, and explore patterns in a theory-driven manner.
LLooM leverages LLMs to iteratively synthesize sampled text and propose concepts of increasing generality. The algorithm includes a Distill operator to condense data into a compact representation, a Cluster operator to group related items, and a Synthesize operator to generate high-level concepts. The Synthesize operator uses LLMs to identify unifying patterns among examples and generate criteria for evaluating those patterns. The algorithm also includes a Score operator to assess how well each example matches a concept, and a Loop operator to refine concepts through multiple iterations.
The LLooM Workbench provides an interactive interface for analysts to explore data in terms of high-level concepts. It includes a Matrix view that displays the prevalence of concepts across user-defined data slices, a Concept Detail view that shows the criteria and examples associated with a concept, and a Slice Detail view that displays the examples within a specific data slice. Analysts can add, edit, merge, or split concepts to refine their analysis.
LLooM has been evaluated on four real-world datasets: toxic online content, political social media data, academic paper abstracts, and AI research impact statements. In these scenarios, LLooM produced concepts that were more specific and nuanced than those generated by traditional topic models. For example, in the toxic content dataset, LLooM identified concepts such as "Empowerment of women" and "Gender inequality," while BERTopic produced topics like "women, power, female." LLooM also covered more examples than BERTopic, with an average coverage of 93% compared to 77.7% for cluster-based models.
Expert case studies demonstrated that LLooM helped researchers uncover new insights even from familiar datasets. For example, in a political social media dataset, LLooM suggested a previously unnoticed concept of attacks on out-party stances. The algorithm's ability to generate concepts defined by explicit criteria allowed analysts to explore data in a more meaningful and interpretable way. LLooM's concepts improved upon the quality and coverage of topic models, enabling analysts to shift their focus from interpreting topics to engaging in theory-driven analysis.