9 Jul 2024 | Suhong Moon*, Marwa Abdulhai*, Minwoo Kang*, Joseph Suh*, Widyadewi Soedarmadji, Eran Kohen Behar, David M. Chan
This paper introduces "Anthology," a method for conditioning large language models (LLMs) to virtual personas using open-ended life narratives, or "backstories." The goal is to enhance the consistency and reliability of experimental outcomes while ensuring better representation of diverse sub-populations. The method involves generating naturalistic backstories that can be used as conditioning context for LLMs. The backstories are generated by LLMs themselves, allowing for the creation of a diverse set of backstories that reflect a wide range of human demographics. These backstories are then used to condition LLMs to represent individual human voices more accurately.
The methodology includes generating backstories, performing demographic surveys on these personas to estimate their demographics, and selecting a representative set of personas that match a desired demographic distribution. The approach is validated through experiments that approximate well-documented large-scale human studies conducted as part of the Pew Research Center's American Trends Panel (ATP) surveys. The results show that Anthology achieves up to 18% improvement in matching the response distributions of human respondents and 27% improvement in consistency metrics compared to baseline methods.
The paper also discusses the use of LLMs to simulate human actors in behavioral studies, highlighting the potential of LLMs as querying models due to their speed and cost-effectiveness compared to traditional human studies. However, the use of LLMs also raises ethical concerns, including the potential for bias and the need for careful consideration of privacy and consent issues.
The study demonstrates that Anthology provides a more accurate approximation of human responses compared to existing methods, particularly in representing under-represented groups. The approach involves matching backstories to target human populations using demographic surveys and matching algorithms, ensuring that the generated personas closely reflect the characteristics of the target population. The results show that Anthology outperforms other methods in terms of representativeness, consistency, and diversity, with significant improvements in Wasserstein distance and consistency metrics.
The paper also discusses the limitations of the approach, including the potential for bias in the generated personas, the need for careful matching strategies, and the ethical implications of using virtual personas. The study emphasizes the importance of ongoing research to refine Anthology and ensure its ethical application in social science and beyond. The methodology is open-source, with a collection of approximately 10,000 backstories and code for generating, processing, and administering surveys made available for future research and applications.This paper introduces "Anthology," a method for conditioning large language models (LLMs) to virtual personas using open-ended life narratives, or "backstories." The goal is to enhance the consistency and reliability of experimental outcomes while ensuring better representation of diverse sub-populations. The method involves generating naturalistic backstories that can be used as conditioning context for LLMs. The backstories are generated by LLMs themselves, allowing for the creation of a diverse set of backstories that reflect a wide range of human demographics. These backstories are then used to condition LLMs to represent individual human voices more accurately.
The methodology includes generating backstories, performing demographic surveys on these personas to estimate their demographics, and selecting a representative set of personas that match a desired demographic distribution. The approach is validated through experiments that approximate well-documented large-scale human studies conducted as part of the Pew Research Center's American Trends Panel (ATP) surveys. The results show that Anthology achieves up to 18% improvement in matching the response distributions of human respondents and 27% improvement in consistency metrics compared to baseline methods.
The paper also discusses the use of LLMs to simulate human actors in behavioral studies, highlighting the potential of LLMs as querying models due to their speed and cost-effectiveness compared to traditional human studies. However, the use of LLMs also raises ethical concerns, including the potential for bias and the need for careful consideration of privacy and consent issues.
The study demonstrates that Anthology provides a more accurate approximation of human responses compared to existing methods, particularly in representing under-represented groups. The approach involves matching backstories to target human populations using demographic surveys and matching algorithms, ensuring that the generated personas closely reflect the characteristics of the target population. The results show that Anthology outperforms other methods in terms of representativeness, consistency, and diversity, with significant improvements in Wasserstein distance and consistency metrics.
The paper also discusses the limitations of the approach, including the potential for bias in the generated personas, the need for careful matching strategies, and the ethical implications of using virtual personas. The study emphasizes the importance of ongoing research to refine Anthology and ensure its ethical application in social science and beyond. The methodology is open-source, with a collection of approximately 10,000 backstories and code for generating, processing, and administering surveys made available for future research and applications.