April 26, 2024 | Benjamin S. Manning, Kehang Zhu, John J. Horton
This paper presents an approach for automatically generating and testing social scientific hypotheses using large language models (LLMs) and structural causal models (SCMs). The method leverages SCMs to define hypotheses, design experiments, and analyze data, enabling the system to simulate social interactions and test causal relationships. The system can autonomously generate hypotheses, design experiments, run simulations with LLM-powered agents, and analyze results. It is demonstrated across four scenarios: a negotiation, a bail hearing, a job interview, and an auction. In each case, the system proposes and tests hypotheses, identifying causal relationships and generating findings. For example, in the auction scenario, the system's results closely matched auction theory, while in the job interview scenario, the candidate's bar exam performance was the key factor in securing the job.
The paper also evaluates the LLM's ability to predict outcomes and path estimates in these scenarios. When given the fitted SCM, the LLM's predictions improved significantly, though they still lagged behind theoretical predictions. The results show that while LLMs can generate insights through simulation, they struggle to accurately predict magnitudes without access to the fitted SCM. The system's use of SCMs ensures that causal relationships are clearly defined and that the experimental design is based on a pre-specified plan, allowing for precise estimation and identification of causal effects.
The paper highlights the advantages of using SCMs over other methods for studying causal relationships in social simulations. SCMs provide a clear framework for hypothesis generation, experimental design, and data analysis, ensuring that the results are reliable and interpretable. The system's ability to generate and test hypotheses autonomously represents a significant advancement in automated social science, enabling efficient exploration of human behavior and social interactions. The paper concludes that such systems could be valuable tools for social science research, offering a scalable and replicable approach to hypothesis generation and testing.This paper presents an approach for automatically generating and testing social scientific hypotheses using large language models (LLMs) and structural causal models (SCMs). The method leverages SCMs to define hypotheses, design experiments, and analyze data, enabling the system to simulate social interactions and test causal relationships. The system can autonomously generate hypotheses, design experiments, run simulations with LLM-powered agents, and analyze results. It is demonstrated across four scenarios: a negotiation, a bail hearing, a job interview, and an auction. In each case, the system proposes and tests hypotheses, identifying causal relationships and generating findings. For example, in the auction scenario, the system's results closely matched auction theory, while in the job interview scenario, the candidate's bar exam performance was the key factor in securing the job.
The paper also evaluates the LLM's ability to predict outcomes and path estimates in these scenarios. When given the fitted SCM, the LLM's predictions improved significantly, though they still lagged behind theoretical predictions. The results show that while LLMs can generate insights through simulation, they struggle to accurately predict magnitudes without access to the fitted SCM. The system's use of SCMs ensures that causal relationships are clearly defined and that the experimental design is based on a pre-specified plan, allowing for precise estimation and identification of causal effects.
The paper highlights the advantages of using SCMs over other methods for studying causal relationships in social simulations. SCMs provide a clear framework for hypothesis generation, experimental design, and data analysis, ensuring that the results are reliable and interpretable. The system's ability to generate and test hypotheses autonomously represents a significant advancement in automated social science, enabling efficient exploration of human behavior and social interactions. The paper concludes that such systems could be valuable tools for social science research, offering a scalable and replicable approach to hypothesis generation and testing.