Can Language Models Serve as Text-Based World Simulators? This paper investigates whether current language models (LLMs) can serve as text-based world simulators, accurately predicting how actions change different world states without requiring extensive manual coding. The authors introduce a new benchmark called BYTE-SIZED32-State-Prediction, containing a dataset of text game state transitions and accompanying game tasks. They evaluate GPT-4 on this dataset and find that, despite its impressive performance, it is still an unreliable world simulator without further innovations. The study contributes new insights into current LLM capabilities and weaknesses, as well as a novel benchmark to track future progress. The paper explores two approaches to leveraging LLMs in world modeling and simulation: neurosymbolic methods, which use LLMs to generate code in symbolic representation, and direct simulation, where LLMs simulate virtual environments directly. The authors focus on direct simulation, using structured representations in JSON schema to improve simulation accuracy and directly probe LLM abilities. The study evaluates the ability of LLMs to simulate text-based environments, where an agent receives observations and proposes actions in natural language to complete objectives. Each environment is formally represented as a goal-conditioned partially observable Markov decision process (POMDP). The authors propose a prediction task, LLM-as-a-Simulator (LLM-Sim), to quantitatively evaluate LLMs' capacity to serve as reliable simulators. The task involves predicting state transitions, including action-driven and environment-driven transitions, as well as game progress. The authors find that predicting action-driven transitions is easier than predicting environment-driven transitions, and that predicting static transitions is easier than dynamic transitions. They also find that GPT-4 can predict game progress in most cases, but humans outperform GPT-4 on the LLM-Sim task. The study highlights that LLMs are better at simulating the results of user actions but struggle with environment-driven transitions and transitions requiring arithmetic, common-sense, or scientific knowledge. The paper concludes that LLMs are not yet able to reliably act as text world simulators. The authors suggest that further error analysis is needed to understand the limitations of LLMs in simulating complex environments. The study also addresses ethical concerns, noting that LLMs could be used to generate misleading or non-factual information in downstream tasks, such as game simulation. The authors urge researchers and practitioners to use their proposed task and dataset in a mindful manner.Can Language Models Serve as Text-Based World Simulators? This paper investigates whether current language models (LLMs) can serve as text-based world simulators, accurately predicting how actions change different world states without requiring extensive manual coding. The authors introduce a new benchmark called BYTE-SIZED32-State-Prediction, containing a dataset of text game state transitions and accompanying game tasks. They evaluate GPT-4 on this dataset and find that, despite its impressive performance, it is still an unreliable world simulator without further innovations. The study contributes new insights into current LLM capabilities and weaknesses, as well as a novel benchmark to track future progress. The paper explores two approaches to leveraging LLMs in world modeling and simulation: neurosymbolic methods, which use LLMs to generate code in symbolic representation, and direct simulation, where LLMs simulate virtual environments directly. The authors focus on direct simulation, using structured representations in JSON schema to improve simulation accuracy and directly probe LLM abilities. The study evaluates the ability of LLMs to simulate text-based environments, where an agent receives observations and proposes actions in natural language to complete objectives. Each environment is formally represented as a goal-conditioned partially observable Markov decision process (POMDP). The authors propose a prediction task, LLM-as-a-Simulator (LLM-Sim), to quantitatively evaluate LLMs' capacity to serve as reliable simulators. The task involves predicting state transitions, including action-driven and environment-driven transitions, as well as game progress. The authors find that predicting action-driven transitions is easier than predicting environment-driven transitions, and that predicting static transitions is easier than dynamic transitions. They also find that GPT-4 can predict game progress in most cases, but humans outperform GPT-4 on the LLM-Sim task. The study highlights that LLMs are better at simulating the results of user actions but struggle with environment-driven transitions and transitions requiring arithmetic, common-sense, or scientific knowledge. The paper concludes that LLMs are not yet able to reliably act as text world simulators. The authors suggest that further error analysis is needed to understand the limitations of LLMs in simulating complex environments. The study also addresses ethical concerns, noting that LLMs could be used to generate misleading or non-factual information in downstream tasks, such as game simulation. The authors urge researchers and practitioners to use their proposed task and dataset in a mindful manner.