July 14–18, 2024 | Erhan Zhang, Xingzhu Wang, Peiyuan Gong, Yankai Lin, Jiaxin Mao
**USimAgent: Large Language Models for Simulating Search Users**
This paper introduces USimAgent, a novel framework that leverages Large Language Models (LLMs) to simulate user search behaviors. The primary goal is to generate realistic search interaction sequences, including querying, clicking, and stopping actions, for specific search tasks. The proposed simulator aims to address the complex and interdependent nature of user behavior, which traditional methods often fail to capture accurately.
**Key Contributions:**
1. **LLM-Based Simulator:** USimAgent uses LLMs to generate queries, predict clicks, and determine stopping points, thereby simulating complete search sessions.
2. **Contextual Reasoning:** Inspired by the ReAct method, USimAgent expands the action space to include reasoning steps, enhancing the coherence and realism of simulated user behavior.
3. **Performance Evaluation:** Experiments on a real user behavior dataset show that USimAgent outperforms existing methods in query generation and is comparable to traditional models in predicting user clicks and stopping behaviors.
**Methodology:**
- **Problem Formulation:** USimAgent generates a search interaction sequence consisting of alternating queries and clicks, concluding with a session stop action.
- **Reasoning Before Acting:** The simulator uses LLMs to generate reasoning tailored to the task and context, followed by decision-making to determine whether to continue or stop the session.
- **Query Reformulation:** LLMs generate queries based on task descriptions and context.
- **Click Prediction:** LLMs predict clicks on search results, considering task relevance and user preferences.
- **Stopping Behavior:** LLMs decide when to stop the search session based on reasoning and context.
**Experiments:**
- **Datasets and Settings:** The evaluation is based on a public user behavior dataset, which includes nine intricate search tasks.
- **Evaluation Metrics:** BLEU score for query generation, accuracy, precision, recall, and F1 score for click and stopping behavior prediction.
- **Results:** USimAgent demonstrates superior performance in query generation and comparable performance in click and stopping behavior prediction compared to traditional methods.
**Conclusion:**
USimAgent shows promise in using LLMs for user simulation, particularly in query generation. Future work could focus on combining LLMs with broader datasets to further enhance predictive accuracy.**USimAgent: Large Language Models for Simulating Search Users**
This paper introduces USimAgent, a novel framework that leverages Large Language Models (LLMs) to simulate user search behaviors. The primary goal is to generate realistic search interaction sequences, including querying, clicking, and stopping actions, for specific search tasks. The proposed simulator aims to address the complex and interdependent nature of user behavior, which traditional methods often fail to capture accurately.
**Key Contributions:**
1. **LLM-Based Simulator:** USimAgent uses LLMs to generate queries, predict clicks, and determine stopping points, thereby simulating complete search sessions.
2. **Contextual Reasoning:** Inspired by the ReAct method, USimAgent expands the action space to include reasoning steps, enhancing the coherence and realism of simulated user behavior.
3. **Performance Evaluation:** Experiments on a real user behavior dataset show that USimAgent outperforms existing methods in query generation and is comparable to traditional models in predicting user clicks and stopping behaviors.
**Methodology:**
- **Problem Formulation:** USimAgent generates a search interaction sequence consisting of alternating queries and clicks, concluding with a session stop action.
- **Reasoning Before Acting:** The simulator uses LLMs to generate reasoning tailored to the task and context, followed by decision-making to determine whether to continue or stop the session.
- **Query Reformulation:** LLMs generate queries based on task descriptions and context.
- **Click Prediction:** LLMs predict clicks on search results, considering task relevance and user preferences.
- **Stopping Behavior:** LLMs decide when to stop the search session based on reasoning and context.
**Experiments:**
- **Datasets and Settings:** The evaluation is based on a public user behavior dataset, which includes nine intricate search tasks.
- **Evaluation Metrics:** BLEU score for query generation, accuracy, precision, recall, and F1 score for click and stopping behavior prediction.
- **Results:** USimAgent demonstrates superior performance in query generation and comparable performance in click and stopping behavior prediction compared to traditional methods.
**Conclusion:**
USimAgent shows promise in using LLMs for user simulation, particularly in query generation. Future work could focus on combining LLMs with broader datasets to further enhance predictive accuracy.