This paper presents NEGOTIATION ARENA, a flexible open-source platform for evaluating and analyzing the negotiation abilities of large language models (LLMs). The platform includes three types of scenarios: resource exchange, multi-turn ultimatum, and seller-buyer games. These scenarios allow for multiple turns of flexible dialogue between LLM agents, enabling complex negotiations. The study finds that LLMs can significantly improve their negotiation outcomes by employing certain behavioral tactics, such as pretending to be desperate or acting aggressively. For example, LLMs can improve their payoffs by 20% when negotiating against the standard GPT-4. The paper also quantifies irrational negotiation behaviors exhibited by LLMs, many of which are also observed in humans. NEGOTIATION ARENA provides a new environment to investigate LLM interactions, enabling new insights into LLMs' theory of mind, irrationality, and reasoning abilities. The study compares the negotiation performance of several LLMs, including GPT-4, GPT-3.5, and Claude models. The results show that GPT-4 is the best negotiator overall, while other models exhibit various biases and limitations. The study also explores the impact of social behaviors on negotiation outcomes, finding that agents who act cunningly or desperately can achieve higher payoffs. Additionally, the paper identifies several irrational behaviors in LLMs, such as anchoring bias, and discusses the implications for LLM reliability. Overall, the study highlights the importance of understanding LLM negotiation behaviors and the potential for improving their performance through further research.This paper presents NEGOTIATION ARENA, a flexible open-source platform for evaluating and analyzing the negotiation abilities of large language models (LLMs). The platform includes three types of scenarios: resource exchange, multi-turn ultimatum, and seller-buyer games. These scenarios allow for multiple turns of flexible dialogue between LLM agents, enabling complex negotiations. The study finds that LLMs can significantly improve their negotiation outcomes by employing certain behavioral tactics, such as pretending to be desperate or acting aggressively. For example, LLMs can improve their payoffs by 20% when negotiating against the standard GPT-4. The paper also quantifies irrational negotiation behaviors exhibited by LLMs, many of which are also observed in humans. NEGOTIATION ARENA provides a new environment to investigate LLM interactions, enabling new insights into LLMs' theory of mind, irrationality, and reasoning abilities. The study compares the negotiation performance of several LLMs, including GPT-4, GPT-3.5, and Claude models. The results show that GPT-4 is the best negotiator overall, while other models exhibit various biases and limitations. The study also explores the impact of social behaviors on negotiation outcomes, finding that agents who act cunningly or desperately can achieve higher payoffs. Additionally, the paper identifies several irrational behaviors in LLMs, such as anchoring bias, and discusses the implications for LLM reliability. Overall, the study highlights the importance of understanding LLM negotiation behaviors and the potential for improving their performance through further research.