This paper explores the use of Markov games as a framework for multi-agent reinforcement learning, particularly focusing on two-player zero-sum games. The author defines Markov games, which extend Markov Decision Processes (MDPs) to include multiple agents with opposing goals. The paper discusses the challenges and insights of using Markov games in reinforcement learning, including the need for probabilistic policies to handle uncertain opponents. The introduction highlights the limitations of MDPs in multi-agent environments and introduces Markov games as a more suitable framework. The paper then delves into the mathematical details of optimal policies in Markov games, including the concept of minimax strategies and the use of linear programming to find optimal solutions. The methods section reviews techniques for finding optimal policies in matrix games, MDPs, and Markov games, emphasizing the similarities between these frameworks. The paper introduces the minimax-Q algorithm, which extends Q-learning to handle Markov games by replacing the "max" operator with a "minimax" operator. The experiments section demonstrates the minimax-Q algorithm through a soccer-like game, comparing it with Q-learning and other methods. The results show that the minimax-Q algorithm performs well against random and hand-built opponents, while Q-learning struggles due to its deterministic nature. The discussion section reflects on the strengths and limitations of the minimax criterion and probabilistic policies, highlighting their applications in single-agent and multi-agent environments. The paper concludes by suggesting future research directions, including the exploration of cooperative and multi-player games.This paper explores the use of Markov games as a framework for multi-agent reinforcement learning, particularly focusing on two-player zero-sum games. The author defines Markov games, which extend Markov Decision Processes (MDPs) to include multiple agents with opposing goals. The paper discusses the challenges and insights of using Markov games in reinforcement learning, including the need for probabilistic policies to handle uncertain opponents. The introduction highlights the limitations of MDPs in multi-agent environments and introduces Markov games as a more suitable framework. The paper then delves into the mathematical details of optimal policies in Markov games, including the concept of minimax strategies and the use of linear programming to find optimal solutions. The methods section reviews techniques for finding optimal policies in matrix games, MDPs, and Markov games, emphasizing the similarities between these frameworks. The paper introduces the minimax-Q algorithm, which extends Q-learning to handle Markov games by replacing the "max" operator with a "minimax" operator. The experiments section demonstrates the minimax-Q algorithm through a soccer-like game, comparing it with Q-learning and other methods. The results show that the minimax-Q algorithm performs well against random and hand-built opponents, while Q-learning struggles due to its deterministic nature. The discussion section reflects on the strengths and limitations of the minimax criterion and probabilistic policies, highlighting their applications in single-agent and multi-agent environments. The paper concludes by suggesting future research directions, including the exploration of cooperative and multi-player games.