This book provides an in-depth treatment of reinforcement learning (RL) and dynamic programming (DP) using function approximators. It begins with an introduction to classical DP and RL, followed by an extensive review of state-of-the-art approaches with approximation. Theoretical guarantees are provided, and numerical examples illustrate the properties of individual methods. The remaining chapters present detailed algorithms from three major classes: value iteration, policy iteration, and policy search. The properties and performance of these algorithms are highlighted in simulation and experimental studies on various control applications. The book is suitable for researchers, teachers, graduate students, and practitioners in optimal and adaptive control, machine learning, and artificial intelligence. It is structured to provide a balanced combination of practical algorithms, theoretical analysis, and comprehensive examples. Readers unfamiliar with the field are advised to start with Chapter 1, followed by Chapters 2 and 3. Those familiar with RL and DP can skip to Chapter 3. The book is divided into three main parts: an introduction to DP and RL, methods with function approximation, and detailed algorithms. Each chapter includes experimental studies and examples, with a focus on control applications. The book discusses the challenges of representing solutions for large and continuous state-action spaces, emphasizing the need for function approximators. It covers approximation architectures, including parametric and nonparametric methods, and their convergence properties. The text also explores various algorithms for approximate value iteration, policy iteration, and policy search, with a focus on their performance in control applications. The authors emphasize the importance of function approximators in solving real-world control problems, particularly in scenarios where a model is not available. The book provides a comprehensive overview of RL and DP, with a focus on their application in control systems. It includes detailed discussions on approximation methods, theoretical guarantees, and experimental studies, making it a valuable resource for researchers and practitioners in the field. The book is supported by a website with additional information, including computer code used in experimental studies.This book provides an in-depth treatment of reinforcement learning (RL) and dynamic programming (DP) using function approximators. It begins with an introduction to classical DP and RL, followed by an extensive review of state-of-the-art approaches with approximation. Theoretical guarantees are provided, and numerical examples illustrate the properties of individual methods. The remaining chapters present detailed algorithms from three major classes: value iteration, policy iteration, and policy search. The properties and performance of these algorithms are highlighted in simulation and experimental studies on various control applications. The book is suitable for researchers, teachers, graduate students, and practitioners in optimal and adaptive control, machine learning, and artificial intelligence. It is structured to provide a balanced combination of practical algorithms, theoretical analysis, and comprehensive examples. Readers unfamiliar with the field are advised to start with Chapter 1, followed by Chapters 2 and 3. Those familiar with RL and DP can skip to Chapter 3. The book is divided into three main parts: an introduction to DP and RL, methods with function approximation, and detailed algorithms. Each chapter includes experimental studies and examples, with a focus on control applications. The book discusses the challenges of representing solutions for large and continuous state-action spaces, emphasizing the need for function approximators. It covers approximation architectures, including parametric and nonparametric methods, and their convergence properties. The text also explores various algorithms for approximate value iteration, policy iteration, and policy search, with a focus on their performance in control applications. The authors emphasize the importance of function approximators in solving real-world control problems, particularly in scenarios where a model is not available. The book provides a comprehensive overview of RL and DP, with a focus on their application in control systems. It includes detailed discussions on approximation methods, theoretical guarantees, and experimental studies, making it a valuable resource for researchers and practitioners in the field. The book is supported by a website with additional information, including computer code used in experimental studies.