The paper introduces a new class of algorithms called residual algorithms for reinforcement learning with function approximation systems. It begins by discussing the limitations of direct algorithms, which are guaranteed to converge with lookup tables but can become unstable when implemented with general function-approximation systems. The paper then presents residual gradient algorithms, which perform gradient descent on the mean squared Bellman residual and are guaranteed to converge but may learn slowly in some cases. To address these issues, a broader class of residual algorithms is proposed, which combines the advantages of both direct and residual gradient algorithms while ensuring guaranteed convergence and fast learning. The paper provides theoretical analysis and simulation results to support these claims, demonstrating that residual algorithms can effectively handle high-dimensional Markov Decision Problems (MDPs) and function approximation systems. The algorithms are applicable to various reinforcement learning methods, including value iteration, Q-learning, and advantage learning, and can be used in both deterministic and stochastic MDPs. The paper concludes by highlighting the benefits of residual algorithms in combining fast learning with guaranteed convergence, making them suitable for practical reinforcement learning applications.The paper introduces a new class of algorithms called residual algorithms for reinforcement learning with function approximation systems. It begins by discussing the limitations of direct algorithms, which are guaranteed to converge with lookup tables but can become unstable when implemented with general function-approximation systems. The paper then presents residual gradient algorithms, which perform gradient descent on the mean squared Bellman residual and are guaranteed to converge but may learn slowly in some cases. To address these issues, a broader class of residual algorithms is proposed, which combines the advantages of both direct and residual gradient algorithms while ensuring guaranteed convergence and fast learning. The paper provides theoretical analysis and simulation results to support these claims, demonstrating that residual algorithms can effectively handle high-dimensional Markov Decision Problems (MDPs) and function approximation systems. The algorithms are applicable to various reinforcement learning methods, including value iteration, Q-learning, and advantage learning, and can be used in both deterministic and stochastic MDPs. The paper concludes by highlighting the benefits of residual algorithms in combining fast learning with guaranteed convergence, making them suitable for practical reinforcement learning applications.