This paper presents FedSARSA, a novel federated on-policy reinforcement learning algorithm that addresses the challenges of federated reinforcement learning (FRL) in non-asymptotic settings. The algorithm integrates SARSA, a classic on-policy temporal difference (TD) control method, into a federated learning framework to enable collaborative learning across multiple agents with potentially different environments. FedSARSA is equipped with linear function approximation, which allows it to handle continuous state-action spaces and provides a finite-time error analysis. The paper establishes that FedSARSA converges to a policy that is near-optimal for all agents, with the extent of near-optimality proportional to the level of heterogeneity among the agents' environments. Additionally, it proves that FedSARSA leverages agent collaboration to enable linear speedups as the number of agents increases, which holds for both fixed and adaptive step-size configurations. The authors analyze the convergence of FedSARSA under various conditions, including time-varying behavior policies, environmental heterogeneity, multiple local updates, and client drift. They also provide a finite-time error bound for FedSARSA, demonstrating its sample efficiency and linear speedup in federated collaboration. The results show that FedSARSA can achieve a near-optimal policy for all agents, even in the presence of environmental heterogeneity, and that the algorithm's performance improves with the number of agents. The paper also discusses the implications of the theoretical guarantees, showing that FedSARSA can robustly handle environmental heterogeneity and that the algorithm's performance is significantly improved through collaboration. The simulations demonstrate the effectiveness of FedSARSA in reducing the mean squared error across different levels of heterogeneity and numbers of agents. The results highlight the potential of FedSARSA as a powerful on-policy federated reinforcement learning method for real-world applications.This paper presents FedSARSA, a novel federated on-policy reinforcement learning algorithm that addresses the challenges of federated reinforcement learning (FRL) in non-asymptotic settings. The algorithm integrates SARSA, a classic on-policy temporal difference (TD) control method, into a federated learning framework to enable collaborative learning across multiple agents with potentially different environments. FedSARSA is equipped with linear function approximation, which allows it to handle continuous state-action spaces and provides a finite-time error analysis. The paper establishes that FedSARSA converges to a policy that is near-optimal for all agents, with the extent of near-optimality proportional to the level of heterogeneity among the agents' environments. Additionally, it proves that FedSARSA leverages agent collaboration to enable linear speedups as the number of agents increases, which holds for both fixed and adaptive step-size configurations. The authors analyze the convergence of FedSARSA under various conditions, including time-varying behavior policies, environmental heterogeneity, multiple local updates, and client drift. They also provide a finite-time error bound for FedSARSA, demonstrating its sample efficiency and linear speedup in federated collaboration. The results show that FedSARSA can achieve a near-optimal policy for all agents, even in the presence of environmental heterogeneity, and that the algorithm's performance improves with the number of agents. The paper also discusses the implications of the theoretical guarantees, showing that FedSARSA can robustly handle environmental heterogeneity and that the algorithm's performance is significantly improved through collaboration. The simulations demonstrate the effectiveness of FedSARSA in reducing the mean squared error across different levels of heterogeneity and numbers of agents. The results highlight the potential of FedSARSA as a powerful on-policy federated reinforcement learning method for real-world applications.