This paper addresses the problem of objective inconsistency in federated optimization, where clients have heterogeneous local datasets and computation speeds, leading to varying numbers of local updates per communication round. Naive aggregation of models results in convergence to a stationary point of a mismatched objective function, which can differ significantly from the true objective. The authors propose a general framework to analyze the convergence of federated heterogeneous optimization algorithms, subsuming existing methods like FedAvg and FedProx. They provide a principled understanding of solution bias and convergence slowdown due to objective inconsistency. Based on this analysis, they propose FedNova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence. FedNova normalizes local gradients before averaging, ensuring consistency and improving convergence. The paper also discusses the convergence analysis of FedAvg and FedProx, showing that FedAvg converges to a surrogate objective, while FedProx improves convergence but still suffers from inconsistency. The authors demonstrate that FedNova outperforms existing methods in heterogeneous settings, achieving better accuracy and faster convergence. Theoretical and experimental results show that FedNova effectively handles heterogeneous local progress and improves performance in non-IID data scenarios. The paper concludes that the proposed framework provides a deeper understanding of the impact of heterogeneity on federated learning and offers a novel solution to the objective inconsistency problem.This paper addresses the problem of objective inconsistency in federated optimization, where clients have heterogeneous local datasets and computation speeds, leading to varying numbers of local updates per communication round. Naive aggregation of models results in convergence to a stationary point of a mismatched objective function, which can differ significantly from the true objective. The authors propose a general framework to analyze the convergence of federated heterogeneous optimization algorithms, subsuming existing methods like FedAvg and FedProx. They provide a principled understanding of solution bias and convergence slowdown due to objective inconsistency. Based on this analysis, they propose FedNova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence. FedNova normalizes local gradients before averaging, ensuring consistency and improving convergence. The paper also discusses the convergence analysis of FedAvg and FedProx, showing that FedAvg converges to a surrogate objective, while FedProx improves convergence but still suffers from inconsistency. The authors demonstrate that FedNova outperforms existing methods in heterogeneous settings, achieving better accuracy and faster convergence. Theoretical and experimental results show that FedNova effectively handles heterogeneous local progress and improves performance in non-IID data scenarios. The paper concludes that the proposed framework provides a deeper understanding of the impact of heterogeneity on federated learning and offers a novel solution to the objective inconsistency problem.