Machine Learning (ML) models are increasingly used in weather prediction, often claiming superior performance compared to traditional physics-based models. However, this study highlights key limitations of current ML weather prediction models. The forecasts from ML models show energy spectra significantly different from those of their training reanalysis fields and Numerical Weather Prediction (NWP) models, leading to overly smooth predictions and poor representation of weather phenomena at spatial scales shorter than 300–400 km. These models also fail to accurately represent fundamental physical balances and derived quantities, affecting forecast interpretation and perceived skill. The study analyzes three leading ML models—Pangu-Weather, FourCastNet, and GraphCast—focusing on their physical consistency and fidelity. The results show that these models struggle to reproduce sub-synoptic and mesoscale weather phenomena, lacking the fidelity and physical consistency of physics-based models. This impacts the reliability of their forecasts, particularly for high variability weather events. The spectral characteristics of ML forecasts show reduced energy at higher wavenumbers, leading to smoother predictions and less realistic representation of weather patterns. This is evident in the forecast of Typhoon Doksuri, where Pangu-Weather forecasts showed a less intense cyclone compared to ECMWF IFS forecasts. The study also examines the physical consistency of ML forecasts, finding that they often fail to maintain the geostrophic wind balance and proper divergence and vorticity relationships. This leads to unrealistic vertical motion fields and precipitation patterns. While ML models offer computational advantages, they still rely on physics-based models for training and initialization. The study suggests that ML models should be viewed as estimators of the mean of the forecast probability distribution rather than realistic simulators of the atmosphere. This has implications for their use in ensemble prediction systems. Overall, the study underscores the need for further research to improve the physical consistency and skill of ML weather prediction models, ensuring they can provide reliable forecasts for a wide range of weather phenomena.Machine Learning (ML) models are increasingly used in weather prediction, often claiming superior performance compared to traditional physics-based models. However, this study highlights key limitations of current ML weather prediction models. The forecasts from ML models show energy spectra significantly different from those of their training reanalysis fields and Numerical Weather Prediction (NWP) models, leading to overly smooth predictions and poor representation of weather phenomena at spatial scales shorter than 300–400 km. These models also fail to accurately represent fundamental physical balances and derived quantities, affecting forecast interpretation and perceived skill. The study analyzes three leading ML models—Pangu-Weather, FourCastNet, and GraphCast—focusing on their physical consistency and fidelity. The results show that these models struggle to reproduce sub-synoptic and mesoscale weather phenomena, lacking the fidelity and physical consistency of physics-based models. This impacts the reliability of their forecasts, particularly for high variability weather events. The spectral characteristics of ML forecasts show reduced energy at higher wavenumbers, leading to smoother predictions and less realistic representation of weather patterns. This is evident in the forecast of Typhoon Doksuri, where Pangu-Weather forecasts showed a less intense cyclone compared to ECMWF IFS forecasts. The study also examines the physical consistency of ML forecasts, finding that they often fail to maintain the geostrophic wind balance and proper divergence and vorticity relationships. This leads to unrealistic vertical motion fields and precipitation patterns. While ML models offer computational advantages, they still rely on physics-based models for training and initialization. The study suggests that ML models should be viewed as estimators of the mean of the forecast probability distribution rather than realistic simulators of the atmosphere. This has implications for their use in ensemble prediction systems. Overall, the study underscores the need for further research to improve the physical consistency and skill of ML weather prediction models, ensuring they can provide reliable forecasts for a wide range of weather phenomena.