DiffDA: A Diffusion Model for Weather-scale Data Assimilation

DiffDA: A Diffusion Model for Weather-scale Data Assimilation

10 Jun 2024 | Langwen Huang, Lukas Gianinazzi, Yuejiang Yu, Peter D. Dueben, Torsten Hoefer
DiffDA is a diffusion model designed for weather-scale data assimilation, capable of integrating atmospheric variables using predicted states and sparse observations. The model leverages the pretrained GraphCast neural network as its backbone, enabling high-resolution data assimilation at 0.25° (≈30 km) resolution, a significant improvement over existing ML-based models. Experiments using simulated observations from the ERA5 reanalysis dataset show that DiffDA can produce assimilated data consistent with observations, with assimilated initial conditions from sparse observations (less than 0.96% of gridded data) performing nearly as well as those from ERA5 data, with a maximum lead time loss of 24 hours. This makes DiffDA suitable for real-world applications, including creating reanalysis datasets with autoregressive data assimilation. The method employs a denoising diffusion probabilistic model, which is conditioned on both predicted states and sparse observations. For predicted states, the model uses the forecast model's output as an additional input, while for sparse observations, a soft mask is applied to interpolate observations and ensure consistency. The model's flexibility allows it to integrate with other forecast models, ensuring easy updates and maintenance. DiffDA's key advantage is that conditioning on sparse observations occurs only during inference, avoiding the "curse of dimensionality" associated with training with sparse observations. Experiments demonstrate that as the number of observed data points increases, assimilated data converge to the observations. With less than 1% of grid points used for observations, the errors of assimilated data are comparable to 24-hour forecast errors. When used as input for forecast models, the assimilated data result in a maximum lead time loss of 24 hours compared to using ERA5 data. This enables running data assimilation and simulation in an autoregressive cycle, though constraining errors across iterations remains a challenge. The method is computationally efficient, with data assimilation experiments running on a single high-end PC with a GPU in 15–30 minutes per step, significantly reducing computational costs compared to traditional methods. This efficiency opens the possibility of assimilating more observational data that traditional methods discard, leading to more accurate and stable results. DiffDA represents a significant advancement in ML-based data assimilation, offering a new approach for high-resolution weather forecasting and climate modeling.DiffDA is a diffusion model designed for weather-scale data assimilation, capable of integrating atmospheric variables using predicted states and sparse observations. The model leverages the pretrained GraphCast neural network as its backbone, enabling high-resolution data assimilation at 0.25° (≈30 km) resolution, a significant improvement over existing ML-based models. Experiments using simulated observations from the ERA5 reanalysis dataset show that DiffDA can produce assimilated data consistent with observations, with assimilated initial conditions from sparse observations (less than 0.96% of gridded data) performing nearly as well as those from ERA5 data, with a maximum lead time loss of 24 hours. This makes DiffDA suitable for real-world applications, including creating reanalysis datasets with autoregressive data assimilation. The method employs a denoising diffusion probabilistic model, which is conditioned on both predicted states and sparse observations. For predicted states, the model uses the forecast model's output as an additional input, while for sparse observations, a soft mask is applied to interpolate observations and ensure consistency. The model's flexibility allows it to integrate with other forecast models, ensuring easy updates and maintenance. DiffDA's key advantage is that conditioning on sparse observations occurs only during inference, avoiding the "curse of dimensionality" associated with training with sparse observations. Experiments demonstrate that as the number of observed data points increases, assimilated data converge to the observations. With less than 1% of grid points used for observations, the errors of assimilated data are comparable to 24-hour forecast errors. When used as input for forecast models, the assimilated data result in a maximum lead time loss of 24 hours compared to using ERA5 data. This enables running data assimilation and simulation in an autoregressive cycle, though constraining errors across iterations remains a challenge. The method is computationally efficient, with data assimilation experiments running on a single high-end PC with a GPU in 15–30 minutes per step, significantly reducing computational costs compared to traditional methods. This efficiency opens the possibility of assimilating more observational data that traditional methods discard, leading to more accurate and stable results. DiffDA represents a significant advancement in ML-based data assimilation, offering a new approach for high-resolution weather forecasting and climate modeling.
Reach us at info@study.space