POSEIDON: Efficient Foundation Models for PDEs

POSEIDON: Efficient Foundation Models for PDEs

29 May 2024 | Maximilian Herde, Bogdan Raonic, Tobias Rohner, Roger Käppeli, Roberto Molinaro, Emmanuel de Bézenac, Siddhartha Mishra
POSEIDON is a foundation model for learning the solution operators of partial differential equations (PDEs). It is based on a multiscale operator transformer with time-conditioned layer norms that enable continuous-in-time evaluations. A novel training strategy leveraging the semi-group property of time-dependent PDEs allows for significant scaling-up of the training data. POSEIDON is pretrained on a large-scale dataset for the governing equations of fluid dynamics and evaluated on 15 challenging downstream tasks involving various PDE types. It outperforms baselines in terms of sample efficiency and accuracy, generalizes well to new physics, and scales with model and data size. The model is open-sourced with code and datasets available for use. POSEIDON demonstrates the potential of foundation models for PDEs by learning effective representations from a small set of PDEs during pretraining to generalize to unseen and unrelated PDEs. The model's architecture includes a scalable operator transformer (scOT) with SwinV2 attention, a novel all2all training strategy, and a large-scale pretraining dataset. POSEIDON performs well on a variety of downstream tasks, including those involving PDEs not seen during pretraining, and shows significant gains in sample efficiency and accuracy. The model's performance is evaluated on a suite of 15 tasks, and it outperforms other foundation models and neural operators in terms of both accuracy and efficiency. The results demonstrate the feasibility of PDE foundation models and their potential for general-purpose applications.POSEIDON is a foundation model for learning the solution operators of partial differential equations (PDEs). It is based on a multiscale operator transformer with time-conditioned layer norms that enable continuous-in-time evaluations. A novel training strategy leveraging the semi-group property of time-dependent PDEs allows for significant scaling-up of the training data. POSEIDON is pretrained on a large-scale dataset for the governing equations of fluid dynamics and evaluated on 15 challenging downstream tasks involving various PDE types. It outperforms baselines in terms of sample efficiency and accuracy, generalizes well to new physics, and scales with model and data size. The model is open-sourced with code and datasets available for use. POSEIDON demonstrates the potential of foundation models for PDEs by learning effective representations from a small set of PDEs during pretraining to generalize to unseen and unrelated PDEs. The model's architecture includes a scalable operator transformer (scOT) with SwinV2 attention, a novel all2all training strategy, and a large-scale pretraining dataset. POSEIDON performs well on a variety of downstream tasks, including those involving PDEs not seen during pretraining, and shows significant gains in sample efficiency and accuracy. The model's performance is evaluated on a suite of 15 tasks, and it outperforms other foundation models and neural operators in terms of both accuracy and efficiency. The results demonstrate the feasibility of PDE foundation models and their potential for general-purpose applications.
Reach us at info@study.space
Understanding Poseidon%3A Efficient Foundation Models for PDEs