Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models

Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models

11 Mar 2024 | P. Harris, M. Kagan, J. Krupa, B. Maier, N. Woodward
The paper introduces RS3L (Re-simulation-based Self-Supervised Learning), a novel simulation-based strategy for self-supervised learning (SSL) that enhances data augmentation through re-simulation. RS3L intervenes in the simulation process, fixing the upstream latent state and re-simulating downstream components to generate multiple realizations of an event, thereby creating a set of augmentations covering all physics-driven variations available in the simulator. This approach is particularly useful in high-energy physics (HEP) for developing foundation models, which are pre-trained on generic tasks and can be adapted for various downstream tasks. The authors demonstrate that RS3L enables powerful performance in downstream tasks such as object discrimination and uncertainty mitigation. They focus on jet tagging, a canonical task in HEP, where the goal is to identify the type of elementary particle that initiated a particle shower in detectors at the Large Hadron Collider (LHC). By re-simulating parton showers and hadronization, RS3L captures the variability in jet formation, leading to a more robust and informative representation space. The paper includes a detailed experimental setup, including the use of high-fidelity simulators and graph neural networks (GNNs) for processing jet data. The results show that RS3L outperforms fully-supervised methods in terms of classification accuracy and robustness against domain shift. The authors also provide a systematic study of the gains of re-simulation-driven contrastive learning compared to fully-supervised learning strategies. Finally, the paper makes the RS3L dataset publicly available and discusses future directions, including exploring alternative SSL approaches and varying the dataset size to further understand the performance of SSL, fine-tuning, and fully-supervised strategies.The paper introduces RS3L (Re-simulation-based Self-Supervised Learning), a novel simulation-based strategy for self-supervised learning (SSL) that enhances data augmentation through re-simulation. RS3L intervenes in the simulation process, fixing the upstream latent state and re-simulating downstream components to generate multiple realizations of an event, thereby creating a set of augmentations covering all physics-driven variations available in the simulator. This approach is particularly useful in high-energy physics (HEP) for developing foundation models, which are pre-trained on generic tasks and can be adapted for various downstream tasks. The authors demonstrate that RS3L enables powerful performance in downstream tasks such as object discrimination and uncertainty mitigation. They focus on jet tagging, a canonical task in HEP, where the goal is to identify the type of elementary particle that initiated a particle shower in detectors at the Large Hadron Collider (LHC). By re-simulating parton showers and hadronization, RS3L captures the variability in jet formation, leading to a more robust and informative representation space. The paper includes a detailed experimental setup, including the use of high-fidelity simulators and graph neural networks (GNNs) for processing jet data. The results show that RS3L outperforms fully-supervised methods in terms of classification accuracy and robustness against domain shift. The authors also provide a systematic study of the gains of re-simulation-driven contrastive learning compared to fully-supervised learning strategies. Finally, the paper makes the RS3L dataset publicly available and discusses future directions, including exploring alternative SSL approaches and varying the dataset size to further understand the performance of SSL, fine-tuning, and fully-supervised strategies.
Reach us at info@study.space