OmniJet-α: The first cross-task foundation model for particle physics

OmniJet-α: The first cross-task foundation model for particle physics

8 Mar 2024 | Joschka Birk, Anna Hallin, and Gregor Kasieczka
The paper introduces OMNIJET-α, the first cross-task foundation model for particle physics, which successfully transfers knowledge from jet generation to jet tagging. Foundation models are multi-dataset and multi-task machine learning systems that can be pre-trained and fine-tuned for various downstream applications. The development of such models for physics data could significantly improve physics performance while reducing training time and data requirements. OMNIJET-α is built using a transformer-based architecture, with a focus on tokenization of jet constituents. The model uses a Vector Quantized Variational AutoEncoder (VQ-VAE) to create discrete tokens for jet constituents, with a codebook size increased from 512 to 8192 tokens. This allows for better representation of jet data and improved performance in tasks such as jet classification. The model was trained on the JETCLASS dataset, which contains jet-level and constituent-level features for ten different jet types. The model was tested on tasks such as jet generation and jet tagging, with the results showing that the model can transfer knowledge from jet generation to jet tagging. The model achieved high accuracy in jet classification, even with a small number of training examples. The paper also discusses the benefits of foundation models for particle physics, including improved physics performance, reduced computational resources, and the ability to reuse models across datasets and tasks. The study highlights the potential of foundation models in particle physics and demonstrates the successful transfer of knowledge between different tasks. The results show that OMNIJET-α is a promising foundation model for particle physics, with the potential to improve the efficiency and effectiveness of data analysis in the field.The paper introduces OMNIJET-α, the first cross-task foundation model for particle physics, which successfully transfers knowledge from jet generation to jet tagging. Foundation models are multi-dataset and multi-task machine learning systems that can be pre-trained and fine-tuned for various downstream applications. The development of such models for physics data could significantly improve physics performance while reducing training time and data requirements. OMNIJET-α is built using a transformer-based architecture, with a focus on tokenization of jet constituents. The model uses a Vector Quantized Variational AutoEncoder (VQ-VAE) to create discrete tokens for jet constituents, with a codebook size increased from 512 to 8192 tokens. This allows for better representation of jet data and improved performance in tasks such as jet classification. The model was trained on the JETCLASS dataset, which contains jet-level and constituent-level features for ten different jet types. The model was tested on tasks such as jet generation and jet tagging, with the results showing that the model can transfer knowledge from jet generation to jet tagging. The model achieved high accuracy in jet classification, even with a small number of training examples. The paper also discusses the benefits of foundation models for particle physics, including improved physics performance, reduced computational resources, and the ability to reuse models across datasets and tasks. The study highlights the potential of foundation models in particle physics and demonstrates the successful transfer of knowledge between different tasks. The results show that OMNIJET-α is a promising foundation model for particle physics, with the potential to improve the efficiency and effectiveness of data analysis in the field.
Reach us at info@study.space
[slides and audio] OmniJet-%CE%B1%3A The first cross-task foundation model for particle physics