XLNet is a generalized autoregressive pretraining method designed to address the limitations of BERT, which relies on corrupting input sequences with masks and struggles with bidirectional context modeling. XLNet achieves this by maximizing the expected likelihood over all permutations of the factorization order, enabling the model to learn bidirectional contexts. It also integrates ideas from Transformer-XL, such as segment recurrence and relative encoding, to improve performance on tasks involving long text sequences. Empirically, XLNet outperforms BERT on a wide range of tasks, including question answering, natural language inference, sentiment analysis, and document ranking, often by a significant margin. The key contributions of XLNet include its novel pretraining objective, architectural improvements, and the integration of advanced techniques from Transformer-XL.XLNet is a generalized autoregressive pretraining method designed to address the limitations of BERT, which relies on corrupting input sequences with masks and struggles with bidirectional context modeling. XLNet achieves this by maximizing the expected likelihood over all permutations of the factorization order, enabling the model to learn bidirectional contexts. It also integrates ideas from Transformer-XL, such as segment recurrence and relative encoding, to improve performance on tasks involving long text sequences. Empirically, XLNet outperforms BERT on a wide range of tasks, including question answering, natural language inference, sentiment analysis, and document ranking, often by a significant margin. The key contributions of XLNet include its novel pretraining objective, architectural improvements, and the integration of advanced techniques from Transformer-XL.