This survey provides a comprehensive review of pre-trained models (PTMs) for natural language processing (NLP). It first introduces language representation learning and its research progress. Then, it systematically categorizes existing PTMs based on four perspectives: representation type, model architecture, pre-training task type, and scenario extensions. Next, it describes how to adapt PTM knowledge to downstream tasks. Finally, it outlines potential future research directions. The survey aims to serve as a hands-on guide for understanding, using, and developing PTMs for various NLP tasks. The survey discusses the evolution of PTMs, starting from first-generation models that learned word embeddings (e.g., Skip-Gram, GloVe) to second-generation models that learned contextual word embeddings (e.g., CoVe, ELMo, OpenAI GPT, BERT). It also covers various pre-training tasks, including supervised, unsupervised, and self-supervised learning, and introduces different PTM architectures such as sequence models, non-sequence models, and self-attention models. The survey also discusses the taxonomy of PTMs, their applications, and challenges in the field. The survey highlights the importance of pre-training in NLP, as it allows models to learn universal language representations that can be adapted to various downstream tasks. It also discusses the limitations of current PTMs and suggests future research directions. The survey provides an overview of PTMs, their pre-training tasks, and their applications in various NLP tasks. It also discusses extensions of PTMs, including knowledge-enriched PTMs, multilingual PTMs, multi-modal PTMs, and domain-specific PTMs. The survey concludes with a discussion of the current challenges and future directions in the field of PTMs.This survey provides a comprehensive review of pre-trained models (PTMs) for natural language processing (NLP). It first introduces language representation learning and its research progress. Then, it systematically categorizes existing PTMs based on four perspectives: representation type, model architecture, pre-training task type, and scenario extensions. Next, it describes how to adapt PTM knowledge to downstream tasks. Finally, it outlines potential future research directions. The survey aims to serve as a hands-on guide for understanding, using, and developing PTMs for various NLP tasks. The survey discusses the evolution of PTMs, starting from first-generation models that learned word embeddings (e.g., Skip-Gram, GloVe) to second-generation models that learned contextual word embeddings (e.g., CoVe, ELMo, OpenAI GPT, BERT). It also covers various pre-training tasks, including supervised, unsupervised, and self-supervised learning, and introduces different PTM architectures such as sequence models, non-sequence models, and self-attention models. The survey also discusses the taxonomy of PTMs, their applications, and challenges in the field. The survey highlights the importance of pre-training in NLP, as it allows models to learn universal language representations that can be adapted to various downstream tasks. It also discusses the limitations of current PTMs and suggests future research directions. The survey provides an overview of PTMs, their pre-training tasks, and their applications in various NLP tasks. It also discusses extensions of PTMs, including knowledge-enriched PTMs, multilingual PTMs, multi-modal PTMs, and domain-specific PTMs. The survey concludes with a discussion of the current challenges and future directions in the field of PTMs.