ClinicalBERT is a model designed to represent clinical notes and predict hospital readmission. It is based on BERT, a transformer-based model, and is pre-trained on clinical notes to capture medical concepts and relationships. ClinicalBERT outperforms existing methods in predicting 30-day hospital readmission using discharge summaries and early ICU notes. It also provides interpretable predictions through attention weights, which highlight relevant clinical elements. The model is flexible and can be adapted to other clinical tasks such as mortality prediction and disease prediction. ClinicalBERT is open-sourced, allowing researchers to use its parameters and scripts for training and evaluation. It improves upon previous clinical text processing methods and can be adapted to other predictive tasks with minimal engineering. ClinicalBERT uses a combination of masked language modeling and next sentence prediction during pre-training. It is evaluated on the MIMIC-III dataset, which includes electronic health records of patients in intensive care units. ClinicalBERT demonstrates superior performance in readmission prediction, particularly in terms of recall and precision. It also provides interpretable predictions by visualizing attention weights, which can help clinicians understand the model's decision-making process. The model is trained on clinical notes and can be used for various downstream tasks, including mortality prediction and length-of-stay assessment. ClinicalBERT is a valuable tool for improving clinical decision-making and reducing hospital readmissions.ClinicalBERT is a model designed to represent clinical notes and predict hospital readmission. It is based on BERT, a transformer-based model, and is pre-trained on clinical notes to capture medical concepts and relationships. ClinicalBERT outperforms existing methods in predicting 30-day hospital readmission using discharge summaries and early ICU notes. It also provides interpretable predictions through attention weights, which highlight relevant clinical elements. The model is flexible and can be adapted to other clinical tasks such as mortality prediction and disease prediction. ClinicalBERT is open-sourced, allowing researchers to use its parameters and scripts for training and evaluation. It improves upon previous clinical text processing methods and can be adapted to other predictive tasks with minimal engineering. ClinicalBERT uses a combination of masked language modeling and next sentence prediction during pre-training. It is evaluated on the MIMIC-III dataset, which includes electronic health records of patients in intensive care units. ClinicalBERT demonstrates superior performance in readmission prediction, particularly in terms of recall and precision. It also provides interpretable predictions by visualizing attention weights, which can help clinicians understand the model's decision-making process. The model is trained on clinical notes and can be used for various downstream tasks, including mortality prediction and length-of-stay assessment. ClinicalBERT is a valuable tool for improving clinical decision-making and reducing hospital readmissions.