The paper explores the relational knowledge stored in pre-trained language models, particularly BERT, and compares it to traditional knowledge bases. The authors introduce the LAMA (LAnguage Model Analysis) probe to evaluate the factual and commonsense knowledge in these models. Key findings include: 1. **BERT's Relational Knowledge**: BERT, without fine-tuning, captures relational knowledge comparable to traditional NLP methods that have access to oracle knowledge. 2. **Performance on Open-Domain QA**: BERT performs well on open-domain question answering tasks, achieving 57.1% precision@10 compared to 63.5% for a supervised baseline. 3. **Knowledge Type Sensitivity**: BERT learns certain types of factual knowledge more readily than others, with poor performance on N-to-M relations. 4. **Model Robustness**: BERT is more robust to query phrasing compared to other models, and it consistently outperforms other language models in recovering factual and commonsense knowledge. The study highlights the potential of pre-trained language models as unsupervised open-domain QA systems, suggesting that they could be used to transfer factual and commonsense knowledge to downstream tasks. The code for reproducing the analysis is available at <https://github.com/facebookresearch/LAMA>.The paper explores the relational knowledge stored in pre-trained language models, particularly BERT, and compares it to traditional knowledge bases. The authors introduce the LAMA (LAnguage Model Analysis) probe to evaluate the factual and commonsense knowledge in these models. Key findings include: 1. **BERT's Relational Knowledge**: BERT, without fine-tuning, captures relational knowledge comparable to traditional NLP methods that have access to oracle knowledge. 2. **Performance on Open-Domain QA**: BERT performs well on open-domain question answering tasks, achieving 57.1% precision@10 compared to 63.5% for a supervised baseline. 3. **Knowledge Type Sensitivity**: BERT learns certain types of factual knowledge more readily than others, with poor performance on N-to-M relations. 4. **Model Robustness**: BERT is more robust to query phrasing compared to other models, and it consistently outperforms other language models in recovering factual and commonsense knowledge. The study highlights the potential of pre-trained language models as unsupervised open-domain QA systems, suggesting that they could be used to transfer factual and commonsense knowledge to downstream tasks. The code for reproducing the analysis is available at <https://github.com/facebookresearch/LAMA>.