This paper presents a comprehensive benchmark study on the performance of structure-based graph neural networks (GNNs) in predicting properties of out-of-distribution (OOD) materials. The study aims to evaluate the generalization capabilities of GNN models in real-world scenarios where materials deviate from the training set distribution. The authors formulate five different categories of OOD problems using three benchmark datasets from the MatBench study. Through extensive experiments, they find that current state-of-the-art GNN algorithms significantly underperform on OOD property prediction tasks compared to their baseline performances in the MatBench study, highlighting a crucial generalization gap. The study also examines the latent physical spaces of these GNN models and identifies the sources of better OOD performance for CGCNN, ALIGNNN, and DeeperGATGNN, while noting the subpar performance of coGN and coNGN. The findings suggest the need for methods like domain adaptation to improve the OOD prediction performance of GNN models. The work provides insights into the challenges and directions for advancing GNNs in OOD materials property prediction.This paper presents a comprehensive benchmark study on the performance of structure-based graph neural networks (GNNs) in predicting properties of out-of-distribution (OOD) materials. The study aims to evaluate the generalization capabilities of GNN models in real-world scenarios where materials deviate from the training set distribution. The authors formulate five different categories of OOD problems using three benchmark datasets from the MatBench study. Through extensive experiments, they find that current state-of-the-art GNN algorithms significantly underperform on OOD property prediction tasks compared to their baseline performances in the MatBench study, highlighting a crucial generalization gap. The study also examines the latent physical spaces of these GNN models and identifies the sources of better OOD performance for CGCNN, ALIGNNN, and DeeperGATGNN, while noting the subpar performance of coGN and coNGN. The findings suggest the need for methods like domain adaptation to improve the OOD prediction performance of GNN models. The work provides insights into the challenges and directions for advancing GNNs in OOD materials property prediction.