This paper presents a comprehensive benchmark study of structure-based graph neural networks (GNNs) for out-of-distribution (OOD) materials property prediction. The study evaluates the performance of GNNs on three benchmark datasets from the MatBench study, focusing on their ability to predict material properties for materials that differ from the training set distribution. The study identifies five categories of OOD problems and evaluates the performance of eight GNN models on these problems. The results show that current state-of-the-art GNN algorithms significantly underperform on average compared to their baselines in the MatBench study, demonstrating a crucial generalization gap in realistic material prediction tasks. The study further examines the latent physical spaces of these GNN models and identifies the sources of the significantly more robust OOD performance of CGCNN, ALIGNN, and DeeperGATGNN compared to the current best models in the MatBench study (coGN and coNGN). The study also highlights the need for methods like domain adaptation to improve the OOD prediction performance of current GNN models. The results show that CGCNN, ALIGNN, and DeeperGATGNN perform more robustly on all OOD problems, with CGCNN outperforming coGN and coNGN in some cases. The study concludes that current GNN models are not robust enough to handle OOD property prediction and that enhanced robustness methods, such as domain adaptation, are needed to improve their performance. The study also identifies that the performance of GNN models varies significantly depending on the type of OOD test set, with some models performing better on certain types of OOD test sets than others. The study provides insights into the factors contributing to the performance of GNN models on OOD materials property prediction and suggests directions for future research in this area.This paper presents a comprehensive benchmark study of structure-based graph neural networks (GNNs) for out-of-distribution (OOD) materials property prediction. The study evaluates the performance of GNNs on three benchmark datasets from the MatBench study, focusing on their ability to predict material properties for materials that differ from the training set distribution. The study identifies five categories of OOD problems and evaluates the performance of eight GNN models on these problems. The results show that current state-of-the-art GNN algorithms significantly underperform on average compared to their baselines in the MatBench study, demonstrating a crucial generalization gap in realistic material prediction tasks. The study further examines the latent physical spaces of these GNN models and identifies the sources of the significantly more robust OOD performance of CGCNN, ALIGNN, and DeeperGATGNN compared to the current best models in the MatBench study (coGN and coNGN). The study also highlights the need for methods like domain adaptation to improve the OOD prediction performance of current GNN models. The results show that CGCNN, ALIGNN, and DeeperGATGNN perform more robustly on all OOD problems, with CGCNN outperforming coGN and coNGN in some cases. The study concludes that current GNN models are not robust enough to handle OOD property prediction and that enhanced robustness methods, such as domain adaptation, are needed to improve their performance. The study also identifies that the performance of GNN models varies significantly depending on the type of OOD test set, with some models performing better on certain types of OOD test sets than others. The study provides insights into the factors contributing to the performance of GNN models on OOD materials property prediction and suggests directions for future research in this area.