18 Feb 2020 | Weihua Hu1*, Bowen Liu2*, Joseph Gomes4, Marinka Zitnik5, Percy Liang1, Vijay Pande1, Jure Leskovec1
This paper addresses the challenge of pre-training Graph Neural Networks (GNNs) for downstream tasks, particularly in the context of graph datasets where task-specific labels are scarce and out-of-distribution samples are common. The authors propose a novel pre-training strategy that combines node-level and graph-level pre-training to improve generalization and avoid negative transfer. They develop two self-supervised methods, Context Prediction and Attribute Masking, to capture domain-specific knowledge at both the node and graph levels. The effectiveness of their strategy is demonstrated through extensive experiments on multiple graph classification datasets, showing significant improvements in ROC-AUC scores over non-pre-trained models and achieving state-of-the-art performance in molecular property prediction and protein function prediction. The pre-training strategy is also shown to lead to faster training and convergence during fine-tuning.This paper addresses the challenge of pre-training Graph Neural Networks (GNNs) for downstream tasks, particularly in the context of graph datasets where task-specific labels are scarce and out-of-distribution samples are common. The authors propose a novel pre-training strategy that combines node-level and graph-level pre-training to improve generalization and avoid negative transfer. They develop two self-supervised methods, Context Prediction and Attribute Masking, to capture domain-specific knowledge at both the node and graph levels. The effectiveness of their strategy is demonstrated through extensive experiments on multiple graph classification datasets, showing significant improvements in ROC-AUC scores over non-pre-trained models and achieving state-of-the-art performance in molecular property prediction and protein function prediction. The pre-training strategy is also shown to lead to faster training and convergence during fine-tuning.