October 21–25, 2024 | Bo Pan, Zheng Zhang, Yifei Zhang, Yuntong Hu, and Liang Zhao
This paper proposes a framework for distilling large language models (LLMs) into graph models for text-attributed graph (TAG) learning. The goal is to enable efficient and effective TAG learning without relying on LLMs during inference. The framework leverages the expressive power of LLMs to train an interpreter model that can generate rationales, which are then used to align a student graph model. The interpreter model is trained using text rationales converted into multi-level graph rationales, and the student model is aligned with the interpreter model based on the features of TAGs. The framework addresses the challenges of transferring text rationales to graph rationales and aligning text and graph information during knowledge distillation. The proposed method is validated through extensive experiments on four TAG datasets, showing significant improvements over baseline methods. The results demonstrate that the framework can effectively distill LLM knowledge into graph models, achieving an average improvement of 6.2% across datasets. The method is efficient, scalable, and addresses issues of cost and privacy associated with using LLMs. The framework is designed to be a localized solution that retains the advanced capabilities of LLMs without their associated drawbacks.This paper proposes a framework for distilling large language models (LLMs) into graph models for text-attributed graph (TAG) learning. The goal is to enable efficient and effective TAG learning without relying on LLMs during inference. The framework leverages the expressive power of LLMs to train an interpreter model that can generate rationales, which are then used to align a student graph model. The interpreter model is trained using text rationales converted into multi-level graph rationales, and the student model is aligned with the interpreter model based on the features of TAGs. The framework addresses the challenges of transferring text rationales to graph rationales and aligning text and graph information during knowledge distillation. The proposed method is validated through extensive experiments on four TAG datasets, showing significant improvements over baseline methods. The results demonstrate that the framework can effectively distill LLM knowledge into graph models, achieving an average improvement of 6.2% across datasets. The method is efficient, scalable, and addresses issues of cost and privacy associated with using LLMs. The framework is designed to be a localized solution that retains the advanced capabilities of LLMs without their associated drawbacks.