This paper proposes ENGINE, a parameter- and memory-efficient fine-tuning method for textual graphs using large language models (LLMs). The key idea is to combine LLMs and graph neural networks (GNNs) through a tunable side structure, which significantly reduces training complexity without impairing the joint model's capacity. The method introduces a lightweight G-Ladder structure that integrates structural information to enhance node representations. ENGINE achieves state-of-the-art performance on various textual graph datasets while significantly improving training efficiency and reducing inference latency. Two variants of ENGINE are proposed: one with caching to accelerate training and another with dynamic early exit to speed up inference. Experimental results show that ENGINE outperforms existing methods in terms of performance, training efficiency, and inference efficiency. The method is applicable to various tasks, including node classification, graph classification, and link prediction. The proposed approach demonstrates the effectiveness of combining LLMs with GNNs in textual graph analysis.This paper proposes ENGINE, a parameter- and memory-efficient fine-tuning method for textual graphs using large language models (LLMs). The key idea is to combine LLMs and graph neural networks (GNNs) through a tunable side structure, which significantly reduces training complexity without impairing the joint model's capacity. The method introduces a lightweight G-Ladder structure that integrates structural information to enhance node representations. ENGINE achieves state-of-the-art performance on various textual graph datasets while significantly improving training efficiency and reducing inference latency. Two variants of ENGINE are proposed: one with caching to accelerate training and another with dynamic early exit to speed up inference. Experimental results show that ENGINE outperforms existing methods in terms of performance, training efficiency, and inference efficiency. The method is applicable to various tasks, including node classification, graph classification, and link prediction. The proposed approach demonstrates the effectiveness of combining LLMs with GNNs in textual graph analysis.