The paper introduces GraphToken, a parameter-efficient method for encoding structured data into sequential form for use in large language models (LLMs). Unlike previous work that focuses on limited domains, GraphToken is designed to handle general structured data for various reasoning tasks. The method learns an encoding function to extend prompts with explicit structured information, significantly improving performance on graph reasoning tasks. Specifically, it achieves up to 73% improvements on node, edge, and graph-level tasks from the GraphQA benchmark. The paper also discusses the architecture and training procedure of GraphToken, and provides extensive experimental results demonstrating its effectiveness. Additionally, it explores the impact of different graph encoders and feature choices, showing that breaking equivariance can enhance graph reasoning capabilities. The work opens new avenues for reasoning with structured data and LLMs, addressing issues such as hallucinations, factuality, and freshness.The paper introduces GraphToken, a parameter-efficient method for encoding structured data into sequential form for use in large language models (LLMs). Unlike previous work that focuses on limited domains, GraphToken is designed to handle general structured data for various reasoning tasks. The method learns an encoding function to extend prompts with explicit structured information, significantly improving performance on graph reasoning tasks. Specifically, it achieves up to 73% improvements on node, edge, and graph-level tasks from the GraphQA benchmark. The paper also discusses the architecture and training procedure of GraphToken, and provides extensive experimental results demonstrating its effectiveness. Additionally, it explores the impact of different graph encoders and feature choices, showing that breaking equivariance can enhance graph reasoning capabilities. The work opens new avenues for reasoning with structured data and LLMs, addressing issues such as hallucinations, factuality, and freshness.