AdaptiGraph is a graph-based neural dynamics framework designed for real-time modeling and control of various materials with unknown physical properties. The framework integrates a physical property-conditioned dynamics model with online physical property estimation, enabling robots to adaptively manipulate objects with varying physical properties. The model uses a graph neural network (GNN) to predict particle motion, representing materials as particles and capturing their interactions. The key innovation is a unified model that predicts the motion of diverse materials without retraining, allowing for few-shot adaptation upon encountering new materials. The framework enables accurate dynamics simulation and prediction for objects like ropes, granular media, rigid boxes, and cloth, adapting to different physical properties such as stiffness, granular size, and center of pressure. The method is evaluated on four types of objects: rigid boxes, granular objects, rope-like objects, and cloth-like objects. Experiments show that AdaptiGraph outperforms non-material-conditioned and non-adaptive models in prediction accuracy and task proficiency. The framework uses inverse optimization to estimate physical properties through few-shot interactions, enhancing model fit to observed data. The results demonstrate that the adaptation module provides consistent and interpretable estimates of physical properties, enabling more accurate dynamics estimation and efficient manipulation, especially for objects with extreme or out-of-distribution physical properties. The framework is tested in both simulation and real-world scenarios, showing robust performance across diverse materials and physical properties. AdaptiGraph's ability to adapt to unknown physical properties makes it a flexible and effective solution for robotic manipulation tasks involving various materials.AdaptiGraph is a graph-based neural dynamics framework designed for real-time modeling and control of various materials with unknown physical properties. The framework integrates a physical property-conditioned dynamics model with online physical property estimation, enabling robots to adaptively manipulate objects with varying physical properties. The model uses a graph neural network (GNN) to predict particle motion, representing materials as particles and capturing their interactions. The key innovation is a unified model that predicts the motion of diverse materials without retraining, allowing for few-shot adaptation upon encountering new materials. The framework enables accurate dynamics simulation and prediction for objects like ropes, granular media, rigid boxes, and cloth, adapting to different physical properties such as stiffness, granular size, and center of pressure. The method is evaluated on four types of objects: rigid boxes, granular objects, rope-like objects, and cloth-like objects. Experiments show that AdaptiGraph outperforms non-material-conditioned and non-adaptive models in prediction accuracy and task proficiency. The framework uses inverse optimization to estimate physical properties through few-shot interactions, enhancing model fit to observed data. The results demonstrate that the adaptation module provides consistent and interpretable estimates of physical properties, enabling more accurate dynamics estimation and efficient manipulation, especially for objects with extreme or out-of-distribution physical properties. The framework is tested in both simulation and real-world scenarios, showing robust performance across diverse materials and physical properties. AdaptiGraph's ability to adapt to unknown physical properties makes it a flexible and effective solution for robotic manipulation tasks involving various materials.