Physics3D is a novel method for learning various physical properties of 3D objects through a video diffusion model. The method involves designing a highly generalizable physical simulation system based on a viscoelastic material model, enabling the simulation of a wide range of materials with high fidelity. The approach distills physical priors from a video diffusion model that contains more understanding of realistic object materials. Extensive experiments demonstrate the effectiveness of the method with both elastic and plastic materials. Physics3D shows great potential for bridging the gap between the physical world and virtual neural space, providing a better integration and application of realistic physical principles in virtual environments. The method uses a viscoelastic Material Point Method (MPM) to simulate 3D dynamics and leverages the capabilities of the differentiable MPM to optimize physical parameters via the Score Distillation Sampling (SDS) strategy. The method achieves high-fidelity and realistic performance in a wide range of materials. The key contributions include proposing a novel generalizable physical simulation system, introducing a viscoelastic MPM to simulate both viscosity and elasticity, and designing a learnable internal filling strategy to optimize part of 3D Gaussians and utilize the SDS strategy to optimize physical parameters from the video diffusion model. The method is effective in creating high-fidelity and realistic 3D dynamics, ready for various interactions across users and objects in the future.Physics3D is a novel method for learning various physical properties of 3D objects through a video diffusion model. The method involves designing a highly generalizable physical simulation system based on a viscoelastic material model, enabling the simulation of a wide range of materials with high fidelity. The approach distills physical priors from a video diffusion model that contains more understanding of realistic object materials. Extensive experiments demonstrate the effectiveness of the method with both elastic and plastic materials. Physics3D shows great potential for bridging the gap between the physical world and virtual neural space, providing a better integration and application of realistic physical principles in virtual environments. The method uses a viscoelastic Material Point Method (MPM) to simulate 3D dynamics and leverages the capabilities of the differentiable MPM to optimize physical parameters via the Score Distillation Sampling (SDS) strategy. The method achieves high-fidelity and realistic performance in a wide range of materials. The key contributions include proposing a novel generalizable physical simulation system, introducing a viscoelastic MPM to simulate both viscosity and elasticity, and designing a learnable internal filling strategy to optimize part of 3D Gaussians and utilize the SDS strategy to optimize physical parameters from the video diffusion model. The method is effective in creating high-fidelity and realistic 3D dynamics, ready for various interactions across users and objects in the future.