PhysAvatar is a novel framework that combines inverse rendering with inverse physics to automatically estimate the shape and appearance of a human from multi-view video data, along with the physical parameters of the fabric of their clothes. The method uses mesh-aligned 4D Gaussian techniques for spatio-temporal mesh tracking and a physically based inverse renderer to estimate intrinsic material properties. A physics simulator estimates the physical parameters of the garments using gradient-based optimization. This integration enables PhysAvatar to create high-quality novel-view renderings of avatars dressed in loose-fitting clothes under unseen motions and lighting conditions. The key contributions include a new inverse rendering paradigm that incorporates the physics of loose garments and a comprehensive pipeline for accurate and efficient mesh reconstruction, tracking, and appearance estimation. PhysAvatar demonstrates state-of-the-art performance in geometry detail and appearance modeling, making it a significant advancement in modeling photorealistic digital humans.PhysAvatar is a novel framework that combines inverse rendering with inverse physics to automatically estimate the shape and appearance of a human from multi-view video data, along with the physical parameters of the fabric of their clothes. The method uses mesh-aligned 4D Gaussian techniques for spatio-temporal mesh tracking and a physically based inverse renderer to estimate intrinsic material properties. A physics simulator estimates the physical parameters of the garments using gradient-based optimization. This integration enables PhysAvatar to create high-quality novel-view renderings of avatars dressed in loose-fitting clothes under unseen motions and lighting conditions. The key contributions include a new inverse rendering paradigm that incorporates the physics of loose garments and a comprehensive pipeline for accurate and efficient mesh reconstruction, tracking, and appearance estimation. PhysAvatar demonstrates state-of-the-art performance in geometry detail and appearance modeling, making it a significant advancement in modeling photorealistic digital humans.