PHYRECON is a novel approach that integrates differentiable rendering and differentiable physics simulation to learn implicit surface representations for 3D scene reconstruction. The method introduces a differentiable particle-based physical simulator and an efficient algorithm, Surface Points Marching Cubes (SP-MC), for transforming SDF-based implicit representations into explicit surface points. This enables accurate computation of 3D rigid body dynamics under gravity, contact, and friction forces, providing detailed information about object shapes. The differentiable pipeline efficiently implements and optimizes the implicit surface representation by integrating feedback from rendering and physical losses. PHYRECON also models both rendering and physical uncertainty to identify and compensate for inconsistent and inaccurate monocular geometric priors. This physical uncertainty further facilitates a novel physics-guided pixel sampling to enhance the learning of slender structures. By integrating these techniques, the model supports differentiable joint modeling of appearance, geometry, and physics. Extensive experiments show that PHYRECON significantly outperforms all state-of-the-art methods in both reconstruction quality and physical plausibility. The results also exhibit superior physical stability in physical simulators, with at least a 40% improvement across all datasets, paving the way for future physics-based applications. The main contributions of PHYRECON are threefold: 1) Introducing the first method that seamlessly bridges neural scene reconstruction and physics simulation through a differentiable particle-based physical simulator and the proposed SP-MC that efficiently transforms implicit representations into explicit surface points. 2) Proposing a novel method that jointly models rendering and physical uncertainties for 3D reconstruction. 3) Extensive experiments demonstrate that the model significantly enhances reconstruction quality and physical plausibility, outperforming state-of-the-art methods. The results exhibit substantial stability improvements, signaling broader potential for physics-demanding applications.PHYRECON is a novel approach that integrates differentiable rendering and differentiable physics simulation to learn implicit surface representations for 3D scene reconstruction. The method introduces a differentiable particle-based physical simulator and an efficient algorithm, Surface Points Marching Cubes (SP-MC), for transforming SDF-based implicit representations into explicit surface points. This enables accurate computation of 3D rigid body dynamics under gravity, contact, and friction forces, providing detailed information about object shapes. The differentiable pipeline efficiently implements and optimizes the implicit surface representation by integrating feedback from rendering and physical losses. PHYRECON also models both rendering and physical uncertainty to identify and compensate for inconsistent and inaccurate monocular geometric priors. This physical uncertainty further facilitates a novel physics-guided pixel sampling to enhance the learning of slender structures. By integrating these techniques, the model supports differentiable joint modeling of appearance, geometry, and physics. Extensive experiments show that PHYRECON significantly outperforms all state-of-the-art methods in both reconstruction quality and physical plausibility. The results also exhibit superior physical stability in physical simulators, with at least a 40% improvement across all datasets, paving the way for future physics-based applications. The main contributions of PHYRECON are threefold: 1) Introducing the first method that seamlessly bridges neural scene reconstruction and physics simulation through a differentiable particle-based physical simulator and the proposed SP-MC that efficiently transforms implicit representations into explicit surface points. 2) Proposing a novel method that jointly models rendering and physical uncertainties for 3D reconstruction. 3) Extensive experiments demonstrate that the model significantly enhances reconstruction quality and physical plausibility, outperforming state-of-the-art methods. The results exhibit substantial stability improvements, signaling broader potential for physics-demanding applications.