This article presents a novel method for the exact inversion of partially coherent dynamical electron scattering data to retrieve picometer-scale atomic structure information. The approach employs a fully differentiable, parametrized scheme based on neural network concepts, allowing the inversion of ptychographic data using entirely physical quantities. The method accurately accounts for thermal diffuse scattering in thick specimens using frozen phonons and incorporates atom types, positions, and partial coherence in line with relativistic scattering theory. The approach utilizes 4D experimental data collected in an aberration-corrected momentum-resolved scanning transmission electron microscopy setup to measure atomic positions in a 20 nm thick PbZr0.2Ti0.8O3 ferroelectric with picometer precision, including the discrimination of different atom types and positions in mixed columns. The method addresses the challenge of phaseless inverse multiple scattering by solving for physical parameters characterizing the specimen and illumination from recorded intensities. It uses a few hundred free parameters, including atom positions, types, and thermal displacements, reducing the number of unknowns by four orders of magnitude compared to established methods. The method enables the retrieval of atomic positions, species, and thermal vibrations with high precision, and allows for the direct determination of the specimen's temperature as a differentiable parameter. The method is validated using both experimental and simulated data, demonstrating its ability to accurately retrieve ferroelectric displacements in PbZr0.2Ti0.8O3. The approach is shown to be effective in distinguishing different atom types and positions in mixed columns, and to accurately recover the atomic structure of the specimen. The method also enables the determination of the local chemical composition by analyzing the gradient of occupancies for mixed atomic columns. The study highlights the importance of incorporating thermal diffuse scattering in the inversion process, as it significantly enhances the chemical sensitivity of the method and provides a direct observable of the specimen's temperature. The method is shown to be robust and efficient, with the ability to handle the complexity of fully incoherent thermal diffuse scattering in inverse direction. The approach is also demonstrated to be effective in the experimental study of PbZr0.2Ti0.8O3, enabling the separation of atomic sites below the Rayleigh limit applied to the projected potential. The method is shown to be capable of accurately retrieving the atomic structure of the specimen, including the determination of ferroelectric displacements and the chemical composition of the specimen. The study demonstrates the potential of the method for deciphering structure-property relationships in nanostructures, and highlights the importance of incorporating thermal diffuse scattering in the inversion process.This article presents a novel method for the exact inversion of partially coherent dynamical electron scattering data to retrieve picometer-scale atomic structure information. The approach employs a fully differentiable, parametrized scheme based on neural network concepts, allowing the inversion of ptychographic data using entirely physical quantities. The method accurately accounts for thermal diffuse scattering in thick specimens using frozen phonons and incorporates atom types, positions, and partial coherence in line with relativistic scattering theory. The approach utilizes 4D experimental data collected in an aberration-corrected momentum-resolved scanning transmission electron microscopy setup to measure atomic positions in a 20 nm thick PbZr0.2Ti0.8O3 ferroelectric with picometer precision, including the discrimination of different atom types and positions in mixed columns. The method addresses the challenge of phaseless inverse multiple scattering by solving for physical parameters characterizing the specimen and illumination from recorded intensities. It uses a few hundred free parameters, including atom positions, types, and thermal displacements, reducing the number of unknowns by four orders of magnitude compared to established methods. The method enables the retrieval of atomic positions, species, and thermal vibrations with high precision, and allows for the direct determination of the specimen's temperature as a differentiable parameter. The method is validated using both experimental and simulated data, demonstrating its ability to accurately retrieve ferroelectric displacements in PbZr0.2Ti0.8O3. The approach is shown to be effective in distinguishing different atom types and positions in mixed columns, and to accurately recover the atomic structure of the specimen. The method also enables the determination of the local chemical composition by analyzing the gradient of occupancies for mixed atomic columns. The study highlights the importance of incorporating thermal diffuse scattering in the inversion process, as it significantly enhances the chemical sensitivity of the method and provides a direct observable of the specimen's temperature. The method is shown to be robust and efficient, with the ability to handle the complexity of fully incoherent thermal diffuse scattering in inverse direction. The approach is also demonstrated to be effective in the experimental study of PbZr0.2Ti0.8O3, enabling the separation of atomic sites below the Rayleigh limit applied to the projected potential. The method is shown to be capable of accurately retrieving the atomic structure of the specimen, including the determination of ferroelectric displacements and the chemical composition of the specimen. The study demonstrates the potential of the method for deciphering structure-property relationships in nanostructures, and highlights the importance of incorporating thermal diffuse scattering in the inversion process.