Structured optical materials are revolutionizing computing paradigms by leveraging photons for applications in machine learning, computer vision, imaging, telecommunications, and sensing. This perspective highlights the potential of free-space optical systems based on engineered surfaces, particularly diffractive networks, which integrate deep-learning principles to enhance the functionality of free-space optical systems. Metasurfaces, with subwavelength units, offer unique optical responses, enabling independent control over various properties of light and advancing computational throughput and data-transfer bandwidth. Unlike integrated photonics, free-space optical processors directly access all optical degrees of freedom without preprocessing, making them advantageous for tasks such as all-optical statistical inference, wireless telecommunications, and computational imaging. The perspective discusses the design and fabrication of diffractive surfaces and metasurfaces, emphasizing their role in advancing free-space optical computing. Diffractive surfaces are engineered at the wavelength scale to modulate light transmission and reflection, while metasurfaces consist of subwavelength units that tailor the optical wavefront. These structures can perform polarization processing, spatial processing, universal linear transformations, and spectral and temporal processing of waves. Key applications include all-optical machine learning, statistical inference, computational camera and microscope design, and telecommunications. The perspective also addresses grand challenges in free-space optical computing, such as computational accuracy, reconfigurability, fabrication complexity, and energy efficiency. It highlights the importance of improving diffraction efficiency, tuning and reconfigurability, and achieving competitive computation accuracy compared to electronic systems. Overall, the integration of diffractive surfaces and metasurfaces in free-space optical computing offers transformative opportunities for next-generation technologies, particularly in wave-based computing, all-optical statistical inference, and wireless communications.Structured optical materials are revolutionizing computing paradigms by leveraging photons for applications in machine learning, computer vision, imaging, telecommunications, and sensing. This perspective highlights the potential of free-space optical systems based on engineered surfaces, particularly diffractive networks, which integrate deep-learning principles to enhance the functionality of free-space optical systems. Metasurfaces, with subwavelength units, offer unique optical responses, enabling independent control over various properties of light and advancing computational throughput and data-transfer bandwidth. Unlike integrated photonics, free-space optical processors directly access all optical degrees of freedom without preprocessing, making them advantageous for tasks such as all-optical statistical inference, wireless telecommunications, and computational imaging. The perspective discusses the design and fabrication of diffractive surfaces and metasurfaces, emphasizing their role in advancing free-space optical computing. Diffractive surfaces are engineered at the wavelength scale to modulate light transmission and reflection, while metasurfaces consist of subwavelength units that tailor the optical wavefront. These structures can perform polarization processing, spatial processing, universal linear transformations, and spectral and temporal processing of waves. Key applications include all-optical machine learning, statistical inference, computational camera and microscope design, and telecommunications. The perspective also addresses grand challenges in free-space optical computing, such as computational accuracy, reconfigurability, fabrication complexity, and energy efficiency. It highlights the importance of improving diffraction efficiency, tuning and reconfigurability, and achieving competitive computation accuracy compared to electronic systems. Overall, the integration of diffractive surfaces and metasurfaces in free-space optical computing offers transformative opportunities for next-generation technologies, particularly in wave-based computing, all-optical statistical inference, and wireless communications.