The paper introduces LiDAR Diffusion Models (LiDMs), a novel framework for generating LiDAR-realistic scenes from latent spaces tailored to capture the realism of LiDAR data. LiDMs address the challenges of preserving curve-like patterns and 3D geometry in LiDAR scenes, which are crucial for realistic scene generation. The method focuses on three key aspects: pattern realism, geometry realism, and object realism. Specifically, it introduces curve-wise compression to maintain realistic LiDAR patterns, point-wise coordinate supervision to learn scene geometry, and patch-wise encoding to capture the full context of 3D objects. These techniques enable LiDMs to achieve state-of-the-art performance in unconditional LiDAR generation under 64-beam scenarios, while maintaining high efficiency compared to point-based diffusion models (up to 107× faster). Additionally, LiDMs support various conditions such as semantic maps, camera views, and text prompts, making them versatile for downstream tasks in autonomous driving and robotics. The paper also presents three new perceptual metrics (FRID, FSVD, FPVD) to evaluate the quality of generated LiDAR scenes, providing a comprehensive assessment of LiDMs' performance.The paper introduces LiDAR Diffusion Models (LiDMs), a novel framework for generating LiDAR-realistic scenes from latent spaces tailored to capture the realism of LiDAR data. LiDMs address the challenges of preserving curve-like patterns and 3D geometry in LiDAR scenes, which are crucial for realistic scene generation. The method focuses on three key aspects: pattern realism, geometry realism, and object realism. Specifically, it introduces curve-wise compression to maintain realistic LiDAR patterns, point-wise coordinate supervision to learn scene geometry, and patch-wise encoding to capture the full context of 3D objects. These techniques enable LiDMs to achieve state-of-the-art performance in unconditional LiDAR generation under 64-beam scenarios, while maintaining high efficiency compared to point-based diffusion models (up to 107× faster). Additionally, LiDMs support various conditions such as semantic maps, camera views, and text prompts, making them versatile for downstream tasks in autonomous driving and robotics. The paper also presents three new perceptual metrics (FRID, FSVD, FPVD) to evaluate the quality of generated LiDAR scenes, providing a comprehensive assessment of LiDMs' performance.