LidarDM is a novel LiDAR generative model that produces realistic, layout-aware, physically plausible, and temporally coherent LiDAR videos. It introduces two key capabilities: (i) LiDAR generation guided by driving scenarios, and (ii) 4D LiDAR point cloud generation. The model uses a novel integrated 4D world generation framework, combining latent diffusion models for 3D scene generation with dynamic actors to form a 4D world. It then generates realistic sensory observations within this virtual environment. Experiments show that LidarDM outperforms existing methods in realism, temporal coherence, and layout consistency. It can be used as a generative world model for training and testing perception models. LidarDM is the first LiDAR generative method with the capability to produce temporally coherent LiDAR videos. It also demonstrates strong performance in map-conditioned LiDAR generation, producing realistic and temporally consistent sequences. The model is trained using a dataset that pairs scene geometry with map conditioning. It uses a latent diffusion model to generate 3D scenes and objects, and employs a physics-based ray casting approach to generate realistic LiDAR point clouds. The model is evaluated on KITTI-360 and Waymo Open datasets, showing superior performance in unconditional and map-conditioned LiDAR generation. LidarDM can also be used to augment real data for 3D perception models, improving their performance. The model is limited by its reliance on latent diffusion models, which are not real-time. Future work includes improving LiDAR intensity modeling. The project is supported by various grants and institutions.LidarDM is a novel LiDAR generative model that produces realistic, layout-aware, physically plausible, and temporally coherent LiDAR videos. It introduces two key capabilities: (i) LiDAR generation guided by driving scenarios, and (ii) 4D LiDAR point cloud generation. The model uses a novel integrated 4D world generation framework, combining latent diffusion models for 3D scene generation with dynamic actors to form a 4D world. It then generates realistic sensory observations within this virtual environment. Experiments show that LidarDM outperforms existing methods in realism, temporal coherence, and layout consistency. It can be used as a generative world model for training and testing perception models. LidarDM is the first LiDAR generative method with the capability to produce temporally coherent LiDAR videos. It also demonstrates strong performance in map-conditioned LiDAR generation, producing realistic and temporally consistent sequences. The model is trained using a dataset that pairs scene geometry with map conditioning. It uses a latent diffusion model to generate 3D scenes and objects, and employs a physics-based ray casting approach to generate realistic LiDAR point clouds. The model is evaluated on KITTI-360 and Waymo Open datasets, showing superior performance in unconditional and map-conditioned LiDAR generation. LidarDM can also be used to augment real data for 3D perception models, improving their performance. The model is limited by its reliance on latent diffusion models, which are not real-time. Future work includes improving LiDAR intensity modeling. The project is supported by various grants and institutions.