LidarDM is a novel generative model designed to produce realistic, layout-aware, physically plausible, and temporally coherent LiDAR videos. It stands out with two unique capabilities: (1) LiDAR generation guided by driving scenarios, which is valuable for autonomous driving simulations, and (2) 4D LiDAR point cloud generation, enabling the creation of realistic and temporally coherent sequences. The core of the model is an integrated 4D world generation framework that uses latent diffusion models to generate 3D scenes, combines them with dynamic actors to form the underlying 4D world, and then produces realistic sensory observations within this virtual environment. Experiments show that LidarDM outperforms competing algorithms in realism, temporal coherency, and layout consistency. It can also be used as a generative world model simulator for training and testing perception models. The paper discusses related work, presents the problem formulation, scene, object, and trajectory generation, physics-informed LiDAR generation, and experimental results, including qualitative comparisons and downstream applications.LidarDM is a novel generative model designed to produce realistic, layout-aware, physically plausible, and temporally coherent LiDAR videos. It stands out with two unique capabilities: (1) LiDAR generation guided by driving scenarios, which is valuable for autonomous driving simulations, and (2) 4D LiDAR point cloud generation, enabling the creation of realistic and temporally coherent sequences. The core of the model is an integrated 4D world generation framework that uses latent diffusion models to generate 3D scenes, combines them with dynamic actors to form the underlying 4D world, and then produces realistic sensory observations within this virtual environment. Experiments show that LidarDM outperforms competing algorithms in realism, temporal coherency, and layout consistency. It can also be used as a generative world model simulator for training and testing perception models. The paper discusses related work, presents the problem formulation, scene, object, and trajectory generation, physics-informed LiDAR generation, and experimental results, including qualitative comparisons and downstream applications.