The paper introduces a diffusion model-based method, SegGuidedDiff, for anatomically-controllable medical image generation. The method leverages multi-class anatomical segmentation masks at each sampling step to enforce precise anatomical constraints in generated images. Additionally, a *random mask ablation* training algorithm is proposed to enable conditioning on selected anatomical constraints while allowing flexibility in other areas. The method is evaluated on breast MRI and abdominal/neck-to-pelvis CT datasets, demonstrating superior faithfulness to input anatomical masks and competitive performance in general anatomical realism. The model also allows for adjusting the anatomical similarity of generated images to real images through interpolation in its latent space. The code is publicly available, and the method has potential applications in cross-modality translation and generating paired or counterfactual data.The paper introduces a diffusion model-based method, SegGuidedDiff, for anatomically-controllable medical image generation. The method leverages multi-class anatomical segmentation masks at each sampling step to enforce precise anatomical constraints in generated images. Additionally, a *random mask ablation* training algorithm is proposed to enable conditioning on selected anatomical constraints while allowing flexibility in other areas. The method is evaluated on breast MRI and abdominal/neck-to-pelvis CT datasets, demonstrating superior faithfulness to input anatomical masks and competitive performance in general anatomical realism. The model also allows for adjusting the anatomical similarity of generated images to real images through interpolation in its latent space. The code is publicly available, and the method has potential applications in cross-modality translation and generating paired or counterfactual data.