This paper introduces a segmentation-guided diffusion model ("SegGuidedDiff") for anatomically-controllable medical image generation. The model uses multi-class anatomical segmentation masks to guide the diffusion process, ensuring generated images align with input anatomical constraints. A mask-ablated training strategy is introduced to allow the model to infer missing anatomical classes, improving flexibility. The model is evaluated on breast MRI and CT datasets, showing superior faithfulness to input masks and comparable anatomical realism to existing methods. SegGuidedDiff also enables adjustable anatomical similarity between generated images and real images through latent space interpolation. The model has applications in cross-modality translation and generating paired or counterfactual data. The code is available at https://github.com/mazurowski-lab/segmentation-guided-diffusion.This paper introduces a segmentation-guided diffusion model ("SegGuidedDiff") for anatomically-controllable medical image generation. The model uses multi-class anatomical segmentation masks to guide the diffusion process, ensuring generated images align with input anatomical constraints. A mask-ablated training strategy is introduced to allow the model to infer missing anatomical classes, improving flexibility. The model is evaluated on breast MRI and CT datasets, showing superior faithfulness to input masks and comparable anatomical realism to existing methods. SegGuidedDiff also enables adjustable anatomical similarity between generated images and real images through latent space interpolation. The model has applications in cross-modality translation and generating paired or counterfactual data. The code is available at https://github.com/mazurowski-lab/segmentation-guided-diffusion.