Peptides, short chains of amino acids, play a crucial role in various biological processes by interacting with target molecules, making them promising candidates for drug discovery. This work introduces *PepFlow*, a novel multi-modal deep generative model based on the flow-matching framework for designing full-atom peptides that target specific protein receptors. *PepFlow* characterizes peptide structures using rigid backbone frames in the SE(3) manifold and side-chain angles on high-dimensional tori, while representing discrete residue types as categorical distributions on the probability simplex. By learning the joint distributions of these modalities, *PepFlow* excels in fine-grained peptide design, including fix-backbone sequence design and side-chain packing through partial sampling. Comprehensive experiments demonstrate that *PepFlow* outperforms existing methods in various benchmarks, highlighting its significant potential in computational peptide design and analysis. The key contributions include the introduction of *PepFlow*, the resolution of challenges in target-specific peptide design, and the establishment of comprehensive benchmarks with a new dataset and in silico metrics.Peptides, short chains of amino acids, play a crucial role in various biological processes by interacting with target molecules, making them promising candidates for drug discovery. This work introduces *PepFlow*, a novel multi-modal deep generative model based on the flow-matching framework for designing full-atom peptides that target specific protein receptors. *PepFlow* characterizes peptide structures using rigid backbone frames in the SE(3) manifold and side-chain angles on high-dimensional tori, while representing discrete residue types as categorical distributions on the probability simplex. By learning the joint distributions of these modalities, *PepFlow* excels in fine-grained peptide design, including fix-backbone sequence design and side-chain packing through partial sampling. Comprehensive experiments demonstrate that *PepFlow* outperforms existing methods in various benchmarks, highlighting its significant potential in computational peptide design and analysis. The key contributions include the introduction of *PepFlow*, the resolution of challenges in target-specific peptide design, and the establishment of comprehensive benchmarks with a new dataset and in silico metrics.