This paper provides a comprehensive survey of Parameter-Efficient Fine-Tuning (PEFT) for large models, focusing on efficient adjustments to pre-trained models for specific downstream tasks. The authors categorize PEFT algorithms into four main types: additive, selective, reparameterized, and hybrid. Additive PEFT introduces new trainable parameters or modifies activations, while selective PEFT tunes only a subset of existing parameters. Reparameterized PEFT constructs a low-dimensional reparameterization of original model parameters, and hybrid PEFT combines elements from different methods. The survey examines the performance and computational overhead of various PEFT algorithms, discusses common techniques to reduce computational costs, and explores real-world applications and system designs. It also highlights the versatility of PEFT in different model architectures and downstream tasks, such as language models, vision transformers, and diffusion models. The paper concludes with a discussion on system design challenges and future research directions, emphasizing the importance of both algorithmic and systemic advancements in PEFT.This paper provides a comprehensive survey of Parameter-Efficient Fine-Tuning (PEFT) for large models, focusing on efficient adjustments to pre-trained models for specific downstream tasks. The authors categorize PEFT algorithms into four main types: additive, selective, reparameterized, and hybrid. Additive PEFT introduces new trainable parameters or modifies activations, while selective PEFT tunes only a subset of existing parameters. Reparameterized PEFT constructs a low-dimensional reparameterization of original model parameters, and hybrid PEFT combines elements from different methods. The survey examines the performance and computational overhead of various PEFT algorithms, discusses common techniques to reduce computational costs, and explores real-world applications and system designs. It also highlights the versatility of PEFT in different model architectures and downstream tasks, such as language models, vision transformers, and diffusion models. The paper concludes with a discussion on system design challenges and future research directions, emphasizing the importance of both algorithmic and systemic advancements in PEFT.