This paper introduces a novel parameter-efficient fine-tuning method called Representation Editing (RED), which directly edits the representations generated by neural network layers rather than adjusting the model's weights. RED significantly reduces the number of trainable parameters compared to full parameter fine-tuning and other PEFT methods like LoRA. For example, RED reduces the number of trainable parameters by a factor of 25,700 compared to full parameter fine-tuning and by a factor of 32 compared to LoRA. RED achieves results comparable or superior to full parameter fine-tuning and other PEFT methods across various tasks, including natural language understanding and generation. RED modifies the representations generated at certain layers through scaling and biasing operations. This approach allows for efficient and effective adaptation of large models to specific downstream tasks with minimal parameter adjustment. The method is implemented by introducing two learnable vectors, one for scaling and one for bias, which are applied to the representations generated by feed-forward sub-layers. This approach requires significantly fewer parameters than traditional PEFT methods, making it both storage and computation efficient. Extensive experiments across various model architectures and scales, including RoBERTa, GPT-2, T5, and LLaMA-2, demonstrate the effectiveness and efficiency of RED. RED achieves competitive performance with minimal parameter adjustment, showing that it can maintain the generalization capability acquired during pre-training and deliver enhanced performance with relatively small training data. The method is also parameter-efficient, with RED requiring approximately 7,200 times fewer trainable parameters than full parameter fine-tuning and 16 times fewer than LoRA. The study also includes an ablation study to examine the impact of different editing operations and positions. The results show that both scaling and biasing operations are crucial for enhancing model performance. Additionally, the study compares RED with other PEFT methods, including Adapter, LoRA, and Prompt Tuning, demonstrating that RED outperforms these methods in terms of parameter efficiency and performance. The paper concludes that RED is a promising PEFT strategy for large-scale neural models, offering a balance between parameter efficiency and performance. The method is particularly effective for large language models, demonstrating that direct editing of hidden representations during fine-tuning can lead to high-quality outputs. The study also highlights the potential for further research in extending RED to other modalities and applications.This paper introduces a novel parameter-efficient fine-tuning method called Representation Editing (RED), which directly edits the representations generated by neural network layers rather than adjusting the model's weights. RED significantly reduces the number of trainable parameters compared to full parameter fine-tuning and other PEFT methods like LoRA. For example, RED reduces the number of trainable parameters by a factor of 25,700 compared to full parameter fine-tuning and by a factor of 32 compared to LoRA. RED achieves results comparable or superior to full parameter fine-tuning and other PEFT methods across various tasks, including natural language understanding and generation. RED modifies the representations generated at certain layers through scaling and biasing operations. This approach allows for efficient and effective adaptation of large models to specific downstream tasks with minimal parameter adjustment. The method is implemented by introducing two learnable vectors, one for scaling and one for bias, which are applied to the representations generated by feed-forward sub-layers. This approach requires significantly fewer parameters than traditional PEFT methods, making it both storage and computation efficient. Extensive experiments across various model architectures and scales, including RoBERTa, GPT-2, T5, and LLaMA-2, demonstrate the effectiveness and efficiency of RED. RED achieves competitive performance with minimal parameter adjustment, showing that it can maintain the generalization capability acquired during pre-training and deliver enhanced performance with relatively small training data. The method is also parameter-efficient, with RED requiring approximately 7,200 times fewer trainable parameters than full parameter fine-tuning and 16 times fewer than LoRA. The study also includes an ablation study to examine the impact of different editing operations and positions. The results show that both scaling and biasing operations are crucial for enhancing model performance. Additionally, the study compares RED with other PEFT methods, including Adapter, LoRA, and Prompt Tuning, demonstrating that RED outperforms these methods in terms of parameter efficiency and performance. The paper concludes that RED is a promising PEFT strategy for large-scale neural models, offering a balance between parameter efficiency and performance. The method is particularly effective for large language models, demonstrating that direct editing of hidden representations during fine-tuning can lead to high-quality outputs. The study also highlights the potential for further research in extending RED to other modalities and applications.