This paper presents a unified framework for parameter-efficient transfer learning methods, aiming to understand the connections between various approaches and identify key design choices. The authors analyze several state-of-the-art methods, including adapters, prefix tuning, and LoRA, and propose a unified view that frames them as modifications to hidden states in pre-trained language models. They define design dimensions such as the function used to perform the modification and the position where the modification is applied. Through empirical studies on tasks like machine translation, text summarization, and text classification, they demonstrate that their unified framework enables the transfer of design elements across methods, leading to more effective parameter-efficient fine-tuning approaches. The proposed methods achieve performance comparable to full fine-tuning while using fewer parameters. The paper also discusses the importance of design choices, such as the type of modification and the insertion form, and highlights the effectiveness of multi-head attention and FFN modifications. The results show that modifying FFN can be more effective than modifying attention, especially with larger parameter budgets. The authors conclude that their unified framework provides insights into parameter-efficient transfer learning and enables the development of more effective methods by transferring favorable design elements across approaches.This paper presents a unified framework for parameter-efficient transfer learning methods, aiming to understand the connections between various approaches and identify key design choices. The authors analyze several state-of-the-art methods, including adapters, prefix tuning, and LoRA, and propose a unified view that frames them as modifications to hidden states in pre-trained language models. They define design dimensions such as the function used to perform the modification and the position where the modification is applied. Through empirical studies on tasks like machine translation, text summarization, and text classification, they demonstrate that their unified framework enables the transfer of design elements across methods, leading to more effective parameter-efficient fine-tuning approaches. The proposed methods achieve performance comparable to full fine-tuning while using fewer parameters. The paper also discusses the importance of design choices, such as the type of modification and the insertion form, and highlights the effectiveness of multi-head attention and FFN modifications. The results show that modifying FFN can be more effective than modifying attention, especially with larger parameter budgets. The authors conclude that their unified framework provides insights into parameter-efficient transfer learning and enables the development of more effective methods by transferring favorable design elements across approaches.