The paper introduces prefix-tuning, a lightweight alternative to fine-tuning for natural language generation (NLG) tasks. Unlike fine-tuning, which updates all parameters of a pre-trained language model (LM), prefix-tuning freezes the LM parameters and optimizes a small continuous task-specific vector, called the prefix. This approach is inspired by prompting, where subsequent tokens attend to the prefix as if it were virtual tokens. The prefix is learned during training, and subsequent tokens can attend to it, guiding the LM's output without modifying its parameters. The method is applied to GPT-2 for table-to-text generation and BART for summarization. Results show that prefix-tuning achieves comparable performance to fine-tuning in full data settings, outperforms fine-tuning in low-data settings, and extrapolates better to unseen topics. Prefix-tuning is modular, requiring only a small number of task-specific parameters, making it efficient and scalable for various tasks and models.The paper introduces prefix-tuning, a lightweight alternative to fine-tuning for natural language generation (NLG) tasks. Unlike fine-tuning, which updates all parameters of a pre-trained language model (LM), prefix-tuning freezes the LM parameters and optimizes a small continuous task-specific vector, called the prefix. This approach is inspired by prompting, where subsequent tokens attend to the prefix as if it were virtual tokens. The prefix is learned during training, and subsequent tokens can attend to it, guiding the LM's output without modifying its parameters. The method is applied to GPT-2 for table-to-text generation and BART for summarization. Results show that prefix-tuning achieves comparable performance to fine-tuning in full data settings, outperforms fine-tuning in low-data settings, and extrapolates better to unseen topics. Prefix-tuning is modular, requiring only a small number of task-specific parameters, making it efficient and scalable for various tasks and models.