BitFit is a parameter-efficient fine-tuning method for transformer-based masked language models, where only the bias terms of the model are modified. The method shows that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, BitFit is competitive with other sparse fine-tuning methods. These findings support the hypothesis that fine-tuning mainly involves exposing knowledge from pre-training rather than learning new task-specific knowledge. BitFit freezes most of the transformer encoder parameters and trains only the bias terms and the task-specific classification layer. It achieves results comparable to full fine-tuning, with only a small portion of the model's parameters being modified. This makes it highly efficient for deployment in memory-constrained environments and enables trainable hardware implementations where most parameters are fixed. The method is parameter-efficient, requiring only the storage of bias term parameter vectors (less than 0.1% of total parameters) and a task-specific classification layer. BitFit achieves performance comparable to full fine-tuning by fine-tuning only the bias terms, and even better results when only a subset of bias parameters (query and middle-of-MLP biases) are fine-tuned. This approach is effective for various tasks, including GLUE benchmark tasks, and outperforms other methods like Diff-Pruning and Adapters in some cases. Experiments show that BitFit outperforms Diff-Pruning on 4 out of 9 tasks using 6x fewer parameters. It also performs well on different base models, including BERT_BASE and RoBERTA_BASE. The results indicate that bias parameters are crucial for performance, and fine-tuning only a subset of them can still achieve good results. BitFit also shows a smaller generalization gap compared to full fine-tuning, indicating better performance on unseen data. BitFit is a targeted fine-tuning method that focuses on modifying a small group of parameters, making it efficient for deployment and suitable for hardware implementations. It raises questions about the role of bias terms in pre-trained models and the dynamics of the fine-tuning process. The method is effective for various tasks and has practical implications for model compression and efficient deployment.BitFit is a parameter-efficient fine-tuning method for transformer-based masked language models, where only the bias terms of the model are modified. The method shows that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, BitFit is competitive with other sparse fine-tuning methods. These findings support the hypothesis that fine-tuning mainly involves exposing knowledge from pre-training rather than learning new task-specific knowledge. BitFit freezes most of the transformer encoder parameters and trains only the bias terms and the task-specific classification layer. It achieves results comparable to full fine-tuning, with only a small portion of the model's parameters being modified. This makes it highly efficient for deployment in memory-constrained environments and enables trainable hardware implementations where most parameters are fixed. The method is parameter-efficient, requiring only the storage of bias term parameter vectors (less than 0.1% of total parameters) and a task-specific classification layer. BitFit achieves performance comparable to full fine-tuning by fine-tuning only the bias terms, and even better results when only a subset of bias parameters (query and middle-of-MLP biases) are fine-tuned. This approach is effective for various tasks, including GLUE benchmark tasks, and outperforms other methods like Diff-Pruning and Adapters in some cases. Experiments show that BitFit outperforms Diff-Pruning on 4 out of 9 tasks using 6x fewer parameters. It also performs well on different base models, including BERT_BASE and RoBERTA_BASE. The results indicate that bias parameters are crucial for performance, and fine-tuning only a subset of them can still achieve good results. BitFit also shows a smaller generalization gap compared to full fine-tuning, indicating better performance on unseen data. BitFit is a targeted fine-tuning method that focuses on modifying a small group of parameters, making it efficient for deployment and suitable for hardware implementations. It raises questions about the role of bias terms in pre-trained models and the dynamics of the fine-tuning process. The method is effective for various tasks and has practical implications for model compression and efficient deployment.