BitDelta is a method that compresses the weight delta of fine-tuned large language models (LLMs) to 1 bit without sacrificing performance. The method decomposes the fine-tuned model into a base model and a delta, then quantizes the delta using a binary matrix and a high-precision scaling factor. This allows multiple fine-tuned models to be represented with a single high-precision base model and multiple 1-bit deltas, significantly reducing GPU memory usage and improving multi-tenant serving efficiency. BitDelta achieves over 10× compression of the delta, with minimal performance degradation across various model sizes and fine-tuning techniques. The method is efficient, with a fast distillation process that calibrates the scaling factors to maintain model quality. BitDelta is validated on Llama2 and Mistral models, demonstrating its effectiveness in reducing storage and inference costs. The method also improves decoding latency by leveraging a fused binary GEMM kernel, enabling faster inference. BitDelta opens new opportunities for efficient model deployment and resource utilization in machine learning, while also highlighting the potential redundancy in fine-tuned information. However, it is a lossy compression method that may lead to the loss of crucial alignment information. The work contributes to environmental sustainability and cost reduction by lowering hardware requirements for serving fine-tuned models.BitDelta is a method that compresses the weight delta of fine-tuned large language models (LLMs) to 1 bit without sacrificing performance. The method decomposes the fine-tuned model into a base model and a delta, then quantizes the delta using a binary matrix and a high-precision scaling factor. This allows multiple fine-tuned models to be represented with a single high-precision base model and multiple 1-bit deltas, significantly reducing GPU memory usage and improving multi-tenant serving efficiency. BitDelta achieves over 10× compression of the delta, with minimal performance degradation across various model sizes and fine-tuning techniques. The method is efficient, with a fast distillation process that calibrates the scaling factors to maintain model quality. BitDelta is validated on Llama2 and Mistral models, demonstrating its effectiveness in reducing storage and inference costs. The method also improves decoding latency by leveraging a fused binary GEMM kernel, enabling faster inference. BitDelta opens new opportunities for efficient model deployment and resource utilization in machine learning, while also highlighting the potential redundancy in fine-tuned information. However, it is a lossy compression method that may lead to the loss of crucial alignment information. The work contributes to environmental sustainability and cost reduction by lowering hardware requirements for serving fine-tuned models.