This paper introduces DP-ZO, a differentially private fine-tuning framework for large language models (LLMs) that uses zeroth-order optimization (ZO) with added noise to privatize the scalar step size. The key insight is that in ZO, the only information from training data is a scalar step size, which can be efficiently privatized. DP-ZO provides a strong privacy-utility trade-off comparable to DP-SGD in terms of (ε, δ)-DP, but with significant memory efficiency gains and higher utility in ε-DP when using the Laplace mechanism. DP-ZO is designed to be memory-efficient and easy to implement, making it suitable for large models. It achieves comparable performance to DP-SGD for the same model size and scales seamlessly to large models like 30B/66B. DP-ZO also provides non-trivial performance under pure ε-DP, achieving 73.52% on SQuAD at ε=4. The method is based on the SPSA algorithm, where the gradient is estimated using the difference in losses between two random perturbations. DP-ZO adds noise to this difference to ensure privacy. The scalar step size is clipped to limit sensitivity and then privatized using the Laplace or Gaussian mechanisms. The paper evaluates DP-ZO across multiple datasets and model sizes, showing that it achieves strong privacy-utility trade-offs under conservative privacy budgets. It also demonstrates that DP-ZO is more efficient in terms of memory and computation compared to DP-SGD, which requires additional engineering effort and more memory for gradient clipping. The results show that DP-ZO outperforms DP-SGD in terms of utility for large models and provides a strong privacy-utility trade-off across different tasks and model sizes. The method is flexible and can be extended to other differential privacy mechanisms, broadening its applicability. The paper also provides empirical privacy analysis, showing that DP-ZO significantly reduces privacy attacks compared to ZO.This paper introduces DP-ZO, a differentially private fine-tuning framework for large language models (LLMs) that uses zeroth-order optimization (ZO) with added noise to privatize the scalar step size. The key insight is that in ZO, the only information from training data is a scalar step size, which can be efficiently privatized. DP-ZO provides a strong privacy-utility trade-off comparable to DP-SGD in terms of (ε, δ)-DP, but with significant memory efficiency gains and higher utility in ε-DP when using the Laplace mechanism. DP-ZO is designed to be memory-efficient and easy to implement, making it suitable for large models. It achieves comparable performance to DP-SGD for the same model size and scales seamlessly to large models like 30B/66B. DP-ZO also provides non-trivial performance under pure ε-DP, achieving 73.52% on SQuAD at ε=4. The method is based on the SPSA algorithm, where the gradient is estimated using the difference in losses between two random perturbations. DP-ZO adds noise to this difference to ensure privacy. The scalar step size is clipped to limit sensitivity and then privatized using the Laplace or Gaussian mechanisms. The paper evaluates DP-ZO across multiple datasets and model sizes, showing that it achieves strong privacy-utility trade-offs under conservative privacy budgets. It also demonstrates that DP-ZO is more efficient in terms of memory and computation compared to DP-SGD, which requires additional engineering effort and more memory for gradient clipping. The results show that DP-ZO outperforms DP-SGD in terms of utility for large models and provides a strong privacy-utility trade-off across different tasks and model sizes. The method is flexible and can be extended to other differential privacy mechanisms, broadening its applicability. The paper also provides empirical privacy analysis, showing that DP-ZO significantly reduces privacy attacks compared to ZO.