SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning This paper investigates the role of optimizer choice in large language model (LLM) unlearning, establishing a clear connection between second-order optimization and influence unlearning. We propose a second-order optimization-based LLM unlearning framework, termed SOUL, which extends the static, one-shot model update using influence unlearning to a dynamic, iterative unlearning process. Our experiments show that SOUL consistently outperforms conventional first-order methods across various unlearning tasks, models, and metrics, indicating that second-order optimization offers an effective and broadly applicable solution for LLM unlearning. LLM unlearning aims to remove undesired data influences and associated model capabilities without compromising utility beyond the scope of unlearning. While the concept is appealing, the development of effective unlearning algorithms remains challenging. A straightforward approach involves retraining the model from scratch after removing the undesired training data, driven by data privacy concerns. However, this method is impractical due to the extremely high cost associated with retraining LLMs from scratch. Therefore, model fine-tuning under a predefined unlearning objective has become the primary approach to solve most LLM unlearning problems. Unfortunately, there is a lack of effective fine-tuning techniques for LLM unlearning. Several recent efforts have been made to develop improved model fine-tuning techniques for LLM unlearning. For example, studies have delved into designing fine-tuning loss functions tailored for LLM unlearning. A currently popular choice is the regularized optimization objective that integrates unlearning efficacy loss with model utility loss. Additionally, other LLM unlearning techniques incorporate the model's prior into fine-tuning. For instance, fine-tuning is selectively applied to a subset of model units deemed essential for the unlearning task. This approach has led to the emergence of localization-informed LLM unlearning. Despite the recent progress of LLM unlearning, the majority of existing fine-tuning-based approaches have relied on first-order (FO) optimization to conduct unlearning. To our knowledge, there have been no prior studies that specifically investigate LLM unlearning from the perspective of optimizer design. In this work, we unveil the power of second-order (SO) optimizer in LLM unlearning and demonstrate its superiority over FO optimizer in various fine-tuning scenarios. We term the second-order optimization-based unlearning framework as SOUL. We will show that SOUL not only offers a viable approach for enhancing unlearning efficacy but also stays effective in preserving model utility. Such an optimizer-induced advantage holds consistently across various LLM unlearning objectives and formulations, providing a generic improvement. Our experiments show that SOUL consistently outperforms conventional first-order methods across various unlearning tasks, models, and metrics. The results indicate that second-order optimization offers an effective and broadly applicable solution for LLM unlearning. The SOUL framework is implemented using the Sophia optimizer, which is a simpleSOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning This paper investigates the role of optimizer choice in large language model (LLM) unlearning, establishing a clear connection between second-order optimization and influence unlearning. We propose a second-order optimization-based LLM unlearning framework, termed SOUL, which extends the static, one-shot model update using influence unlearning to a dynamic, iterative unlearning process. Our experiments show that SOUL consistently outperforms conventional first-order methods across various unlearning tasks, models, and metrics, indicating that second-order optimization offers an effective and broadly applicable solution for LLM unlearning. LLM unlearning aims to remove undesired data influences and associated model capabilities without compromising utility beyond the scope of unlearning. While the concept is appealing, the development of effective unlearning algorithms remains challenging. A straightforward approach involves retraining the model from scratch after removing the undesired training data, driven by data privacy concerns. However, this method is impractical due to the extremely high cost associated with retraining LLMs from scratch. Therefore, model fine-tuning under a predefined unlearning objective has become the primary approach to solve most LLM unlearning problems. Unfortunately, there is a lack of effective fine-tuning techniques for LLM unlearning. Several recent efforts have been made to develop improved model fine-tuning techniques for LLM unlearning. For example, studies have delved into designing fine-tuning loss functions tailored for LLM unlearning. A currently popular choice is the regularized optimization objective that integrates unlearning efficacy loss with model utility loss. Additionally, other LLM unlearning techniques incorporate the model's prior into fine-tuning. For instance, fine-tuning is selectively applied to a subset of model units deemed essential for the unlearning task. This approach has led to the emergence of localization-informed LLM unlearning. Despite the recent progress of LLM unlearning, the majority of existing fine-tuning-based approaches have relied on first-order (FO) optimization to conduct unlearning. To our knowledge, there have been no prior studies that specifically investigate LLM unlearning from the perspective of optimizer design. In this work, we unveil the power of second-order (SO) optimizer in LLM unlearning and demonstrate its superiority over FO optimizer in various fine-tuning scenarios. We term the second-order optimization-based unlearning framework as SOUL. We will show that SOUL not only offers a viable approach for enhancing unlearning efficacy but also stays effective in preserving model utility. Such an optimizer-induced advantage holds consistently across various LLM unlearning objectives and formulations, providing a generic improvement. Our experiments show that SOUL consistently outperforms conventional first-order methods across various unlearning tasks, models, and metrics. The results indicate that second-order optimization offers an effective and broadly applicable solution for LLM unlearning. The SOUL framework is implemented using the Sophia optimizer, which is a simple