MUSE is a comprehensive benchmark for evaluating machine unlearning algorithms on language models. The paper introduces six key properties that unlearning algorithms should satisfy: no verbatim memorization, no knowledge memorization, no privacy leakage, utility preservation on data not intended for removal, scalability with respect to the size of removal requests, and sustainability over sequential unlearning requests. The authors evaluate eight popular unlearning algorithms on 7B-parameter language models using two datasets: news articles and books. The results show that most algorithms can prevent verbatim and knowledge memorization to some extent, but only one algorithm avoids severe privacy leakage. Existing algorithms fail to meet deployer expectations as they often degrade model utility and cannot sustain multiple unlearning requests. The authors also highlight that current unlearning methods are not yet ready for practical deployment due to their limitations in utility preservation, privacy leakage, and scalability. The MUSE benchmark is released to facilitate further evaluations and research in this area.MUSE is a comprehensive benchmark for evaluating machine unlearning algorithms on language models. The paper introduces six key properties that unlearning algorithms should satisfy: no verbatim memorization, no knowledge memorization, no privacy leakage, utility preservation on data not intended for removal, scalability with respect to the size of removal requests, and sustainability over sequential unlearning requests. The authors evaluate eight popular unlearning algorithms on 7B-parameter language models using two datasets: news articles and books. The results show that most algorithms can prevent verbatim and knowledge memorization to some extent, but only one algorithm avoids severe privacy leakage. Existing algorithms fail to meet deployer expectations as they often degrade model utility and cannot sustain multiple unlearning requests. The authors also highlight that current unlearning methods are not yet ready for practical deployment due to their limitations in utility preservation, privacy leakage, and scalability. The MUSE benchmark is released to facilitate further evaluations and research in this area.