Model editing at scale leads to gradual and catastrophic forgetting. This paper evaluates two state-of-the-art model editing methods, ROME and MEMIT, and finds that as models are edited sequentially with multiple facts, they gradually forget previously edited facts and lose the ability to perform downstream tasks. For ROME and MEMIT, this forgetting happens in two phases: an initial gradual phase followed by an abrupt or catastrophic phase. Both types of forgetting limit the usefulness of model editing methods at scale. The paper also highlights other key limitations of ROME and MEMIT at scale. The authors argue that model editing methods must be evaluated for scalability, and call for further research in developing scalable model editing methods. The paper also shows that ROME and MEMIT are prone to catastrophic forgetting, where a single update to the model can cause it to forget all previously edited facts and lose the ability to perform downstream tasks. The paper also finds that disabling edits, which are edits that render the model unusable, are not a result of continuous sequential editing but a fundamental limitation of ROME. The paper also evaluates the downstream performance of edited models and finds that as models are edited, their performance on downstream tasks declines. The paper concludes that model editing methods must be able to preserve the model's existing abilities while allowing for multiple edits. The paper calls for improved evaluation of model editing techniques at scale, including evaluating model performance on downstream tasks and ability to recall previously edited facts.Model editing at scale leads to gradual and catastrophic forgetting. This paper evaluates two state-of-the-art model editing methods, ROME and MEMIT, and finds that as models are edited sequentially with multiple facts, they gradually forget previously edited facts and lose the ability to perform downstream tasks. For ROME and MEMIT, this forgetting happens in two phases: an initial gradual phase followed by an abrupt or catastrophic phase. Both types of forgetting limit the usefulness of model editing methods at scale. The paper also highlights other key limitations of ROME and MEMIT at scale. The authors argue that model editing methods must be evaluated for scalability, and call for further research in developing scalable model editing methods. The paper also shows that ROME and MEMIT are prone to catastrophic forgetting, where a single update to the model can cause it to forget all previously edited facts and lose the ability to perform downstream tasks. The paper also finds that disabling edits, which are edits that render the model unusable, are not a result of continuous sequential editing but a fundamental limitation of ROME. The paper also evaluates the downstream performance of edited models and finds that as models are edited, their performance on downstream tasks declines. The paper concludes that model editing methods must be able to preserve the model's existing abilities while allowing for multiple edits. The paper calls for improved evaluation of model editing techniques at scale, including evaluating model performance on downstream tasks and ability to recall previously edited facts.