DeepEdit: Knowledge Editing as Decoding with Constraints This paper introduces DeepEdit, a new framework for knowledge editing (KE) in large language models (LLMs). KE aims to enable LLMs to incorporate new or updated knowledge, unlike relying solely on outdated parametric knowledge. Multi-hop question answering is a challenging task to evaluate KE, requiring models to answer questions based on a chain of facts including new knowledge. DeepEdit uses decoding constraints to regulate LLMs' reasoning, enhancing logical coherence when incorporating new knowledge. It proposes a new KE framework called DEEPEDIT, which enhances LLMs' ability to generate coherent reasoning chains with new knowledge through depth-first search. The search selects the most important knowledge that satisfies constraints as the reasoning step to efficiently increase the reasoning depth. In addition to DEEPEDIT, the paper proposes two new KE benchmarks: MQUAKE-2002 and MQUAKE-HARD, which provide more precise and challenging assessments of KE approaches. Qualitatively, DEEPEDIT enables LLMs to produce succinct and coherent reasoning chains involving new knowledge. Quantitatively, it yields significant improvements on multiple KE benchmarks. The paper also explores the use of decoding constraints and search algorithms to explicitly control LLMs' reasoning with new knowledge. The central idea is to view KE as a problem of constrained decoding. The paper proposes four constraints: CONCISENESS, COHERENCE, RECEPTIVENESS, and PERTINENCE. These constraints ensure that the reasoning steps are logically coherent, new knowledge is properly integrated, and the reasoning is relevant to the target question. DEEPEDIT is a new KE method that effectively augments LLMs on KE with constrained decoding. It is flexible and applicable to any black-box LLM without requiring access to model parameters or token-wise distributions. It improves both the receptiveness of new knowledge and the coherence of multi-step reasoning by directly controlling the LLMs' reasoning with decoding constraints. The paper also provides two new benchmarks for more precise and challenging evaluation of KE methods: MQUAKE-2002 and MQUAKE-HARD. These benchmarks resolve knowledge-conflicting annotation mistakes in the popular KE benchmark MQUAKE-3K. The paper evaluates DEEPEDIT on MQUAKE-3K, MQUAKE-2002, and MQUAKE-HARD. Qualitatively, DEEPEDIT enhances LLMs to produce succinct reasoning with new knowledge. Quantitatively, DEEPEDIT improves the KE for popular LLMs by a significant margin. The paper also discusses related work, including previous studies on knowledge editing and constrained decoding. It compares DEEPEDIT with existing KE methods and shows that it outperforms them in terms of performance and efficiency. The paper also discusses the efficiency of depth-first search in DEEPEDIT, showing that it significantly reduces the number of reasoning steps and generation time compared to breadth-first search. The paper also presentsDeepEdit: Knowledge Editing as Decoding with Constraints This paper introduces DeepEdit, a new framework for knowledge editing (KE) in large language models (LLMs). KE aims to enable LLMs to incorporate new or updated knowledge, unlike relying solely on outdated parametric knowledge. Multi-hop question answering is a challenging task to evaluate KE, requiring models to answer questions based on a chain of facts including new knowledge. DeepEdit uses decoding constraints to regulate LLMs' reasoning, enhancing logical coherence when incorporating new knowledge. It proposes a new KE framework called DEEPEDIT, which enhances LLMs' ability to generate coherent reasoning chains with new knowledge through depth-first search. The search selects the most important knowledge that satisfies constraints as the reasoning step to efficiently increase the reasoning depth. In addition to DEEPEDIT, the paper proposes two new KE benchmarks: MQUAKE-2002 and MQUAKE-HARD, which provide more precise and challenging assessments of KE approaches. Qualitatively, DEEPEDIT enables LLMs to produce succinct and coherent reasoning chains involving new knowledge. Quantitatively, it yields significant improvements on multiple KE benchmarks. The paper also explores the use of decoding constraints and search algorithms to explicitly control LLMs' reasoning with new knowledge. The central idea is to view KE as a problem of constrained decoding. The paper proposes four constraints: CONCISENESS, COHERENCE, RECEPTIVENESS, and PERTINENCE. These constraints ensure that the reasoning steps are logically coherent, new knowledge is properly integrated, and the reasoning is relevant to the target question. DEEPEDIT is a new KE method that effectively augments LLMs on KE with constrained decoding. It is flexible and applicable to any black-box LLM without requiring access to model parameters or token-wise distributions. It improves both the receptiveness of new knowledge and the coherence of multi-step reasoning by directly controlling the LLMs' reasoning with decoding constraints. The paper also provides two new benchmarks for more precise and challenging evaluation of KE methods: MQUAKE-2002 and MQUAKE-HARD. These benchmarks resolve knowledge-conflicting annotation mistakes in the popular KE benchmark MQUAKE-3K. The paper evaluates DEEPEDIT on MQUAKE-3K, MQUAKE-2002, and MQUAKE-HARD. Qualitatively, DEEPEDIT enhances LLMs to produce succinct reasoning with new knowledge. Quantitatively, DEEPEDIT improves the KE for popular LLMs by a significant margin. The paper also discusses related work, including previous studies on knowledge editing and constrained decoding. It compares DEEPEDIT with existing KE methods and shows that it outperforms them in terms of performance and efficiency. The paper also discusses the efficiency of depth-first search in DEEPEDIT, showing that it significantly reduces the number of reasoning steps and generation time compared to breadth-first search. The paper also presents