The paper introduces a new method called Chain-of-Abstraction (CoA) to enhance large language models (LLMs) in multi-step reasoning tasks. CoA trains LLMs to generate abstract reasoning chains with placeholders, which are then filled with domain-specific knowledge from external tools. This approach enables LLMs to learn more general and robust reasoning strategies, improving their performance in both in-distribution and out-of-distribution test sets. The method also enhances inference efficiency by allowing parallel processing of tool calls and decoding, resulting in faster inference speeds. The authors evaluate CoA on mathematical reasoning and Wikipedia (Wiki) QA domains, demonstrating significant improvements over previous methods in accuracy and efficiency. Human evaluations further validate the effectiveness of CoA in reducing arithmetic and reasoning errors. The paper discusses limitations and future directions, including the potential for integrating self-consistency decoding and adapting CoA to new model backbones and tasks.The paper introduces a new method called Chain-of-Abstraction (CoA) to enhance large language models (LLMs) in multi-step reasoning tasks. CoA trains LLMs to generate abstract reasoning chains with placeholders, which are then filled with domain-specific knowledge from external tools. This approach enables LLMs to learn more general and robust reasoning strategies, improving their performance in both in-distribution and out-of-distribution test sets. The method also enhances inference efficiency by allowing parallel processing of tool calls and decoding, resulting in faster inference speeds. The authors evaluate CoA on mathematical reasoning and Wikipedia (Wiki) QA domains, demonstrating significant improvements over previous methods in accuracy and efficiency. Human evaluations further validate the effectiveness of CoA in reducing arithmetic and reasoning errors. The paper discusses limitations and future directions, including the potential for integrating self-consistency decoding and adapting CoA to new model backbones and tasks.