This paper addresses the issue of how large language models (LLMs) use their intermediate reasoning steps to generate final answers, finding that LLMs do not reliably use these steps. To improve this, the authors introduce FRODO, a framework that consists of an inference module and a reasoning module. The inference module learns to generate correct reasoning steps using an implicit causal reward function, while the reasoning module learns to faithfully reason over these steps using a counterfactual and causal preference objective. Experiments show that FRODO significantly outperforms four competitive baselines on four reasoning tasks, improves robustness and generalization, and generates more faithful rationales compared to standard supervised fine-tuning. The paper also performs a causal mediation analysis to measure the contribution of reasoning steps to the final answer, revealing that LLMs' outputs are influenced more by the reasoning problem than by the reasoning steps.This paper addresses the issue of how large language models (LLMs) use their intermediate reasoning steps to generate final answers, finding that LLMs do not reliably use these steps. To improve this, the authors introduce FRODO, a framework that consists of an inference module and a reasoning module. The inference module learns to generate correct reasoning steps using an implicit causal reward function, while the reasoning module learns to faithfully reason over these steps using a counterfactual and causal preference objective. Experiments show that FRODO significantly outperforms four competitive baselines on four reasoning tasks, improves robustness and generalization, and generates more faithful rationales compared to standard supervised fine-tuning. The paper also performs a causal mediation analysis to measure the contribution of reasoning steps to the final answer, revealing that LLMs' outputs are influenced more by the reasoning problem than by the reasoning steps.