This paper introduces Transfer Q*, a novel decoding strategy for aligning large language models (LLMs) with human preferences. The key challenge in alignment is accessing the optimal value function Q*, which is typically unavailable in practice. Transfer Q* addresses this by estimating Q* using a baseline model aligned with a baseline reward, which may differ from the target reward. This approach allows for efficient and principled decoding without requiring extensive model updates. Theoretical analysis of Transfer Q* provides a rigorous characterization of its optimality, deriving an upper bound on the sub-optimality gap and identifying a hyperparameter to control the deviation from the pre-trained reference SFT model. Empirical evaluations show that Transfer Q* significantly reduces the sub-optimality gap observed in prior state-of-the-art methods and demonstrates superior performance across key metrics such as coherence, diversity, and quality in extensive tests on synthetic and real datasets. The proposed method, Transfer Q*, is evaluated on both direct and indirect transfer scenarios. In direct transfer, it leverages a baseline model aligned with the target reward to estimate Q*. In indirect transfer, it uses a baseline model aligned with a different reward to estimate Q* through a novel indirect transfer decoding method. The method is shown to outperform existing approaches in terms of average reward, coherence, and diversity. Theoretical results show that Transfer Q* is bounded by the KL divergence to the reference policy and the sub-optimality gap. The method is also shown to be KL-efficient, meaning it achieves high rewards while staying close to the reference policy. The approach is effective in both synthetic and real-world scenarios, demonstrating its robustness and adaptability. Overall, Transfer Q* provides a principled solution to efficient decoding for AI alignment, significantly improving the performance of LLMs in terms of alignment with human preferences.This paper introduces Transfer Q*, a novel decoding strategy for aligning large language models (LLMs) with human preferences. The key challenge in alignment is accessing the optimal value function Q*, which is typically unavailable in practice. Transfer Q* addresses this by estimating Q* using a baseline model aligned with a baseline reward, which may differ from the target reward. This approach allows for efficient and principled decoding without requiring extensive model updates. Theoretical analysis of Transfer Q* provides a rigorous characterization of its optimality, deriving an upper bound on the sub-optimality gap and identifying a hyperparameter to control the deviation from the pre-trained reference SFT model. Empirical evaluations show that Transfer Q* significantly reduces the sub-optimality gap observed in prior state-of-the-art methods and demonstrates superior performance across key metrics such as coherence, diversity, and quality in extensive tests on synthetic and real datasets. The proposed method, Transfer Q*, is evaluated on both direct and indirect transfer scenarios. In direct transfer, it leverages a baseline model aligned with the target reward to estimate Q*. In indirect transfer, it uses a baseline model aligned with a different reward to estimate Q* through a novel indirect transfer decoding method. The method is shown to outperform existing approaches in terms of average reward, coherence, and diversity. Theoretical results show that Transfer Q* is bounded by the KL divergence to the reference policy and the sub-optimality gap. The method is also shown to be KL-efficient, meaning it achieves high rewards while staying close to the reference policy. The approach is effective in both synthetic and real-world scenarios, demonstrating its robustness and adaptability. Overall, Transfer Q* provides a principled solution to efficient decoding for AI alignment, significantly improving the performance of LLMs in terms of alignment with human preferences.