22 Jul 2024 | Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, Yaron Lipman
Discrete Flow Matching is a novel approach for generating discrete data, designed to address the limitations of existing methods in handling high-dimensional discrete data like language. The method introduces a general framework for probability paths that interpolate between source and target distributions, enabling efficient sampling using learned posteriors. It allows for flexible scheduling of probability paths, leading to improved generative perplexity compared to previous discrete diffusion and flow models. The approach is capable of generating high-quality discrete data in a non-autoregressive manner, significantly closing the gap between autoregressive models and discrete flow models. The method was tested on various benchmarks, achieving 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1 and 20.6% Pass@10 on 1-shot MBPP coding benchmarks. The model also demonstrated strong performance in conditional text generation, with a perplexity score of 9.7, surpassing a 1.7B autoregressive model. The paper also discusses related work, including discrete flows and discrete diffusion models, and presents experimental results on language modeling, code generation, and image generation. The results show that Discrete Flow Matching outperforms previous methods in terms of generative perplexity and coding task performance, and achieves competitive results in image generation. The method is seen as a significant step in bridging the performance gap between discrete diffusion and autoregressive models, with further improvements possible through exploring the vast design space offered by Discrete Flow Matching.Discrete Flow Matching is a novel approach for generating discrete data, designed to address the limitations of existing methods in handling high-dimensional discrete data like language. The method introduces a general framework for probability paths that interpolate between source and target distributions, enabling efficient sampling using learned posteriors. It allows for flexible scheduling of probability paths, leading to improved generative perplexity compared to previous discrete diffusion and flow models. The approach is capable of generating high-quality discrete data in a non-autoregressive manner, significantly closing the gap between autoregressive models and discrete flow models. The method was tested on various benchmarks, achieving 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1 and 20.6% Pass@10 on 1-shot MBPP coding benchmarks. The model also demonstrated strong performance in conditional text generation, with a perplexity score of 9.7, surpassing a 1.7B autoregressive model. The paper also discusses related work, including discrete flows and discrete diffusion models, and presents experimental results on language modeling, code generation, and image generation. The results show that Discrete Flow Matching outperforms previous methods in terms of generative perplexity and coding task performance, and achieves competitive results in image generation. The method is seen as a significant step in bridging the performance gap between discrete diffusion and autoregressive models, with further improvements possible through exploring the vast design space offered by Discrete Flow Matching.