FlashAttention-3 is a fast and accurate attention mechanism designed for large language models and long-context applications. It builds upon previous work, including FLASHATTENTION and FLASHATTENTION-2, to improve performance on Hopper GPUs. The key innovations include producer-consumer asynchrony, overlapping GEMM-softmax operations, and hardware-accelerated low-precision GEMM. These techniques allow FLASHATTENTION-3 to achieve up to 1.5-2.0× speedup on H100 GPUs with FP16, reaching 740 TFLOPs/s, and up to 1.2 PFLOPs/s with FP8. Additionally, FP8 FLASHATTENTION-3 achieves 2.6× lower numerical error than a baseline FP8 attention implementation. The method leverages hardware features such as asynchrony and low-precision (FP8) to enhance both efficiency and accuracy. It is open-sourced with a permissive license and aims to integrate with PyTorch and Hugging Face libraries to benefit researchers and developers. The paper validates the effectiveness of FLASHATTENTION-3 through empirical benchmarks and ablation studies, demonstrating its superiority over existing attention implementations in terms of speed and accuracy.FlashAttention-3 is a fast and accurate attention mechanism designed for large language models and long-context applications. It builds upon previous work, including FLASHATTENTION and FLASHATTENTION-2, to improve performance on Hopper GPUs. The key innovations include producer-consumer asynchrony, overlapping GEMM-softmax operations, and hardware-accelerated low-precision GEMM. These techniques allow FLASHATTENTION-3 to achieve up to 1.5-2.0× speedup on H100 GPUs with FP16, reaching 740 TFLOPs/s, and up to 1.2 PFLOPs/s with FP8. Additionally, FP8 FLASHATTENTION-3 achieves 2.6× lower numerical error than a baseline FP8 attention implementation. The method leverages hardware features such as asynchrony and low-precision (FP8) to enhance both efficiency and accuracy. It is open-sourced with a permissive license and aims to integrate with PyTorch and Hugging Face libraries to benefit researchers and developers. The paper validates the effectiveness of FLASHATTENTION-3 through empirical benchmarks and ablation studies, demonstrating its superiority over existing attention implementations in terms of speed and accuracy.