SampleAttention is an adaptive structured sparse attention mechanism designed to accelerate long context large language model (LLM) inference with near-lossless accuracy. The method leverages observed sparse patterns in attention scores to dynamically select key-value elements, reducing computational and I/O overhead. By focusing on local window and column stripe patterns, SampleAttention achieves significant improvements in Time-to-First-Token (TTFT) latency, with up to 2.42× speedup compared to FlashAttention. The approach is hardware-efficient and can be seamlessly integrated into existing LLMs without requiring additional training or fine-tuning. Theoretical analysis shows that near-lossless sparse attention can be achieved by maintaining a high sparsity degree while ensuring the cumulative residual attention (CRA) threshold is met. Empirical results demonstrate that attention scores exhibit inherent high sparsity, head-specific patterns, and content-aware variations. These patterns enable efficient sparse attention by dynamically capturing relevant key-value elements during inference. SampleAttention employs a two-stage query-guided key-value filtering approach. In the first stage, attention scores are sampled to identify critical elements. In the second stage, key-value indices are selected based on these scores to maintain near-lossless accuracy. The method is optimized for hardware efficiency, reducing memory and computation requirements while maintaining high performance. Experiments on ChatGLM2 and InternLM2 show that SampleAttention achieves nearly no accuracy loss and significantly outperforms existing methods in TTFT reduction. The approach is effective across various tasks and sequence lengths, demonstrating its versatility and efficiency in long context LLM inference. The method's adaptive sparsity and structured pattern selection enable efficient and accurate attention computation, making it a promising solution for accelerating long context LLMs.SampleAttention is an adaptive structured sparse attention mechanism designed to accelerate long context large language model (LLM) inference with near-lossless accuracy. The method leverages observed sparse patterns in attention scores to dynamically select key-value elements, reducing computational and I/O overhead. By focusing on local window and column stripe patterns, SampleAttention achieves significant improvements in Time-to-First-Token (TTFT) latency, with up to 2.42× speedup compared to FlashAttention. The approach is hardware-efficient and can be seamlessly integrated into existing LLMs without requiring additional training or fine-tuning. Theoretical analysis shows that near-lossless sparse attention can be achieved by maintaining a high sparsity degree while ensuring the cumulative residual attention (CRA) threshold is met. Empirical results demonstrate that attention scores exhibit inherent high sparsity, head-specific patterns, and content-aware variations. These patterns enable efficient sparse attention by dynamically capturing relevant key-value elements during inference. SampleAttention employs a two-stage query-guided key-value filtering approach. In the first stage, attention scores are sampled to identify critical elements. In the second stage, key-value indices are selected based on these scores to maintain near-lossless accuracy. The method is optimized for hardware efficiency, reducing memory and computation requirements while maintaining high performance. Experiments on ChatGLM2 and InternLM2 show that SampleAttention achieves nearly no accuracy loss and significantly outperforms existing methods in TTFT reduction. The approach is effective across various tasks and sequence lengths, demonstrating its versatility and efficiency in long context LLM inference. The method's adaptive sparsity and structured pattern selection enable efficient and accurate attention computation, making it a promising solution for accelerating long context LLMs.