This paper introduces Vman, a visual-modified attention network (VMAN) designed to address challenges in multimodal vision-language tasks (VLT), such as visual question answering (VQA) and visual grounding (VG). Traditional transformers, while effective in unimodal tasks, struggle with high-dependency modeling and heterogeneous modality comprehension in multimodal tasks, leading to issues like noise introduction and insufficient information interaction. VMAN improves upon this by modifying the attention mechanism, introducing a visual-modified attention unit that establishes text-visual correspondence before image self-interaction. This unit refines query features, filtering noise and enhancing dependency modeling and reasoning. Two modified approaches are proposed: a weighted sum-based and a cross-attention-based method. Experiments on five benchmark datasets for VQA and VG show that VMAN achieves 70.99% accuracy on VQA-v2 and a 74.41% improvement on RefCOCOg. The code is available at https://github.com/79song/VMAN. The paper highlights the importance of multimodal VLT in areas like medical treatment, education, and daily life. Recent advances in transformers have enabled their application in multimodal pre-training tasks. However, pre-training is limited by the need for well-aligned data and high costs. End-to-end approaches are essential for improving transformer compatibility with downstream tasks. Current research has made progress in multimodal VLT, but there are still deficiencies in high-dependency modeling and heterogeneous modality comprehension. The complexity and redundancy of image features are not adequately addressed, leading to noise and insufficient cross-modal interaction. To overcome these issues, researchers have focused on designing one-stage VLT models. The paper proposes VMAN, a novel visual-modified attention network that replaces the core attention mechanism with a modified self-attention unit. This unit adjusts query features using advanced text features to obtain more refined query features, establishing a correspondence between visual and textual ends. VMAN is applied to VQA and VG tasks, with extensive experiments on four benchmark datasets confirming its generalizability and competitive performance. The main contributions include revealing the shortcomings of conventional transformers, improving the transformer architecture, proposing an end-to-end universal architecture, and applying VMAN to extensive experiments.This paper introduces Vman, a visual-modified attention network (VMAN) designed to address challenges in multimodal vision-language tasks (VLT), such as visual question answering (VQA) and visual grounding (VG). Traditional transformers, while effective in unimodal tasks, struggle with high-dependency modeling and heterogeneous modality comprehension in multimodal tasks, leading to issues like noise introduction and insufficient information interaction. VMAN improves upon this by modifying the attention mechanism, introducing a visual-modified attention unit that establishes text-visual correspondence before image self-interaction. This unit refines query features, filtering noise and enhancing dependency modeling and reasoning. Two modified approaches are proposed: a weighted sum-based and a cross-attention-based method. Experiments on five benchmark datasets for VQA and VG show that VMAN achieves 70.99% accuracy on VQA-v2 and a 74.41% improvement on RefCOCOg. The code is available at https://github.com/79song/VMAN. The paper highlights the importance of multimodal VLT in areas like medical treatment, education, and daily life. Recent advances in transformers have enabled their application in multimodal pre-training tasks. However, pre-training is limited by the need for well-aligned data and high costs. End-to-end approaches are essential for improving transformer compatibility with downstream tasks. Current research has made progress in multimodal VLT, but there are still deficiencies in high-dependency modeling and heterogeneous modality comprehension. The complexity and redundancy of image features are not adequately addressed, leading to noise and insufficient cross-modal interaction. To overcome these issues, researchers have focused on designing one-stage VLT models. The paper proposes VMAN, a novel visual-modified attention network that replaces the core attention mechanism with a modified self-attention unit. This unit adjusts query features using advanced text features to obtain more refined query features, establishing a correspondence between visual and textual ends. VMAN is applied to VQA and VG tasks, with extensive experiments on four benchmark datasets confirming its generalizability and competitive performance. The main contributions include revealing the shortcomings of conventional transformers, improving the transformer architecture, proposing an end-to-end universal architecture, and applying VMAN to extensive experiments.