This paper proposes a novel multi-task learning architecture called the Multi-Task Attention Network (MTAN), which enables the learning of task-specific feature-level attention. MTAN consists of a single shared network with a global feature pool and a soft-attention module for each task. These modules allow for the learning of task-specific features from the global features while enabling feature sharing across different tasks. The architecture is simple to implement, parameter-efficient, and can be trained end-to-end. It is evaluated on various datasets, including image-to-image prediction and image classification tasks, showing state-of-the-art performance in multi-task learning and being less sensitive to weighting schemes in the multi-task loss function. MTAN is designed to address two key challenges in multi-task learning: network architecture (how to share features) and loss function (how to balance tasks). The architecture allows for the automatic learning of both task-shared and task-specific features, and is robust to different loss weighting schemes. The proposed network is built on any feed-forward neural network and is tested on the CityScapes and NYUv2 datasets for semantic segmentation, depth estimation, and surface normal prediction. It is also tested on the Visual Decathlon Challenge for image classification tasks, where it outperforms several baselines and is competitive with the state-of-the-art. The MTAN architecture includes a shared network and task-specific attention modules that apply soft attention masks to learn task-specific features. The attention masks are automatically learned in an end-to-end manner, allowing for the sharing of features while maintaining task-specific performance. The method is robust to different weighting schemes in the loss function and introduces a novel weighting scheme called Dynamic Weight Average (DWA) that adapts task weighting over time based on the rate of change of the loss for each task. Experiments show that MTAN outperforms other multi-task learning methods in terms of performance and parameter efficiency. It is also more robust to the choice of weighting scheme in the loss function. The method is evaluated on various tasks, including image-to-image prediction and image classification, and shows significant improvements in performance compared to single-task and multi-task baselines. The results demonstrate that MTAN is a promising approach for multi-task learning, offering a balance between feature sharing and task-specific learning.This paper proposes a novel multi-task learning architecture called the Multi-Task Attention Network (MTAN), which enables the learning of task-specific feature-level attention. MTAN consists of a single shared network with a global feature pool and a soft-attention module for each task. These modules allow for the learning of task-specific features from the global features while enabling feature sharing across different tasks. The architecture is simple to implement, parameter-efficient, and can be trained end-to-end. It is evaluated on various datasets, including image-to-image prediction and image classification tasks, showing state-of-the-art performance in multi-task learning and being less sensitive to weighting schemes in the multi-task loss function. MTAN is designed to address two key challenges in multi-task learning: network architecture (how to share features) and loss function (how to balance tasks). The architecture allows for the automatic learning of both task-shared and task-specific features, and is robust to different loss weighting schemes. The proposed network is built on any feed-forward neural network and is tested on the CityScapes and NYUv2 datasets for semantic segmentation, depth estimation, and surface normal prediction. It is also tested on the Visual Decathlon Challenge for image classification tasks, where it outperforms several baselines and is competitive with the state-of-the-art. The MTAN architecture includes a shared network and task-specific attention modules that apply soft attention masks to learn task-specific features. The attention masks are automatically learned in an end-to-end manner, allowing for the sharing of features while maintaining task-specific performance. The method is robust to different weighting schemes in the loss function and introduces a novel weighting scheme called Dynamic Weight Average (DWA) that adapts task weighting over time based on the rate of change of the loss for each task. Experiments show that MTAN outperforms other multi-task learning methods in terms of performance and parameter efficiency. It is also more robust to the choice of weighting scheme in the loss function. The method is evaluated on various tasks, including image-to-image prediction and image classification, and shows significant improvements in performance compared to single-task and multi-task baselines. The results demonstrate that MTAN is a promising approach for multi-task learning, offering a balance between feature sharing and task-specific learning.