This paper proposes a novel decoder-focused method for multi-task dense prediction called Mixture-of-Low-Rank-Experts (MLoRE). The method addresses the limitations of previous multi-task learning (MTL) approaches by explicitly modeling global task relationships and reducing computational costs. MLoRE introduces a generic convolution path to the Mixture-of-Experts (MoE) structure, allowing task features to share parameters for explicit parameter sharing. Additionally, it leverages the low-rank format of a vanilla convolution in the expert network, which reduces parameters and computational costs while maintaining performance. The low-rank experts can be dynamically parameterized into the generic convolution, enabling efficient scaling of the number of experts without significant increases in parameters or computational cost. This design allows for increased representation capacity, facilitating multiple dense tasks learning in a unified network. Extensive experiments on the PASCAL-Context and NYUD-v2 benchmarks show that MLoRE achieves superior performance compared to previous state-of-the-art methods on all metrics. The method also demonstrates efficiency by achieving competitive results with significantly fewer parameters and FLOPs. The proposed MLoRE module is shown to be effective in explicitly modeling global task relationships and improving task-specific feature discrimination. The results demonstrate that MLoRE outperforms previous methods in terms of performance and efficiency, making it a promising approach for multi-task dense prediction.This paper proposes a novel decoder-focused method for multi-task dense prediction called Mixture-of-Low-Rank-Experts (MLoRE). The method addresses the limitations of previous multi-task learning (MTL) approaches by explicitly modeling global task relationships and reducing computational costs. MLoRE introduces a generic convolution path to the Mixture-of-Experts (MoE) structure, allowing task features to share parameters for explicit parameter sharing. Additionally, it leverages the low-rank format of a vanilla convolution in the expert network, which reduces parameters and computational costs while maintaining performance. The low-rank experts can be dynamically parameterized into the generic convolution, enabling efficient scaling of the number of experts without significant increases in parameters or computational cost. This design allows for increased representation capacity, facilitating multiple dense tasks learning in a unified network. Extensive experiments on the PASCAL-Context and NYUD-v2 benchmarks show that MLoRE achieves superior performance compared to previous state-of-the-art methods on all metrics. The method also demonstrates efficiency by achieving competitive results with significantly fewer parameters and FLOPs. The proposed MLoRE module is shown to be effective in explicitly modeling global task relationships and improving task-specific feature discrimination. The results demonstrate that MLoRE outperforms previous methods in terms of performance and efficiency, making it a promising approach for multi-task dense prediction.