Taskonomy: Disentangling Task Transfer Learning This paper proposes a computational approach to model the structure of the space of visual tasks. By analyzing transfer learning dependencies across a dictionary of 26 tasks (2D, 2.5D, 3D, and semantic), we create a computational taxonomic map for task transfer learning. This map reveals non-trivial relationships among tasks, enabling the reduction of labeled data needed for solving multiple tasks. For example, solving 10 tasks requires about 2/3 fewer labeled data points compared to training them independently, while maintaining performance. The approach involves training task-specific networks, then using these to compute transfer functions between tasks in a latent space. These functions are used to determine the optimal transfer policy, which minimizes supervision while maximizing performance. The method uses a Binary Integer Programming formulation to find the globally efficient transfer policy. The task taxonomy is a directed hypergraph that captures task transferability. It includes edges representing feasible transfers between source and target tasks, with weights indicating performance. The taxonomy is built through four steps: training task-specific networks, training transfer functions, normalizing task affinities, and synthesizing a hypergraph that predicts transfer performance. The task dictionary includes 26 tasks covering common themes in computer vision. The dataset used contains 4 million images with annotations for every task. Task-specific networks are trained on these images, and transfer functions are trained to map representations from source tasks to target tasks. The method is evaluated on various tasks, showing that the taxonomy can significantly reduce the need for labeled data. It is also shown to generalize well to novel tasks, outperforming existing methods in many cases. The taxonomy is robust to changes in system choices, indicating a strong and predictable structure in the task space. The paper also discusses the limitations of the approach, including model dependence and the need for further study on the properties of the computed space. The results suggest that the task space has a strong, predictable structure that can be leveraged for efficient transfer learning.Taskonomy: Disentangling Task Transfer Learning This paper proposes a computational approach to model the structure of the space of visual tasks. By analyzing transfer learning dependencies across a dictionary of 26 tasks (2D, 2.5D, 3D, and semantic), we create a computational taxonomic map for task transfer learning. This map reveals non-trivial relationships among tasks, enabling the reduction of labeled data needed for solving multiple tasks. For example, solving 10 tasks requires about 2/3 fewer labeled data points compared to training them independently, while maintaining performance. The approach involves training task-specific networks, then using these to compute transfer functions between tasks in a latent space. These functions are used to determine the optimal transfer policy, which minimizes supervision while maximizing performance. The method uses a Binary Integer Programming formulation to find the globally efficient transfer policy. The task taxonomy is a directed hypergraph that captures task transferability. It includes edges representing feasible transfers between source and target tasks, with weights indicating performance. The taxonomy is built through four steps: training task-specific networks, training transfer functions, normalizing task affinities, and synthesizing a hypergraph that predicts transfer performance. The task dictionary includes 26 tasks covering common themes in computer vision. The dataset used contains 4 million images with annotations for every task. Task-specific networks are trained on these images, and transfer functions are trained to map representations from source tasks to target tasks. The method is evaluated on various tasks, showing that the taxonomy can significantly reduce the need for labeled data. It is also shown to generalize well to novel tasks, outperforming existing methods in many cases. The taxonomy is robust to changes in system choices, indicating a strong and predictable structure in the task space. The paper also discusses the limitations of the approach, including model dependence and the need for further study on the properties of the computed space. The results suggest that the task space has a strong, predictable structure that can be leveraged for efficient transfer learning.