Transfer learning using pre-trained models like ImageNet has become a standard in medical imaging. However, this paper challenges the assumption that transfer learning significantly improves performance in medical tasks. The study evaluates two large-scale medical imaging tasks—retinal fundus images and chest x-rays—and finds that transfer learning offers little benefit. Small, lightweight models perform comparably to standard ImageNet models, and ImageNet performance is not predictive of medical performance. The paper investigates the learned representations and finds that the differences in performance are due to over-parametrization of standard models rather than feature reuse. It isolates where useful feature reuse occurs and outlines implications for more efficient model exploration. It also explores feature-independent benefits of transfer learning, such as improved convergence speed from weight scaling. The study uses the RETINA dataset for retinal fundus images and the CHEXpert dataset for chest x-rays. The results show that transfer learning does not significantly improve performance on these tasks. The paper also finds that smaller models perform comparably to larger models, and that the effects of transfer learning are confounded by model size. The paper further analyzes the representations learned by different models and finds that larger models change less during training, suggesting over-parametrization. It also shows that feature reuse is mostly restricted to the lowest layers of the network. The study concludes that hybrid approaches to transfer learning, where only a subset of pretrained weights are used, can achieve similar performance to full transfer learning while being more efficient. These approaches also offer feature-independent benefits, such as improved convergence speed through weight scaling.Transfer learning using pre-trained models like ImageNet has become a standard in medical imaging. However, this paper challenges the assumption that transfer learning significantly improves performance in medical tasks. The study evaluates two large-scale medical imaging tasks—retinal fundus images and chest x-rays—and finds that transfer learning offers little benefit. Small, lightweight models perform comparably to standard ImageNet models, and ImageNet performance is not predictive of medical performance. The paper investigates the learned representations and finds that the differences in performance are due to over-parametrization of standard models rather than feature reuse. It isolates where useful feature reuse occurs and outlines implications for more efficient model exploration. It also explores feature-independent benefits of transfer learning, such as improved convergence speed from weight scaling. The study uses the RETINA dataset for retinal fundus images and the CHEXpert dataset for chest x-rays. The results show that transfer learning does not significantly improve performance on these tasks. The paper also finds that smaller models perform comparably to larger models, and that the effects of transfer learning are confounded by model size. The paper further analyzes the representations learned by different models and finds that larger models change less during training, suggesting over-parametrization. It also shows that feature reuse is mostly restricted to the lowest layers of the network. The study concludes that hybrid approaches to transfer learning, where only a subset of pretrained weights are used, can achieve similar performance to full transfer learning while being more efficient. These approaches also offer feature-independent benefits, such as improved convergence speed through weight scaling.