This paper explores the application of transfer learning in medical imaging, particularly focusing on the use of ImageNet-pretrained models. The authors evaluate the performance of standard ImageNet architectures and smaller, lightweight models on two large-scale medical imaging tasks: diabetic retinopathy diagnosis using retinal fundus images and chest x-ray pathology diagnosis using CHEXpert dataset. They find that transfer learning does not significantly improve performance, and smaller models can perform comparably to larger, ImageNet-trained models. The study also investigates the learned representations and features, revealing that over-parametrization of standard models is a significant factor in their performance. The authors isolate the layers where meaningful feature reuse occurs and propose hybrid approaches to transfer learning, such as reusing only the scaling of pretrained weights, which can lead to faster convergence and better performance. The findings highlight the need for more efficient model exploration and the importance of considering the specific characteristics of medical imaging tasks when applying transfer learning.This paper explores the application of transfer learning in medical imaging, particularly focusing on the use of ImageNet-pretrained models. The authors evaluate the performance of standard ImageNet architectures and smaller, lightweight models on two large-scale medical imaging tasks: diabetic retinopathy diagnosis using retinal fundus images and chest x-ray pathology diagnosis using CHEXpert dataset. They find that transfer learning does not significantly improve performance, and smaller models can perform comparably to larger, ImageNet-trained models. The study also investigates the learned representations and features, revealing that over-parametrization of standard models is a significant factor in their performance. The authors isolate the layers where meaningful feature reuse occurs and propose hybrid approaches to transfer learning, such as reusing only the scaling of pretrained weights, which can lead to faster convergence and better performance. The findings highlight the need for more efficient model exploration and the importance of considering the specific characteristics of medical imaging tasks when applying transfer learning.