The paper introduces Selective Kernel Networks (SKNets), a novel architecture that allows neurons in Convolutional Neural Networks (CNNs) to adaptively adjust their receptive field sizes based on multiple scales of input information. Inspired by the modulated receptive field sizes of visual cortical neurons, SKNets use a building block called Selective Kernel (SK) units, which fuse multiple branches with different kernel sizes using softmax attention. This approach enables neurons to capture target objects at various scales, improving the model's performance in object recognition tasks. Empirical results on the ImageNet and CIFAR datasets demonstrate that SKNets outperform existing state-of-the-art architectures with lower model complexity. Detailed analyses show that SKNet neurons can adaptively adjust their receptive fields, verifying the effectiveness of the proposed mechanism. The code and models are available at https://github.com/implus/SKNet.The paper introduces Selective Kernel Networks (SKNets), a novel architecture that allows neurons in Convolutional Neural Networks (CNNs) to adaptively adjust their receptive field sizes based on multiple scales of input information. Inspired by the modulated receptive field sizes of visual cortical neurons, SKNets use a building block called Selective Kernel (SK) units, which fuse multiple branches with different kernel sizes using softmax attention. This approach enables neurons to capture target objects at various scales, improving the model's performance in object recognition tasks. Empirical results on the ImageNet and CIFAR datasets demonstrate that SKNets outperform existing state-of-the-art architectures with lower model complexity. Detailed analyses show that SKNet neurons can adaptively adjust their receptive fields, verifying the effectiveness of the proposed mechanism. The code and models are available at https://github.com/implus/SKNet.