Robustness is crucial for deploying deep learning models in real-world applications. Vision Transformers (ViTs) have shown strong robustness and state-of-the-art performance in various vision tasks since their introduction in the 2020s, outperforming traditional CNNs. Recent advancements in large kernel convolutions have revived interest in CNNs, showing they can match or exceed ViT performance. However, the robustness of large kernel networks remains to be determined. This study investigates whether large kernel networks are inherently robust, how their robustness compares to CNNs and ViTs, and what factors contribute to their robustness.
Researchers from the Shanghai Jiao Tong University, Meituan, and several Chinese universities comprehensively evaluated the robustness of large kernel convolutional networks (convents) compared to traditional CNNs and ViTs across six benchmark datasets. Their experiments demonstrated that large kernel convents exhibit remarkable robustness, sometimes even outperforming ViTs. Through a series of nine experiments, they identified unique properties such as occlusion invariance, kernel attention patterns, and frequency characteristics that contribute to this robustness. This study challenges the prevailing belief that self-attention is necessary for achieving strong robustness, suggesting that traditional CNNs can achieve comparable levels of robustness and advocating for further advancements in large kernel network development.
Large kernel convolutional networks date back to early deep learning models but were overshadowed by small kernel networks like VGG-Net and ResNet. Recently, models like ConvNeXt and RepLKNet have revived interest in large kernels, improving performance, especially downstream tasks. However, their robustness still needs to be explored. The study addresses this gap by evaluating the robustness of large kernel networks through various experiments. ViTs are known for their strong robustness across vision tasks. Previous studies have shown that ViTs outperform CNNs in robustness, inspiring further research. This study compares large kernel networks’ robustness to ViTs and CNNs, providing new insights.
The study investigates whether large kernel convents are robust and how their robustness compares to traditional CNNs and ViTs. Using RepLKNet as the primary model, experiments were conducted across six robustness benchmarks. Models like ResNet-50, BiT, and ViT were used for comparison. Results show that RepLKNet outperforms traditional CNNs and ViTs in various robustness tests, including natural adversarial challenges, common corruptions, and domain adaptation. RepLKNet demonstrates superior robustness, particularly in occlusion scenarios and background dependency, highlighting the potential of large kernel convents in robust learning tasks.
Large Kernel ConvNets exhibit robust performance due to their occlusion invariance and kernel attention patterns. Experiments show that these networks better handle high occlusion, adversarial attacks, model perturbations, and noise frequency than traditional models like ResNet and ViT. They maintain performance even when layers are removed or subjected to frequency-based noise. The robustness is largely attributed to the large kernel size, as replacing large kernels with smaller ones significantly degrades performance. This robustness improves with increasing kernel size, showing consistent enhancements in various benchmarks and confirming the importance of large kernels in ConvNet design.
While our empirical analysis strongly supports the robustness of large kernel ConvNets, the study acknowledges the need for more direct theoretical proofs given the intricate nature of deep learning. Moreover, computational constraints limited our ability to conduct kernel size ablations on ImageNet-21K, focusing instead on ImageNet-1K. Nevertheless, our research confirms the significant robustness of large kernel ConvNets across six standard benchmark datasets, accompanied by a thorough quantitative and qualitative examination. These insights shed light on the factors underlying their resilience, suggesting promising avenues for advancing the application and development of large kernel ConvNets in future research and practical use.
Check out the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter.
Join our Telegram Channel and LinkedIn Group.
If you like our work, you will love our newsletter..
Don’t Forget to join our 46k+ ML SubReddit
Sana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.
