Scaling on Scales (S²) is a method that enables smaller vision models to achieve performance comparable to or better than larger models by processing images at multiple scales. This approach involves running a pre-trained and frozen smaller model (e.g., ViT-B or ViT-L) on multiple image scales, generating multi-scale features that outperform larger models in tasks like classification, segmentation, depth estimation, and multimodal LLM (MLLM) benchmarks. S² achieves state-of-the-art performance on the V* benchmark for detailed MLLM understanding, surpassing models like GPT-4V. The study shows that multi-scale smaller models can approximate the features of larger models, suggesting that most representations learned by large pre-trained models can also be obtained from smaller models. S² scaling is found to be as effective as or even better than model size scaling in many scenarios, with significantly fewer parameters and comparable GFLOPS. However, larger models still outperform smaller ones on hard examples, indicating that while S² is a viable alternative, larger models may still be necessary in certain cases. The S² approach involves a parameter-free mechanism called S²-Wrapper, which extends any pre-trained vision model to multiple image scales by splitting images into sub-images of the same size as the original input. This allows for efficient and effective multi-scale feature extraction without additional parameters. Experiments show that S² scaling can achieve comparable or better performance than model size scaling on various tasks, including image classification, semantic segmentation, depth estimation, and MLLMs. In robotic manipulation tasks, S² scaling also improves performance, demonstrating its effectiveness in practical applications. The study further shows that smaller models with S² scaling can match or exceed the generalization capabilities of larger models when pre-trained appropriately. This suggests that S² scaling can be a preferred approach for achieving powerful visual representations without the need for significantly larger models.Scaling on Scales (S²) is a method that enables smaller vision models to achieve performance comparable to or better than larger models by processing images at multiple scales. This approach involves running a pre-trained and frozen smaller model (e.g., ViT-B or ViT-L) on multiple image scales, generating multi-scale features that outperform larger models in tasks like classification, segmentation, depth estimation, and multimodal LLM (MLLM) benchmarks. S² achieves state-of-the-art performance on the V* benchmark for detailed MLLM understanding, surpassing models like GPT-4V. The study shows that multi-scale smaller models can approximate the features of larger models, suggesting that most representations learned by large pre-trained models can also be obtained from smaller models. S² scaling is found to be as effective as or even better than model size scaling in many scenarios, with significantly fewer parameters and comparable GFLOPS. However, larger models still outperform smaller ones on hard examples, indicating that while S² is a viable alternative, larger models may still be necessary in certain cases. The S² approach involves a parameter-free mechanism called S²-Wrapper, which extends any pre-trained vision model to multiple image scales by splitting images into sub-images of the same size as the original input. This allows for efficient and effective multi-scale feature extraction without additional parameters. Experiments show that S² scaling can achieve comparable or better performance than model size scaling on various tasks, including image classification, semantic segmentation, depth estimation, and MLLMs. In robotic manipulation tasks, S² scaling also improves performance, demonstrating its effectiveness in practical applications. The study further shows that smaller models with S² scaling can match or exceed the generalization capabilities of larger models when pre-trained appropriately. This suggests that S² scaling can be a preferred approach for achieving powerful visual representations without the need for significantly larger models.