The paper introduces $M^3$: Matryoshka Multimodal Models, which aims to address the inefficiency of large multimodal models (LMMs) by representing visual content as nested sets of visual tokens. These tokens capture information across multiple coarse-to-fine granularities, allowing for flexible control over the visual granularity during inference. The approach is inspired by the concept of Matryoshka Dolls, where each level of tokens is derived from the previous level, ensuring a nested structure. Key benefits of $M^3$ include: 1. **Flexibility in Granularity**: Users can control the number of visual tokens used based on the complexity of the input image or video, improving efficiency and performance. 2. **Efficiency Analysis**: $M^3$ reduces the number of tokens, leading to faster processing and lower computational costs. 3. **Performance-Efficiency Trade-off**: The model can achieve better performance with fewer tokens, highlighting a significant gap between the oracle upper bound and current fixed-scale representations. 4. **Dataset Complexity Evaluation**: $M^3$ serves as a framework to evaluate the visual complexity required for different datasets, finding that many benchmarks can be handled with only a few tokens. The paper also includes experimental results showing that $M^3$ maintains or improves performance on various image and video understanding benchmarks, even with fewer tokens. Additionally, it discusses the broader impact and potential future directions, such as developing an effective visual token predictor to bridge the gap between the oracle and the model's actual performance.The paper introduces $M^3$: Matryoshka Multimodal Models, which aims to address the inefficiency of large multimodal models (LMMs) by representing visual content as nested sets of visual tokens. These tokens capture information across multiple coarse-to-fine granularities, allowing for flexible control over the visual granularity during inference. The approach is inspired by the concept of Matryoshka Dolls, where each level of tokens is derived from the previous level, ensuring a nested structure. Key benefits of $M^3$ include: 1. **Flexibility in Granularity**: Users can control the number of visual tokens used based on the complexity of the input image or video, improving efficiency and performance. 2. **Efficiency Analysis**: $M^3$ reduces the number of tokens, leading to faster processing and lower computational costs. 3. **Performance-Efficiency Trade-off**: The model can achieve better performance with fewer tokens, highlighting a significant gap between the oracle upper bound and current fixed-scale representations. 4. **Dataset Complexity Evaluation**: $M^3$ serves as a framework to evaluate the visual complexity required for different datasets, finding that many benchmarks can be handled with only a few tokens. The paper also includes experimental results showing that $M^3$ maintains or improves performance on various image and video understanding benchmarks, even with fewer tokens. Additionally, it discusses the broader impact and potential future directions, such as developing an effective visual token predictor to bridge the gap between the oracle and the model's actual performance.