This paper introduces a novel concept-based explainability framework for large multimodal models (LMMs). The framework uses dictionary learning to extract concepts that are semantically grounded in both visual and textual domains. The method involves extracting token representations from a pretrained LMM, decomposing them into a dictionary of concepts, and then grounding these concepts in both visual and textual modalities. The extracted concepts are shown to be useful for interpreting the internal representations of test samples. The framework is evaluated on various tasks, including visual and textual grounding of concepts, and is shown to outperform other methods in terms of concept disentanglement and grounding quality. The method is applied to the DePALM model for captioning tasks on the COCO dataset, and the results demonstrate the effectiveness of the approach in understanding the internal representations of LMMs. The framework is also shown to be effective in interpreting the representations of test samples for different tokens, such as 'Dog', 'Cat', and 'Bus'. The results indicate that the extracted concepts provide meaningful and diverse information about the input samples, and that the multimodal grounding of these concepts enhances the interpretability of LMMs. The method is also shown to be effective in analyzing the quality of multimodal grounding across different layers of the LMM. Overall, the framework provides a novel approach to understanding the internal representations of LMMs through concept-based explainability.This paper introduces a novel concept-based explainability framework for large multimodal models (LMMs). The framework uses dictionary learning to extract concepts that are semantically grounded in both visual and textual domains. The method involves extracting token representations from a pretrained LMM, decomposing them into a dictionary of concepts, and then grounding these concepts in both visual and textual modalities. The extracted concepts are shown to be useful for interpreting the internal representations of test samples. The framework is evaluated on various tasks, including visual and textual grounding of concepts, and is shown to outperform other methods in terms of concept disentanglement and grounding quality. The method is applied to the DePALM model for captioning tasks on the COCO dataset, and the results demonstrate the effectiveness of the approach in understanding the internal representations of LMMs. The framework is also shown to be effective in interpreting the representations of test samples for different tokens, such as 'Dog', 'Cat', and 'Bus'. The results indicate that the extracted concepts provide meaningful and diverse information about the input samples, and that the multimodal grounding of these concepts enhances the interpretability of LMMs. The method is also shown to be effective in analyzing the quality of multimodal grounding across different layers of the LMM. Overall, the framework provides a novel approach to understanding the internal representations of LMMs through concept-based explainability.