The paper introduces MA-LMM (Memory-Augmented Large Multimodal Model), a novel approach to enhance large multimodal models for long-term video understanding. Unlike existing models that struggle with the context length and GPU memory constraints of LLMs when processing long videos, MA-LMM processes video frames sequentially and stores past video information in a memory bank. This design allows the model to reference historical content without exceeding LLMs' context length or GPU memory limits. The memory bank can be seamlessly integrated into existing multimodal LLMs. MA-LMM consists of three main components: a visual encoder, a querying transformer (Q-Former) for temporal modeling, and a large language model. The visual encoder extracts visual features from video frames, which are then processed by the Q-Former to align visual and text embeddings. The Q-Former uses a long-term memory bank to store and accumulate past video features, enabling effective long-term temporal modeling. The memory bank is designed to be compatible with the Q-Former, acting as key and value in the attention operation. To address the computational demands of long videos, MA-LMM employs a memory bank compression technique that maintains the length of the memory bank constant relative to the input video length. This technique reduces temporal redundancies while preserving discriminative features. Experiments on various video understanding tasks, including long-term video understanding, video question answering, and video captioning, demonstrate that MA-LMM achieves state-of-the-art performance across multiple datasets. The model's ability to process long videos efficiently and effectively highlights its potential for real-time video analysis and applications requiring real-time analytical capabilities.The paper introduces MA-LMM (Memory-Augmented Large Multimodal Model), a novel approach to enhance large multimodal models for long-term video understanding. Unlike existing models that struggle with the context length and GPU memory constraints of LLMs when processing long videos, MA-LMM processes video frames sequentially and stores past video information in a memory bank. This design allows the model to reference historical content without exceeding LLMs' context length or GPU memory limits. The memory bank can be seamlessly integrated into existing multimodal LLMs. MA-LMM consists of three main components: a visual encoder, a querying transformer (Q-Former) for temporal modeling, and a large language model. The visual encoder extracts visual features from video frames, which are then processed by the Q-Former to align visual and text embeddings. The Q-Former uses a long-term memory bank to store and accumulate past video features, enabling effective long-term temporal modeling. The memory bank is designed to be compatible with the Q-Former, acting as key and value in the attention operation. To address the computational demands of long videos, MA-LMM employs a memory bank compression technique that maintains the length of the memory bank constant relative to the input video length. This technique reduces temporal redundancies while preserving discriminative features. Experiments on various video understanding tasks, including long-term video understanding, video question answering, and video captioning, demonstrate that MA-LMM achieves state-of-the-art performance across multiple datasets. The model's ability to process long videos efficiently and effectively highlights its potential for real-time video analysis and applications requiring real-time analytical capabilities.