Video-LLaMA is a multi-modal framework that enables large language models (LLMs) to understand both visual and auditory content in videos. It is built by bootstrapping cross-modal training from frozen pre-trained visual and audio encoders and frozen LLMs. Unlike previous works that only process visual or audio signals, Video-LLaMA addresses two challenges: capturing temporal changes in visual scenes and integrating audio-visual signals. To tackle the first challenge, a Video Q-former is introduced to assemble a pre-trained image encoder into the video encoder and a video-to-text generation task is used to learn video-language correspondence. For the second challenge, ImageBind is used as a pre-trained audio encoder, and an Audio Q-former is introduced to learn reasonable auditory query embeddings for the LLM module. To align the output of both visual and audio encoders with the LLM's embedding space, Video-LLaMA is first trained on massive video/image-caption pairs and then fine-tuned with visual-instruction datasets. Video-LLaMA demonstrates the ability to perceive and comprehend video content and generate meaningful responses based on visual and auditory information. Video-LLaMA is designed with two branches: the Vision-Language Branch and the Audio-Language Branch. The Vision-Language Branch processes video frames and converts them into query representations compatible with LLMs. It uses a pre-trained image encoder, position embedding layer, video Q-former, and linear layer to project video representations into the same dimension as text embeddings. The Audio-Language Branch processes audio signals and converts them into query representations compatible with LLMs. It uses a pre-trained audio encoder, position embedding layer, audio Q-former, and linear layer to map audio representations into the embedding space of LLMs. Video-LLaMA is trained with a multi-branch cross-modal pre-training approach. The vision-language branch is pre-trained on large-scale video caption datasets and then fine-tuned on high-quality instruction-following datasets. The audio-language branch is trained using visual-text data due to the scarcity of audio-text data. The audio-language branch leverages ImageBind to align different modalities to a common embedding space. Video-LLaMA is capable of understanding audio during inference, even though the audio interface has never been trained on audio data. Video-LLaMA has been shown to have strong capabilities in audio and video-grounded conversations. It can understand and respond to both visual and auditory content in videos, capture temporal dynamics in videos, perceive and understand static images, and recognize common-knowledge concepts in visual signals. Video-LLaMA is open-sourced, with the entire training code and model weights made available for developers. It also provides online demo websites and offline deployment guides for users to experience its capabilities. Video-LLaMA is a promising prototype for audio-visual AI assistants and has the potential to be further improved and maintained.Video-LLaMA is a multi-modal framework that enables large language models (LLMs) to understand both visual and auditory content in videos. It is built by bootstrapping cross-modal training from frozen pre-trained visual and audio encoders and frozen LLMs. Unlike previous works that only process visual or audio signals, Video-LLaMA addresses two challenges: capturing temporal changes in visual scenes and integrating audio-visual signals. To tackle the first challenge, a Video Q-former is introduced to assemble a pre-trained image encoder into the video encoder and a video-to-text generation task is used to learn video-language correspondence. For the second challenge, ImageBind is used as a pre-trained audio encoder, and an Audio Q-former is introduced to learn reasonable auditory query embeddings for the LLM module. To align the output of both visual and audio encoders with the LLM's embedding space, Video-LLaMA is first trained on massive video/image-caption pairs and then fine-tuned with visual-instruction datasets. Video-LLaMA demonstrates the ability to perceive and comprehend video content and generate meaningful responses based on visual and auditory information. Video-LLaMA is designed with two branches: the Vision-Language Branch and the Audio-Language Branch. The Vision-Language Branch processes video frames and converts them into query representations compatible with LLMs. It uses a pre-trained image encoder, position embedding layer, video Q-former, and linear layer to project video representations into the same dimension as text embeddings. The Audio-Language Branch processes audio signals and converts them into query representations compatible with LLMs. It uses a pre-trained audio encoder, position embedding layer, audio Q-former, and linear layer to map audio representations into the embedding space of LLMs. Video-LLaMA is trained with a multi-branch cross-modal pre-training approach. The vision-language branch is pre-trained on large-scale video caption datasets and then fine-tuned on high-quality instruction-following datasets. The audio-language branch is trained using visual-text data due to the scarcity of audio-text data. The audio-language branch leverages ImageBind to align different modalities to a common embedding space. Video-LLaMA is capable of understanding audio during inference, even though the audio interface has never been trained on audio data. Video-LLaMA has been shown to have strong capabilities in audio and video-grounded conversations. It can understand and respond to both visual and auditory content in videos, capture temporal dynamics in videos, perceive and understand static images, and recognize common-knowledge concepts in visual signals. Video-LLaMA is open-sourced, with the entire training code and model weights made available for developers. It also provides online demo websites and offline deployment guides for users to experience its capabilities. Video-LLaMA is a promising prototype for audio-visual AI assistants and has the potential to be further improved and maintained.