USER-LLM is a novel framework that leverages user embeddings to contextualize large language models (LLMs). The embeddings, distilled from diverse user interactions through self-supervised pretraining, capture latent user preferences and their evolution over time. These embeddings are integrated with LLMs using cross-attention and soft-prompting, enabling LLMs to dynamically adapt to user context. The framework was evaluated on MovieLens, Amazon Review, and Google Local Review datasets, demonstrating significant performance gains across various tasks. Notably, USER-LLM outperforms text-prompt-based contextualization on long sequence tasks and tasks requiring deep user understanding while being computationally efficient. The integration of Perceiver layers further streamlines the process, reducing computational demands. The framework consists of two key phases: generating high-quality user embeddings and contextualizing LLMs with these embeddings. The user encoder is pretrained on user interaction data using a multi-feature autoregressive Transformer to capture long-range dependencies and contextual relationships. The embeddings are then integrated with the LLM during finetuning using cross-attention, allowing dynamic context injection. The approach supports various encoder architectures and multimodal fusion mechanisms. USER-LLM shows great potential for enhancing performance in user understanding, personalized recommendations, and text generation. The framework offers flexible training strategies, including full finetuning, encoder-only finetuning, LoRA-based finetuning, and projection-only finetuning. The experiments demonstrate that USER-LLM achieves competitive performance compared to non-LLM baselines and text-prompt-based LLM personalization techniques, particularly in handling long sequences and understanding users deeply. The framework is computationally efficient and preserves LLM knowledge, making it suitable for real-world user understanding applications. Future research could focus on optimizing user embedding generation, investigating the alignment between user embeddings and the language model space, and training USER-LLM on a diverse range of tasks to enhance its generalization abilities and adaptability.USER-LLM is a novel framework that leverages user embeddings to contextualize large language models (LLMs). The embeddings, distilled from diverse user interactions through self-supervised pretraining, capture latent user preferences and their evolution over time. These embeddings are integrated with LLMs using cross-attention and soft-prompting, enabling LLMs to dynamically adapt to user context. The framework was evaluated on MovieLens, Amazon Review, and Google Local Review datasets, demonstrating significant performance gains across various tasks. Notably, USER-LLM outperforms text-prompt-based contextualization on long sequence tasks and tasks requiring deep user understanding while being computationally efficient. The integration of Perceiver layers further streamlines the process, reducing computational demands. The framework consists of two key phases: generating high-quality user embeddings and contextualizing LLMs with these embeddings. The user encoder is pretrained on user interaction data using a multi-feature autoregressive Transformer to capture long-range dependencies and contextual relationships. The embeddings are then integrated with the LLM during finetuning using cross-attention, allowing dynamic context injection. The approach supports various encoder architectures and multimodal fusion mechanisms. USER-LLM shows great potential for enhancing performance in user understanding, personalized recommendations, and text generation. The framework offers flexible training strategies, including full finetuning, encoder-only finetuning, LoRA-based finetuning, and projection-only finetuning. The experiments demonstrate that USER-LLM achieves competitive performance compared to non-LLM baselines and text-prompt-based LLM personalization techniques, particularly in handling long sequences and understanding users deeply. The framework is computationally efficient and preserves LLM knowledge, making it suitable for real-world user understanding applications. Future research could focus on optimizing user embedding generation, investigating the alignment between user embeddings and the language model space, and training USER-LLM on a diverse range of tasks to enhance its generalization abilities and adaptability.