This study explores the novel application of residual-based large language models (LLMs) as encoders for biomedical imaging tasks, a domain typically lacking textual data. By integrating a frozen transformer block from pre-trained LLMs into visual encoders, the researchers achieve significant performance boosts across various 2D and 3D biomedical imaging tasks. This approach diverges from traditional multi-modal vision-language frameworks, which rely on language-driven prompts and inputs. The method is shown to enhance performance on datasets such as MedMNIST-2D and 3D, achieving state-of-the-art results. The study highlights the versatility and effectiveness of LLMs in biomedical imaging, opening new avenues for their application in this specialized domain. The code for this research is available at <https://github.com/ZhixinLai/LLMBoostMedical>.This study explores the novel application of residual-based large language models (LLMs) as encoders for biomedical imaging tasks, a domain typically lacking textual data. By integrating a frozen transformer block from pre-trained LLMs into visual encoders, the researchers achieve significant performance boosts across various 2D and 3D biomedical imaging tasks. This approach diverges from traditional multi-modal vision-language frameworks, which rely on language-driven prompts and inputs. The method is shown to enhance performance on datasets such as MedMNIST-2D and 3D, achieving state-of-the-art results. The study highlights the versatility and effectiveness of LLMs in biomedical imaging, opening new avenues for their application in this specialized domain. The code for this research is available at <https://github.com/ZhixinLai/LLMBoostMedical>.