The paper introduces Conv-LoRA, a parameter-efficient fine-tuning approach for the Segment Anything Model (SAM), which is a foundational framework for image segmentation. While SAM excels in zero-shot generalization, it struggles in specialized domains like medical imagery and remote sensing due to its plain ViT encoder lacking vision-specific inductive biases. Conv-LoRA integrates lightweight convolutional parameters into Low-Rank Adaptation (LoRA), enhancing SAM's ability to capture high-level image semantics and local priors. The method uses a Mixture-of-Experts (MoE) approach to dynamically select the appropriate feature scale for injecting local priors, improving performance across various benchmarks in natural images, agriculture, remote sensing, and healthcare. Experiments show that Conv-LoRA outperforms other fine-tuning methods, demonstrating its effectiveness in adapting SAM to real-world semantic segmentation tasks.The paper introduces Conv-LoRA, a parameter-efficient fine-tuning approach for the Segment Anything Model (SAM), which is a foundational framework for image segmentation. While SAM excels in zero-shot generalization, it struggles in specialized domains like medical imagery and remote sensing due to its plain ViT encoder lacking vision-specific inductive biases. Conv-LoRA integrates lightweight convolutional parameters into Low-Rank Adaptation (LoRA), enhancing SAM's ability to capture high-level image semantics and local priors. The method uses a Mixture-of-Experts (MoE) approach to dynamically select the appropriate feature scale for injecting local priors, improving performance across various benchmarks in natural images, agriculture, remote sensing, and healthcare. Experiments show that Conv-LoRA outperforms other fine-tuning methods, demonstrating its effectiveness in adapting SAM to real-world semantic segmentation tasks.