LoRA-XS is a novel parameter-efficient fine-tuning method that significantly reduces the number of trainable parameters while maintaining or improving model performance. Building on the principles of Low-Rank Adaptation (LoRA), LoRA-XS introduces a small $r \times r$ weight matrix between frozen LoRA matrices, which are constructed using Singular Value Decomposition (SVD) of the original weight matrix. This approach ensures independence from model dimensions, enabling more parameter-efficient fine-tuning, especially for larger models. LoRA-XS achieves a remarkable reduction of over 100x in the number of trainable parameters compared to LoRA in 7B models. Benchmarking across various scales, including GLUE, GSM8k, and MATH datasets, shows that LoRA-XS outperforms LoRA and recent state-of-the-art approaches like VeRA in terms of parameter efficiency while maintaining competitive performance. The method's flexibility in controlling the number of additional parameters and its ability to handle large-scale models make it a promising solution for efficient model personalization and task-specific optimization.LoRA-XS is a novel parameter-efficient fine-tuning method that significantly reduces the number of trainable parameters while maintaining or improving model performance. Building on the principles of Low-Rank Adaptation (LoRA), LoRA-XS introduces a small $r \times r$ weight matrix between frozen LoRA matrices, which are constructed using Singular Value Decomposition (SVD) of the original weight matrix. This approach ensures independence from model dimensions, enabling more parameter-efficient fine-tuning, especially for larger models. LoRA-XS achieves a remarkable reduction of over 100x in the number of trainable parameters compared to LoRA in 7B models. Benchmarking across various scales, including GLUE, GSM8k, and MATH datasets, shows that LoRA-XS outperforms LoRA and recent state-of-the-art approaches like VeRA in terms of parameter efficiency while maintaining competitive performance. The method's flexibility in controlling the number of additional parameters and its ability to handle large-scale models make it a promising solution for efficient model personalization and task-specific optimization.