AutoLoRA is a meta learning-based framework designed to automatically determine the optimal ranks for each layer in low-rank adaptation (LoRA), a method used for efficient fine-tuning of large pretrained models. LoRA updates low-rank incremental matrices on top of frozen pretrained weights, but it suffers from uniform rank assignment across layers and the need for exhaustive hyperparameter search. AutoLoRA addresses these limitations by associating each rank-1 matrix in the update matrix with a selection variable, which determines whether the matrix should be discarded. The selection variables are learned through a meta learning process, where the weights in the update matrices are optimized on the training dataset, and the selection variables are updated on the validation dataset. The optimal rank is then determined by thresholding the values of these variables. Extensive experiments on natural language understanding, generation, and sequence labeling tasks demonstrate the effectiveness of AutoLoRA, showing superior performance and computational efficiency compared to baseline methods.AutoLoRA is a meta learning-based framework designed to automatically determine the optimal ranks for each layer in low-rank adaptation (LoRA), a method used for efficient fine-tuning of large pretrained models. LoRA updates low-rank incremental matrices on top of frozen pretrained weights, but it suffers from uniform rank assignment across layers and the need for exhaustive hyperparameter search. AutoLoRA addresses these limitations by associating each rank-1 matrix in the update matrix with a selection variable, which determines whether the matrix should be discarded. The selection variables are learned through a meta learning process, where the weights in the update matrices are optimized on the training dataset, and the selection variables are updated on the validation dataset. The optimal rank is then determined by thresholding the values of these variables. Extensive experiments on natural language understanding, generation, and sequence labeling tasks demonstrate the effectiveness of AutoLoRA, showing superior performance and computational efficiency compared to baseline methods.