The paper addresses the challenge of uncertainty quantification in fine-tuned large language models (LLMs) using low-rank adaptation (LoRA) ensembles. Fine-tuning LLMs can improve task-specific performance, but understanding what the model has learned and forgotten remains challenging. The authors derive principled posterior approximations for fine-tuned LLMs using computationally efficient LoRA ensembles. They analyze three common multiple-choice datasets—CommonsenseQA (CQA), MMLU STEM, and MMLU Social Sciences—using the pre-trained Mistral-7b model. By quantifying uncertainty using predictive entropy and mutual information, they explore how these measures can be used to understand dataset complexity and model efficacy on different target domains. The results show that LoRA ensembles provide valuable insights into the model's knowledge and limitations, particularly in identifying out-of-domain samples and understanding the architecture's limitations. The paper contributes to the field by providing a systematic approach to uncertainty quantification in fine-tuned LLMs, which is crucial for making informed decisions about the reliability and interpretability of LLMs.The paper addresses the challenge of uncertainty quantification in fine-tuned large language models (LLMs) using low-rank adaptation (LoRA) ensembles. Fine-tuning LLMs can improve task-specific performance, but understanding what the model has learned and forgotten remains challenging. The authors derive principled posterior approximations for fine-tuned LLMs using computationally efficient LoRA ensembles. They analyze three common multiple-choice datasets—CommonsenseQA (CQA), MMLU STEM, and MMLU Social Sciences—using the pre-trained Mistral-7b model. By quantifying uncertainty using predictive entropy and mutual information, they explore how these measures can be used to understand dataset complexity and model efficacy on different target domains. The results show that LoRA ensembles provide valuable insights into the model's knowledge and limitations, particularly in identifying out-of-domain samples and understanding the architecture's limitations. The paper contributes to the field by providing a systematic approach to uncertainty quantification in fine-tuned LLMs, which is crucial for making informed decisions about the reliability and interpretability of LLMs.