This review discusses the evaluation of generalization in bioacoustics models, focusing on the challenges and considerations in bioacoustics data and the need for robust evaluation protocols. Bioacoustics is increasingly important for conservation, with applications in monitoring species, biodiversity, and environmental changes. Passive acoustic monitoring (PAM) collects large amounts of ambient sound data, but transforming this data into useful insights is challenging due to noise, class imbalance, and distribution shifts. Bioacoustics foundation models, which are general-purpose and adaptable, are needed to address these challenges. These models must be evaluated for their ability to generalize across different species, environments, and tasks. The review highlights the importance of using diverse and representative datasets, considering data shifts, and developing evaluation methods that reflect real-world deployment conditions. It also discusses the need for robust metrics and evaluation protocols that account for class imbalance, domain shifts, and the unique characteristics of bioacoustics data. The review emphasizes the importance of evaluating models not only on their performance but also on their ability to handle real-world scenarios, including rare species detection and environmental variability. The paper also explores various evaluation methods, including segment-based and event-based approaches, and discusses the use of metrics such as precision, recall, and ROC AUC. It highlights the importance of threshold-free metrics and the need for unbiased evaluation practices. The review concludes that developing robust evaluation protocols is essential for the effective deployment of bioacoustics models in a wide range of settings.This review discusses the evaluation of generalization in bioacoustics models, focusing on the challenges and considerations in bioacoustics data and the need for robust evaluation protocols. Bioacoustics is increasingly important for conservation, with applications in monitoring species, biodiversity, and environmental changes. Passive acoustic monitoring (PAM) collects large amounts of ambient sound data, but transforming this data into useful insights is challenging due to noise, class imbalance, and distribution shifts. Bioacoustics foundation models, which are general-purpose and adaptable, are needed to address these challenges. These models must be evaluated for their ability to generalize across different species, environments, and tasks. The review highlights the importance of using diverse and representative datasets, considering data shifts, and developing evaluation methods that reflect real-world deployment conditions. It also discusses the need for robust metrics and evaluation protocols that account for class imbalance, domain shifts, and the unique characteristics of bioacoustics data. The review emphasizes the importance of evaluating models not only on their performance but also on their ability to handle real-world scenarios, including rare species detection and environmental variability. The paper also explores various evaluation methods, including segment-based and event-based approaches, and discusses the use of metrics such as precision, recall, and ROC AUC. It highlights the importance of threshold-free metrics and the need for unbiased evaluation practices. The review concludes that developing robust evaluation protocols is essential for the effective deployment of bioacoustics models in a wide range of settings.