The paper "Birds, bats and beyond: evaluating generalization in bioacoustics models" by Bart van Merriënboer, Jenny Hamer, Vincent Dumoulin, Eleni Triantafillou, and Tom Denton discusses the need for better models in passive acoustic monitoring (PAM) to reliably gain insights from large amounts of raw, unlabeled data. The authors emphasize the importance of bioacoustics foundation models, which are general-purpose and adaptable, for a wide range of downstream tasks. They highlight the complexity of evaluating such models due to the unique characteristics of bioacoustics data, including noise, class imbalance, and distribution shifts. The paper reviews various fields relevant to the evaluation of bioacoustics models, such as sound event detection, machine learning metrics, and transfer learning. It contextualizes these topics using the specific challenges of bioacoustics data, such as the large amounts of noise and the strong class imbalance. The authors aim to inform the design of robust evaluation protocols that can accurately predict the reliability of bioacoustics models in various settings. Key sections of the paper include: 1. **Introduction**: Discusses the importance of bioacoustics in conservation and the challenges of transforming raw data into actionable insights. 2. **Bioacoustics Data and Challenges**: Overviews different types of bioacoustics data, including large-scale data, targeted data, and annotated PAM data, and highlights their utility in model development. 3. **Robustness and Practical Utility**: Emphasizes the importance of minimizing the evaluation gap and considering real-world deployment conditions. 4. **Survey Overview**: Reviews relevant fields such as sound event detection, metrics, and evaluation of out-of-distribution generalization. 5. **Existing Efforts in Bioacoustics Evaluation**: Examines prominent benchmarks and competitions in bioacoustics and their alignment with the challenges discussed. The authors conclude that developing robust evaluation protocols for bioacoustics foundation models requires a paradigm shift away from traditional evaluation methods, focusing on robustness, generalization to new domains, and adaptation techniques like few-shot or zero-shot learning.The paper "Birds, bats and beyond: evaluating generalization in bioacoustics models" by Bart van Merriënboer, Jenny Hamer, Vincent Dumoulin, Eleni Triantafillou, and Tom Denton discusses the need for better models in passive acoustic monitoring (PAM) to reliably gain insights from large amounts of raw, unlabeled data. The authors emphasize the importance of bioacoustics foundation models, which are general-purpose and adaptable, for a wide range of downstream tasks. They highlight the complexity of evaluating such models due to the unique characteristics of bioacoustics data, including noise, class imbalance, and distribution shifts. The paper reviews various fields relevant to the evaluation of bioacoustics models, such as sound event detection, machine learning metrics, and transfer learning. It contextualizes these topics using the specific challenges of bioacoustics data, such as the large amounts of noise and the strong class imbalance. The authors aim to inform the design of robust evaluation protocols that can accurately predict the reliability of bioacoustics models in various settings. Key sections of the paper include: 1. **Introduction**: Discusses the importance of bioacoustics in conservation and the challenges of transforming raw data into actionable insights. 2. **Bioacoustics Data and Challenges**: Overviews different types of bioacoustics data, including large-scale data, targeted data, and annotated PAM data, and highlights their utility in model development. 3. **Robustness and Practical Utility**: Emphasizes the importance of minimizing the evaluation gap and considering real-world deployment conditions. 4. **Survey Overview**: Reviews relevant fields such as sound event detection, metrics, and evaluation of out-of-distribution generalization. 5. **Existing Efforts in Bioacoustics Evaluation**: Examines prominent benchmarks and competitions in bioacoustics and their alignment with the challenges discussed. The authors conclude that developing robust evaluation protocols for bioacoustics foundation models requires a paradigm shift away from traditional evaluation methods, focusing on robustness, generalization to new domains, and adaptation techniques like few-shot or zero-shot learning.