The paper introduces the Language Agent Biology Benchmark (LAB-Bench), a comprehensive dataset designed to evaluate the practical capabilities of large language models (LLMs) in biology research. LAB-Bench consists of over 2,400 multiple-choice questions covering various tasks such as literature recall, figure interpretation, database access, protocol writing, and DNA/protein sequence manipulation. The authors highlight the importance of practical evaluation benchmarks for AI systems in scientific research, as existing benchmarks often focus on rote knowledge and textbook-style questions rather than real-world tasks. The paper details the construction of LAB-Bench, which includes both programmatic and manual question generation strategies. It also presents the performance of several frontier models on LAB-Bench tasks, comparing their results to those of human experts. The models generally show lower performance on tasks requiring complex sequence manipulation and database access, while performing better on tasks like figure interpretation and protocol troubleshooting. Key findings include: 1. **Model Performance**: Models often refuse to answer questions, especially those requiring information lookup, and struggle with tasks involving DNA and protein sequences. 2. **Human Performance**: Human experts outperform models in most categories, with notable exceptions in TableQA and primer selection tasks. 3. **Cloning Scenarios**: These complex, multi-step tasks are particularly challenging for models, with high accuracy achieved only through heuristics and guessing. 4. **Distractors**: High-quality distractors are crucial for accurate performance assessment, as models often rely on elimination and guessing rather than deductive reasoning. The authors discuss the limitations of the current benchmark, including the difficulty of designing plausible distractors and the challenge of evaluating human-hard tasks. They also emphasize the need for effective benchmarking strategies as AI systems continue to advance and are increasingly used in scientific research. The paper concludes with a call for community input on expanding the benchmark to cover more topics and improving evaluation methods.The paper introduces the Language Agent Biology Benchmark (LAB-Bench), a comprehensive dataset designed to evaluate the practical capabilities of large language models (LLMs) in biology research. LAB-Bench consists of over 2,400 multiple-choice questions covering various tasks such as literature recall, figure interpretation, database access, protocol writing, and DNA/protein sequence manipulation. The authors highlight the importance of practical evaluation benchmarks for AI systems in scientific research, as existing benchmarks often focus on rote knowledge and textbook-style questions rather than real-world tasks. The paper details the construction of LAB-Bench, which includes both programmatic and manual question generation strategies. It also presents the performance of several frontier models on LAB-Bench tasks, comparing their results to those of human experts. The models generally show lower performance on tasks requiring complex sequence manipulation and database access, while performing better on tasks like figure interpretation and protocol troubleshooting. Key findings include: 1. **Model Performance**: Models often refuse to answer questions, especially those requiring information lookup, and struggle with tasks involving DNA and protein sequences. 2. **Human Performance**: Human experts outperform models in most categories, with notable exceptions in TableQA and primer selection tasks. 3. **Cloning Scenarios**: These complex, multi-step tasks are particularly challenging for models, with high accuracy achieved only through heuristics and guessing. 4. **Distractors**: High-quality distractors are crucial for accurate performance assessment, as models often rely on elimination and guessing rather than deductive reasoning. The authors discuss the limitations of the current benchmark, including the difficulty of designing plausible distractors and the challenge of evaluating human-hard tasks. They also emphasize the need for effective benchmarking strategies as AI systems continue to advance and are increasingly used in scientific research. The paper concludes with a call for community input on expanding the benchmark to cover more topics and improving evaluation methods.