AGENTBOARD is a comprehensive benchmark and open-source evaluation framework for multi-turn large language models (LLMs) as agents. It addresses the challenges of evaluating LLM agents in diverse scenarios, particularly in partially-observable environments and multi-round interactions. Current evaluation frameworks focus on final success rates, offering limited insights into the process. AGENTBOARD introduces a fine-grained progress rate metric and a comprehensive evaluation toolkit with interactive visualization for detailed analysis of LLM agents. This framework enables a deeper understanding of agent capabilities and limitations, enhancing interpretability of their performance. AGENTBOARD includes nine diverse tasks across embodied, game, web, and tool environments, covering a range of agent tasks. Each environment is carefully crafted to ensure multi-round and partially observable characteristics. Subgoals are defined for each task, introducing a unified progress rate metric for tracking agent advancements. The framework provides an open-source toolkit with an analytical web panel for interactive visualization of various agent abilities. AGENTBOARD evaluates LLM agents in partially-observable environments, where agents must actively explore to understand their surroundings. It supports a wide range of tasks, including embodied AI, game agents, web agents, and tool agents. The framework includes a unified interface for easy customization and analysis. AGENTBOARD also provides detailed performance breakdowns for hard and easy examples, long-range interaction assessment, grounding accuracy, and trajectory analysis. In experiments, AGENTBOARD evaluates various LLM agents, including proprietary and open-source models. Results show that proprietary models, such as GPT-4, outperform open-source models in terms of progress rates and success rates. Strong code skills are beneficial for agent tasks, with DeepSeek-67b leading open-source models. Open-source models show varying deficiencies in grounding, world modeling, and self-reflection. AGENTBOARD offers an interactive visualization panel for analyzing LLM agents, providing insights into their capabilities and performance. The framework supports detailed analysis of agent behavior, including exploration behavior, sub-skill analysis, and long-range interaction. AGENTBOARD aims to facilitate detailed evaluation and understanding of LLM agents, driving further advancements in the field.AGENTBOARD is a comprehensive benchmark and open-source evaluation framework for multi-turn large language models (LLMs) as agents. It addresses the challenges of evaluating LLM agents in diverse scenarios, particularly in partially-observable environments and multi-round interactions. Current evaluation frameworks focus on final success rates, offering limited insights into the process. AGENTBOARD introduces a fine-grained progress rate metric and a comprehensive evaluation toolkit with interactive visualization for detailed analysis of LLM agents. This framework enables a deeper understanding of agent capabilities and limitations, enhancing interpretability of their performance. AGENTBOARD includes nine diverse tasks across embodied, game, web, and tool environments, covering a range of agent tasks. Each environment is carefully crafted to ensure multi-round and partially observable characteristics. Subgoals are defined for each task, introducing a unified progress rate metric for tracking agent advancements. The framework provides an open-source toolkit with an analytical web panel for interactive visualization of various agent abilities. AGENTBOARD evaluates LLM agents in partially-observable environments, where agents must actively explore to understand their surroundings. It supports a wide range of tasks, including embodied AI, game agents, web agents, and tool agents. The framework includes a unified interface for easy customization and analysis. AGENTBOARD also provides detailed performance breakdowns for hard and easy examples, long-range interaction assessment, grounding accuracy, and trajectory analysis. In experiments, AGENTBOARD evaluates various LLM agents, including proprietary and open-source models. Results show that proprietary models, such as GPT-4, outperform open-source models in terms of progress rates and success rates. Strong code skills are beneficial for agent tasks, with DeepSeek-67b leading open-source models. Open-source models show varying deficiencies in grounding, world modeling, and self-reflection. AGENTBOARD offers an interactive visualization panel for analyzing LLM agents, providing insights into their capabilities and performance. The framework supports detailed analysis of agent behavior, including exploration behavior, sub-skill analysis, and long-range interaction. AGENTBOARD aims to facilitate detailed evaluation and understanding of LLM agents, driving further advancements in the field.