Large Language Models (LLMs) have shown impressive capabilities in various scientific tasks, including mathematics, physics, and chemistry. However, their effectiveness in handling complex statistical tasks remains underexplored. To address this gap, the authors introduce StatQA, a new benchmark for statistical analysis tasks. StatQA includes 11,623 examples to evaluate LLMs' proficiency in specialized statistical tasks and their ability to assess the applicability of statistical methods. The benchmark is designed to cover a wide range of statistical tasks, including descriptive statistics and inferential statistics, with a focus on hypothesis testing methods. The authors conduct systematic experiments with various LLMs, including both open-source and proprietary models, using different prompting strategies. Results show that even state-of-the-art models like GPT-4o achieve a best performance of 64.83%, indicating significant room for improvement. Open-source LLMs like LLaMA-3 show limited capability, while fine-tuned models outperform all in-context learning-based methods. Comparative human experiments reveal that LLMs primarily make applicability errors, whereas humans mostly make statistical task confusion errors, highlighting distinct areas of proficiency and deficiency. StatQA is constructed using an automated pipeline that synthesizes statistical tasks and their corresponding answers. The benchmark includes a diverse range of statistical tasks and methods, with a focus on hypothesis testing. The authors also conduct extensive evaluations on widely used LLMs to establish benchmarks for statistical tasks and explore strategies like domain-specific prompts and fine-tuning to enhance LLM performance. The study finds that LLMs, particularly fine-tuned models, show promise in statistical tasks but still struggle with accurately assessing the applicability of statistical methods. Humans, on the other hand, perform better in tasks requiring statistical task confusion but are less effective in applicability assessments. The study highlights the potential for complementary strengths between LLMs and humans, suggesting that further research into their collaborative potential could lead to more effective statistical analysis. The authors also discuss the importance of developing models that can better understand and utilize detailed methodological prerequisites and application contexts for statistical tasks.Large Language Models (LLMs) have shown impressive capabilities in various scientific tasks, including mathematics, physics, and chemistry. However, their effectiveness in handling complex statistical tasks remains underexplored. To address this gap, the authors introduce StatQA, a new benchmark for statistical analysis tasks. StatQA includes 11,623 examples to evaluate LLMs' proficiency in specialized statistical tasks and their ability to assess the applicability of statistical methods. The benchmark is designed to cover a wide range of statistical tasks, including descriptive statistics and inferential statistics, with a focus on hypothesis testing methods. The authors conduct systematic experiments with various LLMs, including both open-source and proprietary models, using different prompting strategies. Results show that even state-of-the-art models like GPT-4o achieve a best performance of 64.83%, indicating significant room for improvement. Open-source LLMs like LLaMA-3 show limited capability, while fine-tuned models outperform all in-context learning-based methods. Comparative human experiments reveal that LLMs primarily make applicability errors, whereas humans mostly make statistical task confusion errors, highlighting distinct areas of proficiency and deficiency. StatQA is constructed using an automated pipeline that synthesizes statistical tasks and their corresponding answers. The benchmark includes a diverse range of statistical tasks and methods, with a focus on hypothesis testing. The authors also conduct extensive evaluations on widely used LLMs to establish benchmarks for statistical tasks and explore strategies like domain-specific prompts and fine-tuning to enhance LLM performance. The study finds that LLMs, particularly fine-tuned models, show promise in statistical tasks but still struggle with accurately assessing the applicability of statistical methods. Humans, on the other hand, perform better in tasks requiring statistical task confusion but are less effective in applicability assessments. The study highlights the potential for complementary strengths between LLMs and humans, suggesting that further research into their collaborative potential could lead to more effective statistical analysis. The authors also discuss the importance of developing models that can better understand and utilize detailed methodological prerequisites and application contexts for statistical tasks.