The paper "Are Large Language Models Good Statisticians?" by Yizhang Zhu, Shiyan Du, Boyan Li, Yuyu Luo, and Nan Tang explores the capabilities of large language models (LLMs) in handling complex statistical tasks. The authors introduce StatQA, a new benchmark designed to evaluate LLMs' proficiency in statistical analysis, particularly in hypothesis testing methods. StatQA consists of 11,623 examples tailored to assess LLMs' ability to select appropriate statistical methods and relevant data columns. The study systematically experiments with various LLMs, including open-source models like LLaMA-3 and proprietary models such as GPT-4o, using different prompting strategies and fine-tuning methods. The results show that even state-of-the-art models achieve only a 64.83% performance, indicating significant room for improvement. Fine-tuned models outperform all in-context learning-based methods, while open-source models show limited capability. Human experiments are also conducted to compare LLMs with human statisticians. The findings highlight that LLMs primarily make applicability errors, while humans often make statistical task confusion errors. This divergence suggests that combining LLMs and human expertise could lead to complementary strengths, inviting further investigation into their collaborative potential. The paper concludes by discussing research opportunities, including developing models that better understand and utilize methodological prerequisites, expanding the benchmark dataset, and exploring human-AI collaboration strategies. The authors believe that StatQA fills a significant gap and provides a valuable resource for advancing LLMs in statistical analysis tasks.The paper "Are Large Language Models Good Statisticians?" by Yizhang Zhu, Shiyan Du, Boyan Li, Yuyu Luo, and Nan Tang explores the capabilities of large language models (LLMs) in handling complex statistical tasks. The authors introduce StatQA, a new benchmark designed to evaluate LLMs' proficiency in statistical analysis, particularly in hypothesis testing methods. StatQA consists of 11,623 examples tailored to assess LLMs' ability to select appropriate statistical methods and relevant data columns. The study systematically experiments with various LLMs, including open-source models like LLaMA-3 and proprietary models such as GPT-4o, using different prompting strategies and fine-tuning methods. The results show that even state-of-the-art models achieve only a 64.83% performance, indicating significant room for improvement. Fine-tuned models outperform all in-context learning-based methods, while open-source models show limited capability. Human experiments are also conducted to compare LLMs with human statisticians. The findings highlight that LLMs primarily make applicability errors, while humans often make statistical task confusion errors. This divergence suggests that combining LLMs and human expertise could lead to complementary strengths, inviting further investigation into their collaborative potential. The paper concludes by discussing research opportunities, including developing models that better understand and utilize methodological prerequisites, expanding the benchmark dataset, and exploring human-AI collaboration strategies. The authors believe that StatQA fills a significant gap and provides a valuable resource for advancing LLMs in statistical analysis tasks.