This paper investigates the comparative study of annotation quality between crowdsourcing and Large Language Models (LLMs) via label aggregation. The authors address two key issues: the lack of existing datasets for reliable evaluations and the importance of aggregated labels in crowdsourcing. They propose a benchmark using existing crowdsourcing datasets and evaluate the quality of individual crowd and LLM labels. Additionally, they introduce a Crowd-LLM hybrid label aggregation method and find that adding LLM labels to existing crowdsourcing datasets enhances the quality of aggregated labels, which is higher than the quality of LLM labels alone. The study also highlights that while good LLMs can improve label quality, normal LLMs (e.g., Vicuna) often cannot replace the quality of crowd workers. The research concludes with recommendations for practical applications and future directions, including the need to explore other types of labels.This paper investigates the comparative study of annotation quality between crowdsourcing and Large Language Models (LLMs) via label aggregation. The authors address two key issues: the lack of existing datasets for reliable evaluations and the importance of aggregated labels in crowdsourcing. They propose a benchmark using existing crowdsourcing datasets and evaluate the quality of individual crowd and LLM labels. Additionally, they introduce a Crowd-LLM hybrid label aggregation method and find that adding LLM labels to existing crowdsourcing datasets enhances the quality of aggregated labels, which is higher than the quality of LLM labels alone. The study also highlights that while good LLMs can improve label quality, normal LLMs (e.g., Vicuna) often cannot replace the quality of crowd workers. The research concludes with recommendations for practical applications and future directions, including the need to explore other types of labels.