The article discusses the role of humans and artificial intelligence (AI) in determining what is relevant in information retrieval and other tasks. While large language models (LLMs) like ChatGPT can provide relevance judgments, there are several issues with relying solely on them. These include bias toward specific LLMs, bias toward certain user groups, and the risk of misinformation. Additionally, LLMs may not be able to accurately assess factuality, and there is a risk of hallucinations. The article suggests that a spectrum of human-AI collaboration is needed, ranging from humans making all judgments to LLMs completely replacing humans. It highlights the importance of human oversight in decision-making processes, especially in scenarios where LLMs may not be reliable. The article also discusses the potential for LLMs to assist humans in tasks such as highlighting relevant passages and generating rationales. It concludes that while LLMs can be useful tools, they should be used to amplify rather than replace human intelligence, particularly in scenarios where human judgment is critical. The article calls for further research into how to best combine human and machine intelligence to improve the efficiency, effectiveness, and fairness of decision-making processes.The article discusses the role of humans and artificial intelligence (AI) in determining what is relevant in information retrieval and other tasks. While large language models (LLMs) like ChatGPT can provide relevance judgments, there are several issues with relying solely on them. These include bias toward specific LLMs, bias toward certain user groups, and the risk of misinformation. Additionally, LLMs may not be able to accurately assess factuality, and there is a risk of hallucinations. The article suggests that a spectrum of human-AI collaboration is needed, ranging from humans making all judgments to LLMs completely replacing humans. It highlights the importance of human oversight in decision-making processes, especially in scenarios where LLMs may not be reliable. The article also discusses the potential for LLMs to assist humans in tasks such as highlighting relevant passages and generating rationales. It concludes that while LLMs can be useful tools, they should be used to amplify rather than replace human intelligence, particularly in scenarios where human judgment is critical. The article calls for further research into how to best combine human and machine intelligence to improve the efficiency, effectiveness, and fairness of decision-making processes.