A modified Moral Turing Test (m-MTT) was conducted to assess whether people can distinguish moral evaluations generated by a popular AI language model (GPT-4) from those generated by humans. Participants rated the quality of moral evaluations from both sources, finding that AI-generated responses were often perceived as more virtuous, intelligent, and trustworthy than human ones. When asked to identify the source of each evaluation, participants performed significantly above chance levels, suggesting they could distinguish between human and computer-generated responses. However, this was not due to the AI's inferior moral reasoning but potentially due to its perceived superiority. The study highlights concerns that people may uncritically accept potentially harmful moral guidance from AI. The results suggest that people's attributions toward AI moral discourse are distinct from those toward other AI technologies, and that the perceived moral intelligence of AI systems could significantly impact human welfare. The study also underscores the need for safeguards around generative language models in matters of morality. The findings have implications for understanding how ordinary people perceive and interact with AI in moral domains, and for developing future AI systems that better replicate human moral reasoning. The study was conducted with a representative sample of U.S. adults, and the results were analyzed using statistical methods to assess the quality of moral evaluations and the ability to identify their source. The study found that participants were able to distinguish between human and computer-generated moral evaluations, but this was not due to the AI's inferior moral reasoning but potentially due to its perceived superiority. The study also found that participants rated AI-generated moral evaluations as higher quality than human ones, suggesting that people may be more likely to accept AI moral guidance without critical evaluation. The study has implications for the development and use of AI in moral domains, and for the need for safeguards to prevent harmful interactions.A modified Moral Turing Test (m-MTT) was conducted to assess whether people can distinguish moral evaluations generated by a popular AI language model (GPT-4) from those generated by humans. Participants rated the quality of moral evaluations from both sources, finding that AI-generated responses were often perceived as more virtuous, intelligent, and trustworthy than human ones. When asked to identify the source of each evaluation, participants performed significantly above chance levels, suggesting they could distinguish between human and computer-generated responses. However, this was not due to the AI's inferior moral reasoning but potentially due to its perceived superiority. The study highlights concerns that people may uncritically accept potentially harmful moral guidance from AI. The results suggest that people's attributions toward AI moral discourse are distinct from those toward other AI technologies, and that the perceived moral intelligence of AI systems could significantly impact human welfare. The study also underscores the need for safeguards around generative language models in matters of morality. The findings have implications for understanding how ordinary people perceive and interact with AI in moral domains, and for developing future AI systems that better replicate human moral reasoning. The study was conducted with a representative sample of U.S. adults, and the results were analyzed using statistical methods to assess the quality of moral evaluations and the ability to identify their source. The study found that participants were able to distinguish between human and computer-generated moral evaluations, but this was not due to the AI's inferior moral reasoning but potentially due to its perceived superiority. The study also found that participants rated AI-generated moral evaluations as higher quality than human ones, suggesting that people may be more likely to accept AI moral guidance without critical evaluation. The study has implications for the development and use of AI in moral domains, and for the need for safeguards to prevent harmful interactions.