When Are Combinations of Humans and AI Useful?

When Are Combinations of Humans and AI Useful?

9 May 2024 | Michelle Vaccaro¹², Abdullah Almaatouq¹, and Thomas Malone¹
Human-AI collaboration can be beneficial or detrimental depending on the task type. A meta-analysis of 106 experiments with 370 effect sizes found that human-AI systems generally performed worse than either humans or AI alone. However, they performed better than humans alone in most cases. The study identified that decision tasks led to performance losses, while creation tasks led to performance gains. When humans outperformed AI alone, human-AI systems showed gains, but when AI outperformed humans, performance losses occurred. Task type and relative performance of humans and AI significantly influenced synergy. The study also found that the type of AI and experimental design affected synergy. While strong synergy was rare, weak synergy was common. The findings suggest that human-AI systems are most effective in creation tasks and when humans outperform AI. Future research should focus on improving human-AI systems through better task design, process innovation, and robust evaluation criteria. The study highlights the need for standardized metrics and commensurability criteria to facilitate systematic comparisons and progress in human-AI collaboration.Human-AI collaboration can be beneficial or detrimental depending on the task type. A meta-analysis of 106 experiments with 370 effect sizes found that human-AI systems generally performed worse than either humans or AI alone. However, they performed better than humans alone in most cases. The study identified that decision tasks led to performance losses, while creation tasks led to performance gains. When humans outperformed AI alone, human-AI systems showed gains, but when AI outperformed humans, performance losses occurred. Task type and relative performance of humans and AI significantly influenced synergy. The study also found that the type of AI and experimental design affected synergy. While strong synergy was rare, weak synergy was common. The findings suggest that human-AI systems are most effective in creation tasks and when humans outperform AI. Future research should focus on improving human-AI systems through better task design, process innovation, and robust evaluation criteria. The study highlights the need for standardized metrics and commensurability criteria to facilitate systematic comparisons and progress in human-AI collaboration.
Reach us at info@study.space
[slides and audio] When combinations of humans and AI are useful%3A A systematic review and meta-analysis