2024 | JIAYIN WANG, Tsinghua University, China; WEIZHI MA, Tsinghua University, China; PEIJIE SUN, Tsinghua University, China; MIN ZHANG, Tsinghua University, China; JIAN-YUN NIE, University of Montreal, Canada
This study addresses the critical need to understand user satisfaction with large language models (LLMs) by exploring four key aspects: comprehending user intents, scrutinizing user experiences, addressing major user concerns about current LLM services, and charting future research paths to enhance human-AI collaborations. The authors develop a taxonomy of 7 user intents in LLM interactions, grounded in real-world user interaction logs and human verification. A user survey with 411 anonymous responses uncovers 11 insights into user engagement with LLMs, including usage frequency, experiences across intents, and predominant concerns. Based on this empirical analysis, the study identifies 6 future research directions prioritizing the user perspective in LLM development. The findings highlight the importance of a user-centered approach to ensure that LLMs are not only technologically advanced but also resonate with the intricate realities of human interactions and real-world applications. The study advocates for a holistic approach to advancing human-LLM collaborations, emphasizing the need for fine-grained evaluations, user intent modeling, personalization, tool utilization, and trustworthiness.This study addresses the critical need to understand user satisfaction with large language models (LLMs) by exploring four key aspects: comprehending user intents, scrutinizing user experiences, addressing major user concerns about current LLM services, and charting future research paths to enhance human-AI collaborations. The authors develop a taxonomy of 7 user intents in LLM interactions, grounded in real-world user interaction logs and human verification. A user survey with 411 anonymous responses uncovers 11 insights into user engagement with LLMs, including usage frequency, experiences across intents, and predominant concerns. Based on this empirical analysis, the study identifies 6 future research directions prioritizing the user perspective in LLM development. The findings highlight the importance of a user-centered approach to ensure that LLMs are not only technologically advanced but also resonate with the intricate realities of human interactions and real-world applications. The study advocates for a holistic approach to advancing human-LLM collaborations, emphasizing the need for fine-grained evaluations, user intent modeling, personalization, tool utilization, and trustworthiness.