May 11–16, 2024 | Joel Wester, Tim Schrills, Henning Pohl, Niels van Berkel
The paper investigates the impact of different denial styles in large language models (LLMs) when they cannot or should not fulfill a user's request. The study evaluates four denial styles—baseline, factual, diverting, and opinionated—across two studies, focusing on technical limitations and social policy restrictions. The results show that diverting denials are perceived as less frustrating and more useful, appropriate, and relevant compared to baseline denials. Factual denials are rated lower than diverting denials on all measures except frustration, while opinionated denials are perceived more positively than baseline denials. The qualitative results highlight that users appreciate diverting denials for providing valuable information even when the original request cannot be fulfilled. The paper provides design recommendations for LLM denials, such as providing clear reasons for denials, using diverting denials to redirect users, and incorporating corrective denials to guide users to improve their requests.The paper investigates the impact of different denial styles in large language models (LLMs) when they cannot or should not fulfill a user's request. The study evaluates four denial styles—baseline, factual, diverting, and opinionated—across two studies, focusing on technical limitations and social policy restrictions. The results show that diverting denials are perceived as less frustrating and more useful, appropriate, and relevant compared to baseline denials. Factual denials are rated lower than diverting denials on all measures except frustration, while opinionated denials are perceived more positively than baseline denials. The qualitative results highlight that users appreciate diverting denials for providing valuable information even when the original request cannot be fulfilled. The paper provides design recommendations for LLM denials, such as providing clear reasons for denials, using diverting denials to redirect users, and incorporating corrective denials to guide users to improve their requests.