This study explores the impact of proactive conversational assistants on usability evaluation, focusing on the timing of automatic suggestions. The research aims to enhance UX evaluation through human-AI collaboration, particularly by leveraging natural language processing. The study involved 24 UX evaluators using a hybrid Wizard-of-Oz approach with ChatGPT to generate problem suggestions and a human moderator to respond to questions. The timing of suggestions—before, in sync with, or after potential usability problems—was investigated. While timing did not significantly affect analytic performance, suggestions appearing after potential problems were preferred, enhancing trust and efficiency. Participants found the suggestions useful but collectively identified more problems than the AI, highlighting the irreplaceable role of human expertise. The findings offer insights into future human-AI collaborative tools for UX evaluation, emphasizing the importance of efficient and timely AI assistance.This study explores the impact of proactive conversational assistants on usability evaluation, focusing on the timing of automatic suggestions. The research aims to enhance UX evaluation through human-AI collaboration, particularly by leveraging natural language processing. The study involved 24 UX evaluators using a hybrid Wizard-of-Oz approach with ChatGPT to generate problem suggestions and a human moderator to respond to questions. The timing of suggestions—before, in sync with, or after potential usability problems—was investigated. While timing did not significantly affect analytic performance, suggestions appearing after potential problems were preferred, enhancing trust and efficiency. Participants found the suggestions useful but collectively identified more problems than the AI, highlighting the irreplaceable role of human expertise. The findings offer insights into future human-AI collaborative tools for UX evaluation, emphasizing the importance of efficient and timely AI assistance.