This paper introduces BOLT, a computational framework designed to systematically assess and compare the conversational behavior of large language models (LLMs) used as therapists in mental health support. The authors aim to understand how LLMs respond to clients and their potential capabilities and limitations in high-risk settings like mental health. BOLT uses psychotherapy techniques to quantify and classify the behavior of LLMs, comparing them against high- and low-quality human therapy sessions. The study finds that while LLMs often exhibit behaviors more typical of low-quality therapy, such as offering excessive problem-solving advice, they also show more reflection on client emotions and strengths compared to human therapists. The analysis suggests that LLMs are not yet fully consistent with high-quality care and require further research to improve their efficacy. The paper also explores methods to modulate LLM behavior to align more closely with desirable therapeutic practices.This paper introduces BOLT, a computational framework designed to systematically assess and compare the conversational behavior of large language models (LLMs) used as therapists in mental health support. The authors aim to understand how LLMs respond to clients and their potential capabilities and limitations in high-risk settings like mental health. BOLT uses psychotherapy techniques to quantify and classify the behavior of LLMs, comparing them against high- and low-quality human therapy sessions. The study finds that while LLMs often exhibit behaviors more typical of low-quality therapy, such as offering excessive problem-solving advice, they also show more reflection on client emotions and strengths compared to human therapists. The analysis suggests that LLMs are not yet fully consistent with high-quality care and require further research to improve their efficacy. The paper also explores methods to modulate LLM behavior to align more closely with desirable therapeutic practices.