VoicePilot introduces a framework for integrating Large Language Models (LLMs) as speech interfaces for physically assistive robots, focusing on human-centric design. The framework, developed through three iterative stages, includes components for prompt engineering, system rollout, and user evaluation. It was tested with 11 older adults at an independent living facility, resulting in design guidelines for LLM-based speech interfaces. The framework addresses key considerations such as customization, multi-step instructions, consistency, comparable time to caregivers, and social engagement. The study highlights the importance of user customization, sequential command execution, and efficient feedback mechanisms in creating intuitive and effective assistive interfaces. The findings emphasize the need for LLMs to handle both predefined and non-predefined commands, ensuring consistent performance and user satisfaction. The framework and guidelines aim to enhance the usability and effectiveness of LLM-based speech interfaces for physically assistive robots.VoicePilot introduces a framework for integrating Large Language Models (LLMs) as speech interfaces for physically assistive robots, focusing on human-centric design. The framework, developed through three iterative stages, includes components for prompt engineering, system rollout, and user evaluation. It was tested with 11 older adults at an independent living facility, resulting in design guidelines for LLM-based speech interfaces. The framework addresses key considerations such as customization, multi-step instructions, consistency, comparable time to caregivers, and social engagement. The study highlights the importance of user customization, sequential command execution, and efficient feedback mechanisms in creating intuitive and effective assistive interfaces. The findings emphasize the need for LLMs to handle both predefined and non-predefined commands, ensuring consistent performance and user satisfaction. The framework and guidelines aim to enhance the usability and effectiveness of LLM-based speech interfaces for physically assistive robots.