The paper explores how people prompt generative AI to create interactive VR scenes, focusing on the implicit expectations and behaviors of users when interacting with programming agents. Through a Wizard of Oz elicitation study involving 22 participants, the authors identified four key expectations: (1) embodied knowledge of the environment, (2) understanding of embodied prompts, (3) conversational memory, and (4) common sense knowledge about object behavior. Participants demonstrated a preference for in situ prompting, where they could combine verbal instructions with physical gestures, and showed a need for clear and concise prompts. The study also revealed that users expected agents to understand their presence and spatial relationships within the scene, and to recall past interactions and states. Based on these findings, the authors developed Ostaad, a conversational programming agent designed to facilitate the creation of interactive VR experiences by non-programmers. The paper highlights the importance of considering these user expectations in the design of future conversational programming agents for VR environments.The paper explores how people prompt generative AI to create interactive VR scenes, focusing on the implicit expectations and behaviors of users when interacting with programming agents. Through a Wizard of Oz elicitation study involving 22 participants, the authors identified four key expectations: (1) embodied knowledge of the environment, (2) understanding of embodied prompts, (3) conversational memory, and (4) common sense knowledge about object behavior. Participants demonstrated a preference for in situ prompting, where they could combine verbal instructions with physical gestures, and showed a need for clear and concise prompts. The study also revealed that users expected agents to understand their presence and spatial relationships within the scene, and to recall past interactions and states. Based on these findings, the authors developed Ostaad, a conversational programming agent designed to facilitate the creation of interactive VR experiences by non-programmers. The paper highlights the importance of considering these user expectations in the design of future conversational programming agents for VR environments.