This paper explores how people prompt generative AI to create interactive virtual environments. Through an elicitation study with 22 participants, we identified four implicit expectations people have when prompting an intelligent agent to create virtual environments. These expectations include: (1) the agent should have embodied knowledge of the environment, (2) the agent should understand embodied prompts, (3) the agent should recall previous states of the scene and conversation, and (4) the agent should have common sense knowledge of objects in the scene. We also found that participants prompted differently when prompting in situ (within the VR environment) versus ex situ (viewing the environment from the outside). Based on these findings, we designed Ostaad, a conversational programming agent that allows non-programmers to design interactive VR experiences they inhabit. Ostaad enables users to use embodied voice prompts to create 3D scenes and interactions. Our work highlights the need for conversational programming agents to understand the state of the world, the user's relationship with that world, and be able to interact with the user to support the user's design vision. We also identify new opportunities and challenges for conversational programming agents that create VR environments.This paper explores how people prompt generative AI to create interactive virtual environments. Through an elicitation study with 22 participants, we identified four implicit expectations people have when prompting an intelligent agent to create virtual environments. These expectations include: (1) the agent should have embodied knowledge of the environment, (2) the agent should understand embodied prompts, (3) the agent should recall previous states of the scene and conversation, and (4) the agent should have common sense knowledge of objects in the scene. We also found that participants prompted differently when prompting in situ (within the VR environment) versus ex situ (viewing the environment from the outside). Based on these findings, we designed Ostaad, a conversational programming agent that allows non-programmers to design interactive VR experiences they inhabit. Ostaad enables users to use embodied voice prompts to create 3D scenes and interactions. Our work highlights the need for conversational programming agents to understand the state of the world, the user's relationship with that world, and be able to interact with the user to support the user's design vision. We also identify new opportunities and challenges for conversational programming agents that create VR environments.