The Copilot Evaluation Harness is a comprehensive framework for evaluating the performance of Large Language Models (LLMs) in software development scenarios within Integrated Development Environments (IDEs). This paper introduces the Copilot evaluation harness, which provides a set of data and tools for assessing LLM-guided programming interactions across various programming tasks and languages. The framework includes metrics designed to evaluate the effectiveness, accuracy, and efficiency of LLMs in real-world development scenarios. The evaluation harness covers five major software development scenarios: documentation generation from code, bug-fixing, code generation from natural language, test case generation for code, and workspace understanding and query resolution. These scenarios encompass a wide range of developer tasks, including code generation, documentation creation, test case development, bug detection, and understanding of the codebase. The metrics in the evaluation harness are designed to assess the performance of LLMs within a given IDE and its respective parameter space. The framework allows for any IDE to be plugged in and evaluated using these metrics. The evaluation harness is designed to provide a system for tuning the IDE parameter space to attain superior LLM-integration outcomes. The paper discusses the results of evaluating three prominent LLMs: OpenAI's GPT-3.5, GPT-4, and CodeLlama, on the documentation generation and bug fixing scenarios. The evaluation results show that GPT-4 generally outperforms GPT-3.5 and CodeLlama in documentation generation, while GPT-3.5 outperforms GPT-4 and CodeLlama in bug fixing. The evaluation also highlights the importance of providing additional context to the models to ensure accurate and effective bug fixes. The Copilot Evaluation Harness provides a robust and comprehensive evaluation system for LLMs in software development. It allows for the assessment of LLMs in various programming scenarios and languages, and provides insights into how to better integrate LLMs with IDEs. The evaluation harness is designed to help developers and researchers understand the capabilities and limitations of LLMs in software development, and to improve the integration of LLMs with IDEs.The Copilot Evaluation Harness is a comprehensive framework for evaluating the performance of Large Language Models (LLMs) in software development scenarios within Integrated Development Environments (IDEs). This paper introduces the Copilot evaluation harness, which provides a set of data and tools for assessing LLM-guided programming interactions across various programming tasks and languages. The framework includes metrics designed to evaluate the effectiveness, accuracy, and efficiency of LLMs in real-world development scenarios. The evaluation harness covers five major software development scenarios: documentation generation from code, bug-fixing, code generation from natural language, test case generation for code, and workspace understanding and query resolution. These scenarios encompass a wide range of developer tasks, including code generation, documentation creation, test case development, bug detection, and understanding of the codebase. The metrics in the evaluation harness are designed to assess the performance of LLMs within a given IDE and its respective parameter space. The framework allows for any IDE to be plugged in and evaluated using these metrics. The evaluation harness is designed to provide a system for tuning the IDE parameter space to attain superior LLM-integration outcomes. The paper discusses the results of evaluating three prominent LLMs: OpenAI's GPT-3.5, GPT-4, and CodeLlama, on the documentation generation and bug fixing scenarios. The evaluation results show that GPT-4 generally outperforms GPT-3.5 and CodeLlama in documentation generation, while GPT-3.5 outperforms GPT-4 and CodeLlama in bug fixing. The evaluation also highlights the importance of providing additional context to the models to ensure accurate and effective bug fixes. The Copilot Evaluation Harness provides a robust and comprehensive evaluation system for LLMs in software development. It allows for the assessment of LLMs in various programming scenarios and languages, and provides insights into how to better integrate LLMs with IDEs. The evaluation harness is designed to help developers and researchers understand the capabilities and limitations of LLMs in software development, and to improve the integration of LLMs with IDEs.