This paper introduces Codex, a GPT language model fine-tuned on publicly available code from GitHub, and evaluates its Python code-writing capabilities. Codex powers GitHub Copilot and is tested on the HumanEval dataset, which measures functional correctness for synthesizing programs from docstrings. Codex solves 28.8% of the problems on HumanEval, while GPT-3 solves 0% and GPT-J solves 11.4%. Repeated sampling from Codex is surprisingly effective, solving 70.2% of the problems with 100 samples per problem. The model's limitations include difficulty with docstrings describing long chains of operations and binding operations to variables. The paper evaluates the performance of Codex on a dataset of 164 programming problems with unit tests, assessing language comprehension, algorithms, and simple mathematics. Codex-12B solves 28.8% of the problems with a single sample, while Codex-S, fine-tuned on correctly implemented functions, solves 37.7% of the problems. Generating 100 samples per problem and selecting the sample with the highest mean log-probability solves 44.5% of the problems, while selecting the sample that passes unit tests solves 77.5% of the problems. The paper discusses the evaluation framework, including the pass@k metric, which measures the fraction of problems solved by any of the generated samples. The framework includes a sandbox environment for safely executing model-generated code. Codex is fine-tuned on code to produce Codex, which performs well on the HumanEval dataset. The paper also compares Codex with other models like GPT-Neo and Tabnine, finding that Codex outperforms them in terms of pass@k metrics. The paper discusses the limitations of Codex, including its difficulty with long chains of operations and binding operations to variables. It also discusses the broader impacts of deploying powerful code generation technologies, including safety, security, and economic considerations. The paper concludes that while Codex is effective at generating code, it has limitations and potential risks that need to be addressed.This paper introduces Codex, a GPT language model fine-tuned on publicly available code from GitHub, and evaluates its Python code-writing capabilities. Codex powers GitHub Copilot and is tested on the HumanEval dataset, which measures functional correctness for synthesizing programs from docstrings. Codex solves 28.8% of the problems on HumanEval, while GPT-3 solves 0% and GPT-J solves 11.4%. Repeated sampling from Codex is surprisingly effective, solving 70.2% of the problems with 100 samples per problem. The model's limitations include difficulty with docstrings describing long chains of operations and binding operations to variables. The paper evaluates the performance of Codex on a dataset of 164 programming problems with unit tests, assessing language comprehension, algorithms, and simple mathematics. Codex-12B solves 28.8% of the problems with a single sample, while Codex-S, fine-tuned on correctly implemented functions, solves 37.7% of the problems. Generating 100 samples per problem and selecting the sample with the highest mean log-probability solves 44.5% of the problems, while selecting the sample that passes unit tests solves 77.5% of the problems. The paper discusses the evaluation framework, including the pass@k metric, which measures the fraction of problems solved by any of the generated samples. The framework includes a sandbox environment for safely executing model-generated code. Codex is fine-tuned on code to produce Codex, which performs well on the HumanEval dataset. The paper also compares Codex with other models like GPT-Neo and Tabnine, finding that Codex outperforms them in terms of pass@k metrics. The paper discusses the limitations of Codex, including its difficulty with long chains of operations and binding operations to variables. It also discusses the broader impacts of deploying powerful code generation technologies, including safety, security, and economic considerations. The paper concludes that while Codex is effective at generating code, it has limitations and potential risks that need to be addressed.