Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun|July 7, 2021arXiv

Key Takeaway

Fine-tuning language models on code produces functional programming assistants — Codex proved LLMs could write real, working software.

Summary

Introduces Codex, a GPT model fine-tuned on code from GitHub, and the HumanEval benchmark for measuring functional code generation. Codex powers GitHub Copilot and solves 28.8% of HumanEval problems.

evaluation applications