Since its open-source release two years ago, GitHub Copilot has revolutionized software development. It enables developers to write code 55% faster, with 85% of users feeling more confident in their code and 88% experiencing better workflow focus.
Yet, a critical question arises: does GitHub Copilot improve the objective quality of code?
Study on Code Quality with GitHub Copilot
To address this, GitHub conducted a randomized controlled trial examining various aspects of code quality—functionality, readability, reliability, maintainability, and conciseness. The study involved 202 experienced developers, split evenly into two groups: one with access to GitHub Copilot and another instructed to avoid AI tools. Participants were tasked with creating API endpoints for a web server, with their code evaluated via unit tests and expert peer reviews.
Key findings from the study include:
- Enhanced functionality: Developers using GitHub Copilot were 56% more likely to pass all 10 unit tests, demonstrating its ability to produce more functional code.
- Improved readability: Code written with Copilot had fewer readability issues, with developers producing 13.6% more readable lines of code on average.
- Overall quality gains: Readability improved by 3.62%, reliability by 2.94%, maintainability by 2.47%, and conciseness by 4.16%. All improvements were statistically significant.
- Higher approval rates: Code created with Copilot received 5% more approvals, facilitating faster merges, bug fixes, and feature deployments.
The findings suggest that GitHub Copilot significantly enhances the quality and functionality of code while accelerating development workflows, concludes NIXsolutions. These benefits align with broader trends reported in the DORA 2024 report. As GitHub Copilot continues to evolve, we’ll keep you updated on further developments and integrations.