MARCH 2024 | VOL. 67 | NO. 3 | BY ALBERT ZIEGLER, EIRINI KALLIAMVAKOU, X. ALICE LI, ANDREW RICE, DEVON RIFKIN, SHAWN SIMISTER, GANESH SITTAMPALAM, AND EDWARD AFTANDILIAN
The article investigates the impact of GitHub Copilot on developer productivity, focusing on the relationship between usage measurements and perceived productivity. The study collected data from 2,631 developers using GitHub Copilot and matched it with usage metrics from the IDE. Key findings include:
1. **Perceived Productivity and Usage Metrics**: The acceptance rate of suggestions (the fraction of shown suggestions accepted) is a strong predictor of perceived productivity, outperforming other detailed measures like persistence and code contributions.
2. **Demographic and Experience Factors**: The acceptance rate varies significantly among different developer groups, including those with varying levels of experience and proficiency in the programming language used with Copilot. Junior developers reported higher productivity gains and accepted more suggestions.
3. **Temporal Patterns**: The acceptance rate shows distinct patterns over time, with higher rates during weekends and non-working hours, and lower rates during typical working hours.
4. **Conversational Framework**: The interaction with Copilot is hypothesized to be similar to natural conversations, suggesting that the value of suggestions lies in their usefulness as templates rather than their correctness.
5. **Conclusion**: The study concludes that while correctness is important, the driving factor for productivity improvements is the usefulness of suggestions as starting points for further development. The findings highlight the need for a broader perspective on tool evaluation, considering both objective metrics and self-reported data.The article investigates the impact of GitHub Copilot on developer productivity, focusing on the relationship between usage measurements and perceived productivity. The study collected data from 2,631 developers using GitHub Copilot and matched it with usage metrics from the IDE. Key findings include:
1. **Perceived Productivity and Usage Metrics**: The acceptance rate of suggestions (the fraction of shown suggestions accepted) is a strong predictor of perceived productivity, outperforming other detailed measures like persistence and code contributions.
2. **Demographic and Experience Factors**: The acceptance rate varies significantly among different developer groups, including those with varying levels of experience and proficiency in the programming language used with Copilot. Junior developers reported higher productivity gains and accepted more suggestions.
3. **Temporal Patterns**: The acceptance rate shows distinct patterns over time, with higher rates during weekends and non-working hours, and lower rates during typical working hours.
4. **Conversational Framework**: The interaction with Copilot is hypothesized to be similar to natural conversations, suggesting that the value of suggestions lies in their usefulness as templates rather than their correctness.
5. **Conclusion**: The study concludes that while correctness is important, the driving factor for productivity improvements is the usefulness of suggestions as starting points for further development. The findings highlight the need for a broader perspective on tool evaluation, considering both objective metrics and self-reported data.