The first controlled study to examine GitHub Copilot’s impact on code quality shows that the quality of code authored with GitHub Copilot has increased functionality and improved readability, among other things. Take a deep dive into the results to see what else we found. https://lnkd.in/gmNEGzY5
Interesting results. The problem with this study, as well as the previous one which measured developer satisfaction, is methodological. In both cases the experiment included writing the type of code that Copilot is really good for: Mostly boilerplate, web-development oriented, Javascript or Python components. I know it may shock to some, but most development isn't about writing APIs in Python. 🤣 Considering I didn't spot anything about particular measures taken to ensure the external validity of the study, I hope to see a more varied approach to these experiments, with different languages and domains.
I've heard a lot of mixed things about the impact of Copilot. I've been using it for a few weeks and I definitely appreciate the way it's able to fill in small functions or explain bugs, but when I try to push it to its limits by asking it to develop more complex programs the results tend to implement poor coding practices and require a TON of refactoring and editing to make it functional. All that to say that I do really appreciate how it speeds up simple actions, but I'm not worried about software developers going extinct.
Stefan A.
1d
In addition to Dimitrios Platis comments on the methodological issues, I will call out the elephant in the room and say that these measures, even when taken at face value, are thoroughly underwhelming and in no way close to motivating any larger efforts to adopt it, given the potential of landing on the wrong side of the median and/or introducing non-negible amounts of slop into your codebase.
Mark S.
1d
It would be interesting to see the reproducibility of this.
Ken Eucker
1d
Oh yeah, all of those jokes it inserts are just such high quality! 👎
Conseiller principal chez CGI
1dLet's all remember that GitHub Copilot is a coding assistant. What we are saying here is this tool helps you improve readability and quality of the code YOU write. If the tool generates gibberish code and you accept it, this is on you. It's still is your code. Even if the tool suggestions were wrong most of the time, which is not the case, the few times it actually suggests great code is bonus for you. This is what the tool is about, helping you.