Hard-Coded: Closing the AI-Driven Code Quality Gap

Why more code does not mean better code?

While AI coding assistants can deliver major productivity gains , many of these tools currently equate productivity with generating new code faster . Unfortunately, there is early evidence to suggest that, if not used judiciously, coding assistants can actually erode the quality of codebases .

Code analytics company GitClear analyzed 153 million changed lines of code authored between January 2020 and December 2023 and noted distinct changes after the launch of GitHub Copilot. Lines of code that were moved (code cut and pasted within the same file or to a new file) dropped by 17% in 2023. Code that has been moved often indicates refactoring.

Lines of churned code – code which is committed but reverted, removed, or updated within the following two weeks – increased by 39% in 2023. Code churn may indicate that incomplete and erroneous code was committed. Another 2023 research paper concluded that many GitHub Copilot solutions for the HumanEval programming problem dataset contained code smells, estimating that 172 minutes of work would be needed to tackle the resulting technical debt.

Why code quality matters

Quality code is readable, reliable, maintainable, efficient, and secure. Code quality has a major impact on both the developer experience and development velocity since low-quality code contains more bugs, and those bugs take longer to fix . Making changes in a low-quality codebase is difficult and dangerous, so shipping new features becomes stressful for developers and unreliable for users. Last but not least, the technical debt caused by low-quality code compounds over time, especially in the large and complex codebases of enterprises.?

What causes code quality problems???

The lure of the new: AI coding assistants often focus on adding new code, but to maintain codebase health, developers must also bug-fix and refactor existing code. JetBrains researchers found that quality improvement – finding problems, refactoring code, and adjusting code to follow best practices – is one of the main usage patterns of JetBrains AI Assistant users. Refactoring is an inherently more challenging task than writing new code. According to a paper by JetBrains researchers , 76% of refactoring suggestions from GPT-3.5 were unusable.?

Automation bias: Most coding assistants are incentivized to make suggestions that will be accepted and not to generate code that is easy to maintain. With the advent of AI Assistants, developers must spend time evaluating code suggestions and are at risk of placing too much trust in those suggestions or simply not putting enough effort into reviewing them before accepting. Less experienced developers are also more likely to accept AI suggestions.

Lack of a holistic view: While the quality of individual AI code suggestions varies between products and use cases, keeping the quality of suggestions high is not enough to solve the problem. The real challenge is how to integrate AI-generated and human-written code in a way that preserves or improves the health of codebases that may be in use for decades.?

Closing the AI-driven code quality gap?

We are still in the early days of the AI revolution in software development, and some unexpected consequences were, well, to be expected. One way to address an AI-driven code quality gap could be to combine traditional code quality best practices like testing, refactoring, static analysis, and code reviews with novel AI-driven solutions. For example, when the JetBrains research team used static analysis to enhance and rank refactoring suggestions generated by LLMs, 81% of surveyed developers found the suggestions useful.?

How will the tidal wave of AI-generated code change codebases? Can we increase code quality in parallel with code quantity? Please share your thoughts in the comments.??