An analysis of more than 300 open source repositories published today by Ox Security, a provider of an application security posture management (ASPM) platform, identifies 10 systematic behaviors of artificial intelligence (AI) coding tools that ignore established best practices for software engineering.
Eyal Paz, vice president of research at OX Security, said that while policies can be put in place to ensure best practices are followed, the report makes it clear DevOps teams will need to validate the output of the code generated because of anti-patterns that AI coding tools are not capable of consistently avoiding.
Specifically, systematic behaviors that run afoul for best software engineering practices include:
Comments Everywhere: Excessive inline commenting dramatically increases computational burden and makes code harder to check
By-The-Book Fixation: Rigidly follows conventional rules, missing opportunities for more innovative, improved solutions
Over-Specification: Creates hyper-specific, single-use solutions instead of generalizable, reusable components
Avoidance of Refactors: Generates functional code for immediate prompts but never refactors or architecturally improves existing code
Bugs Déjà-Vu: Violates code reuse principles, causing identical bugs to recur throughout codebases, requiring redundant fixes
Worked on My Machine Syndrome: Lacks deployment environment awareness, generating code that runs locally but fails in production
Return of Monoliths: Defaults to tightly-coupled monolithic architectures, reversing decade-long progress toward microservices
Fake Test Coverage: Inflates coverage metrics with meaningless tests rather than validating logic
Vanilla Style: Reimplements from scratch instead of using established libraries, SDKs, or proven solutions
Phantom Bugs: Over-engineers for improbable edge cases, causing performance degradation and resource waste
The degree to which AI coding tools are susceptible to these issues tends to vary. Issues such as Comments Everywhere, By-The-Book Fixation, Over-Specification and Avoidance of Refactors can be found in 80% of the code analyzed, according to the report. Phantom bugs are, in comparison, an issue that arises in 20 to 30% of the code generated.
In effect, AI coding tools are providing organizations with an army of junior developers that lack judgement and context, said Paz. In fact, it’s not until a senior developer that understands the nuance of writing code adheres to best practices that organizations really derive value from investments in AI coding tools that generate more code faster, noted Paz.
The challenge is that as more non-technical users and junior human developers are using AI coding tools to generate code, that code still needs to be reviewed by senior developers to ensure best practices are followed, he added. In fact, unlike human developers, an AI coding tool is never going to question the validity of the task being assigned and will in many instances continue to, for example, generate false unit tests unless specifically told to desist, noted Paz
On the plus side, however, the report surprisingly finds that AI coding tools are not generating any more security vulnerabilities than human developers. Instead, DevOps teams are simply being overwhelmed by vulnerabilities as the total amount of code being generated exponentially increases, said Paz.
The only way to effectively respond to that onslaught is to build security instruction sets directly into AI coding processes rather than continuing to rely on security reviews that can’t be conducted at the level of scale required, he noted.
Each DevOps team will, depending on the use case, need to determine for itself to what degree to rely on the output generated by AI coding tools, but the one thing that is clear is that many of these tools if left ungoverned may turn out to be far too much of what otherwise might be a potentially good thing.
