Boosting Code Development with AI Tools: Lessons from the Trenches

As a senior engineering manager, I’ve been on a mission to enhance our team’s productivity and code quality. Over the past year, we’ve integrated AI tools into our development process, and the results have been eye-opening. In this article, I’ll share my experience, insights, and a cautionary tale about the pitfalls of overreliance on AI.

The AI Workshop: A Turning Point

Our journey began with a workshop focused on writing better prompts for AI tools. We wanted to ensure accurate results and maximize the benefits of these tools. The workshop included hands-on practice, and we learned how to harness AI for various tasks, including:

1. Writing New Features: AI-generated snippets helped us kickstart new features, saving time and reducing boilerplate code.
2. Refactoring Existing Code: Converting code from one language to another became a breeze with AI assistance.
3. Unit Tests: AI-generated test cases improved our code coverage and reduced bugs.

Positive Changes Post-Workshop

Shortly after implementing AI tools, I noticed significant improvements:

1. Increased Capacity: Our team accomplished more within each sprint cycle.
2. Enhanced Code Quality: Better unit tests and improved coverage led to fewer bugs slipping through.
3. Faster Technology Adoption: Engineers adapted to new technologies more swiftly.

The Experience Gap

However, as time passed, I observed a divergence in how experienced engineers and less-experienced ones used AI tools. Here’s what I noticed:

1. Experienced Engineers: Used AI to speed up coding tasks while maintaining a deep understanding of the generated code. They struck a balance between automation and human judgment.
2. Less-Experienced Engineers: Leaned heavily on AI tools to write code for them. They often did so without fully comprehending the logic behind the generated code or considering whether it was a good fit for our existing codebase. Essentially, they treated AI as a “black box” solution, relying on its output without a deep understanding of what it produced. This approach had implications for code quality and consistency within our team.

The Rise of “Tab Tab Tab Developers”

Our team faced an unexpected challenge: the emergence of what I call “Tab Tab Tab developers”. These individuals would rapidly select suggested code snippets using the tab key, often without fully understanding the context or correctness of the solution. This led to several issues:

1. Inconsistent Code Styles: When reviewing pull requests, we noticed varying coding styles from the same engineer. AI-generated code lacked consistency.
2. Lack of Explanation: When asked to explain their code, some engineers struggled. They couldn’t articulate why they choose a particular approach or what problem it solved.
3. Long pull request cycles: Each pull request took as long as the initial development time (if not longer) until all the issues were fixed. This prolonged review process not only impacted the engineers submitting the PRs but also affected other team members who had to review the same PR repeatedly, ultimately affecting the team’s overall velocity.

Balancing AI and Human Judgment

To address this, we took a balanced approach:

1. Guided Usage: We encourage engineers to use AI as a guide rather than a crutch. Understand the code snippets it suggests and adapt them thoughtfully.
2. Code Ownership: Engineers must take ownership of their code. AI is a tool, not a replacement for critical thinking.
3. Code Reviews: Continue with rigorous code reviews to help maintain consistency and ensure everyone understands the rationale behind decisions.

Conclusions

Our journey taught us valuable lessons:

1. Balance Matters: Use AI judiciously alongside human judgment.
2. Learning Over Tools: Less-experienced engineers (or experienced engineers learning new technology) should focus on learning rather than relying solely on AI – Yes, I suggest to partially limit the usage of AI tools in this case, to make the engineers learn and practice before taking the easy way (them and the team will benefit in the long term).
3. Continuous Evolution: Keep engineers growing by exposing them to new capabilities beyond just tools.

Remember, AI is a powerful ally, but it’s no substitute for understanding and ownership. Let’s keep evolving!