Coderev: AI-Powered Code Review from the Command Line

Introduction

I'm pleased to share Coderev, an AI-powered CLI tool that simplifies the code review process for developers. This post details how Coderev was built entirely using AI, without me writing any code manually. Coderev is built to use local git (not the GitHub API) to analyse pull requests and output a markdown report, making it easy to review code right from the command line. It supports multiple LLM providers through the litellm library (like OpenAI, Anthropic, Mistral, Gemini, Ollama, and others), ensuring flexibility and choice in which models are used.

Coderev is released as a Python package on PyPI: coderev on PyPI. It’s simple to install and use: just run pip install coderev to get started. The project is open source, licensed under Apache 2.0, and the code is available on GitHub.

Quick Example: Reviewing a Pull Request

Here’s a simple example of how to use Coderev to review a PR from a project:

# Step into your project directory
git checkout feature-branch

# Run Coderev to review changes from the current branch compared to main
coderev review

# Alternatively, you can explicitly specify the base branch:
coderev review --base-branch main

# Alternatively, if you are already on the main branch and want to review a feature branch:
coderev review feature-branch        

Coderev will analyse the differences between the current branch and the main branch and provide a detailed markdown-based code review. For more details, visit the package on PyPI.

Building Coderev with LLMs

One of the most exciting aspects of Coderev is how it came to life: it was built using LLMs like OpenAI's o1-preview and Anthropic's Claude 3.5 Sonnet. Here's a quick rundown of my process:

• Designing the CLI Interface: I started by working with OpenAI's o1-preview model to help me design the initial CLI interface for Coderev. This set the structure and flow for the tool.
• First Implementation: With the CLI defined, I moved to implementation. Using Claude 3.5 Sonnet, I created the first version of the code review feature, which provided practical markdown-based code reviews from git PRs.
• Manual Testing & Bug Fixes: initially tested Coderev manually, identifying and fixing basic bugs. I also went back and forth a few times with Claude 3.5 Sonnet and Cursor to fix issues and remove unnecessary functionality.
• Adding Tests: Once I was happy with the functionality, I added tests to solidify the main features, ensuring that future updates wouldn't break the core review capabilities. After that, I changed some error messages and reorganised parts of the code, but only after having tests in place.
• Iterative Feature Addition: With a stable base, I added more nuanced features, such as custom system prompts for LLMs and generalised instructions for how they should perform the reviews. I also implemented default values for the model selection and temperature, and added related tests.
• Renaming & UX Improvements: I later worked with o1-preview to refine parameter names and make the CLI more consistent and user-friendly.
• Release: Finally, I released the package first on TestPyPI for validation and feedback, and then on PyPI for general availability—a practice I learned through working with Claude 3.5 Sonnet.

Reflections & Lessons Learned

Reflecting on Coderev's development, here are some insights I gained:

• Post-Release Bug Discovery: After the initial release, I discovered an issue where saving the model temperature using coderev config set temperature 0.5 resulted in the value being saved as a string in the config file, causing a crash when used. I added a test to cover this scenario, fixed the bug, and released a new version. This experience highlighted that for stable project maintenance with LLMs, it's crucial to have sufficient tests to enable iterative improvements.
• Build First, Test Later: For the initial version, I focused on building the core interface and functionality, manually testing until I was satisfied. Once I was comfortable with the design and core behaviour, I added tests to ensure reliability. This worked well for keeping the early stages flexible.
• Big Picture Design First: Initially designing the entire CLI interface with an LLM like o1-preview turned out to be a great decision. It allowed me to see how the pieces fit together before diving into implementation. Had I started with tiny pieces of functionality, I might have ended up with a fragmented interface.
• Test-Driven Development for Additions: For smaller, additional features, adopting TDD was effective. It kept the focus on maintaining high code quality as new capabilities were added.

Coderev has been an experiment in leveraging AI models not just for functionality but for designing, coding, testing, and refining the tool. While it was an interesting journey, this experience made it clear that normal coding practices like TDD (Test Driven Development) and some form of QA are still essential, and even more important when using LLMs, as they tend to introduce new bugs into existing functionality when adding features and have a tendency to hallucinate details.