Open Code Review

What it is

Open Code Review is a hybrid code review tool that combines deterministic rules with a large language model (LLM) agent. It first runs deterministic pipelines to check for issues like null pointer exceptions, thread-safety, cross-site scripting, and SQL injection. If these rules cannot express a finding, the LLM agent is layered on top to provide additional feedback.

The tool provides line-level comments instead of lengthy feedback, making it easier to review and defend to a security team. Each deterministic finding has a rule ID behind it, adding transparency to the review process.

Where it fits in a deployment stack

Code review tools like Open Code Review are designed to improve the quality and security of codebases. They typically sit alongside version control systems, continuous integration/continuous deployment (CI/CD) pipelines, and testing frameworks. By automating code review, these tools can help reduce the workload of human reviewers and catch issues earlier in the development process.

In a production system, code review tools often take over the task of checking for common errors and security vulnerabilities, freeing up human reviewers to focus on more complex issues. They can also provide feedback to developers on coding standards and best practices.

How to evaluate it

To evaluate Open Code Review, you need to consider its performance, accuracy, and integration with your existing development workflow. This involves testing the tool with your own codebase, injecting failures, and measuring its output. You should also inspect the tool's configuration options and APIs to ensure they meet your needs.

• Feed it a diff with known security vulnerabilities to see how it detects them
• Measure p95 latency of the retrieval step under 50 concurrent sessions
• Inspect the output of the deterministic pipelines to verify rule coverage
• Inject a failure in the LLM agent to see how it handles errors
• Test its compatibility with OpenAI and Anthropic compatible APIs
• Verify the tool's ability to provide line-level comments instead of essay-length feedback

What to verify before you commit

Before committing to Open Code Review, you need to verify several aspects of the tool, including its licence, data handling, operational cost, failure behaviour, and lock-in. You should check the tool's documentation, configuration options, and APIs to ensure they meet your requirements. It's also essential to evaluate the tool's support for your specific use case and development workflow.

• Check the licence terms to ensure they align with your organization's policies
• Verify how the tool handles sensitive data, such as code repositories and user credentials
• Estimate the operational cost of running the tool, including infrastructure and maintenance costs
• Test the tool's failure behaviour to ensure it doesn't disrupt your development workflow
• Check for any potential lock-in, such as proprietary APIs or data formats
• Verify the tool's support for your specific programming languages and frameworks