Skip to content
Mardi Gras Developers
Menu+
← All articles

· 7 min read

We asked an AI code reviewer what it checks in Kotlin. Security was missing

Open Code Review 1.13.11 reviews Kotlin with a 683-word checklist that never mentions security, while a SQL file gets four security questions. One config entry fixes it.

Written with the help of AI, reviewed and added to by Kevin Bos.

The only thing Alibaba’s AI code reviewer is told about building strings in Kotlin is to use string templates. Its Kotlin checklist is 683 words long and ends like this:

$ ocr rules check src/main/kotlin/shop/OrderService.kt
Pattern: **/*.{kt,kts}
[…]
- **Immutability**: Prefer `val` over `var`.
- **String handling**: Use string templates (`"Value: $value"`) instead of concatenation.

Nowhere in those 683 words does it say that a template inside a SQL string is how an injection starts. There is no line about input validation, access checks or secrets either. In the same project, the Dockerfile is reviewed with the question “Is permission validation complete?”.

0 of 683words about security in the checklist an AI reviewer gets for a Kotlin fileOur run in the sandbox, Open Code Review 1.13.11, 17 November 2026

What that means in a pull request

Take two versions of one function. This is our own illustration, not output of the tool:

// Ticks every box on the built-in Kotlin checklist: no !!, no var, a string template.
fun findOrders(status: String) =
    jdbc.queryForList("SELECT * FROM orders WHERE status = '$status'")

// What a security line would ask for: a bound parameter.
fun findOrders(status: String) =
    jdbc.queryForList("SELECT * FROM orders WHERE status = ?", status)

The first version puts request input straight into the query. We did not run a model on it, so we do not know whether the reviewer would flag it anyway. What we do know is that nothing in its instructions asks it to, and that the one instruction about strings points the other way.

Why we looked

Open Code Review gained 951 stars on 17 November and 3,056 that week, according to GitHub’s trending list. It is a Go command-line tool under the Apache-2.0 licence, and its README says it started as Alibaba’s internal reviewer. The repository describes it as having a “built-in multi-language ruleset (NPE, thread-safety, XSS, SQL injection)”.

Half of the tool is ordinary code. Before any model is called, it decides which changed files get reviewed and which rule text goes with each file. That half runs without an API key, so it can be measured. We build Kotlin and Java backends, so we wanted to know what a team like ours gets on day one.

What we did

We downloaded the 1.13.11 release binary (59 MB, checksum verified) into a container and created a 17-file sample project: a small Kotlin and Java order service with tests, a MyBatis mapper, a SQL migration, a Dockerfile, Terraform, a Flutter screen, a Vue page and a GitHub workflow. Then we asked the tool three things with ocr delegate preview and ocr rules check: which files it would review, which checklist each file gets, and what a project rule file changes. No model was called. The script and the full output are in this site’s repository.

Two files, two checklists

The reviewer is asked about OrderService.kt V2__add_status.sql
Checklist Kotlin rule, 683 words Fallback rule, 101 words
SQL injection and XSS No Yes
Handling of sensitive information No Yes
Permission validation No Yes
val instead of var Yes No
GlobalScope in coroutines Yes No
@JvmStatic for Java callers Yes No

The Kotlin checklist is a good one for what it covers: null safety, coroutines, resource handling with use. It reads like a style and correctness guide, and that is what the model receives. Java is in the same position with 461 words on typos, dead code, logic errors, performance and thread safety.

For every checklist we counted the lines that contain one of six terms: injection, XSS, security, vulnerab, sensitive, permission.

Go
7 lines
PHP
7 lines
TypeScript and JavaScript
6 lines
MyBatis mapper XML
5 lines
Terraform
5 lines
No language rule (.sql, .dart, .vue, Dockerfile)
4 lines
Python
3 lines
Java
0 lines
Kotlin
0 lines
A SQL migration gets four security lines in its checklist. A Java or Kotlin file gets none. Our run in the sandbox, Open Code Review 1.13.11, 17 November 2026

The files the tool knows least about get the fallback checklist, and that one has a security section:

$ ocr rules check src/main/resources/db/migration/V2__add_status.sql
Pattern: default
[…]
#### Security
Are there security vulnerabilities such as SQL injection or XSS?
Is sensitive information handled correctly?
Is permission validation complete?

Across all 66 built-in rule files, 23 contain none of the six terms. That number is a keyword count and it undercounts: the C rule has a section called “Buffer Overflow Protection” and matches nothing. For Java and Kotlin we read the full text, so those two zeros are real.

Why a language rule removes the security questions

The mapping lives in system_rules.json: 67 glob patterns, each pointing at one rule file, plus default.md. The lookup in system_rules.go walks the patterns in order and returns the first match. The default is used only when nothing matches. A language rule therefore replaces the default instead of extending it, and a language rule only asks about security if its author wrote that in. The Go, PHP and TypeScript rules have such lines. The Java and Kotlin rules do not.

Changed filefrom the git diffFile filtertests, lockfiles outRule lookupfirst match winsLanguage rulejava.md: no securityDefault ruleasks about injectionYour rule.jsonmerged only on requestModel promptReview Checklist
The first matching rule wins, so a Java file never sees the fallback checklist that asks about injection.

The resolved text is pasted into the prompt under the heading “Review Checklist”. The system prompt of the main review task in that release does not contain the word security either. A model can still report an injection it happens to notice, and the planning prompt defines high severity as “May cause security vulnerabilities, data loss, system crashes, or critical functional failures”. But the README’s own argument for the design is that rule matching keeps “the model’s attention sharply focused”, and it states that recall is lower than a general-purpose agent’s, “a deliberate trade-off favoring precision over noise”. A checklist that points the model at spelling and dead code points it away from everything else.

What else the preview showed

$ ocr delegate preview --commit HEAD
[…]
~~- `docs/README.md` [added] +2/-0 (excluded: unsupported_ext)~~
[…]
~~- `package-lock.json` [added] +1/-0 (excluded: default_path)~~
[…]
~~- `src/test/java/shop/PriceCalculatorTest.java` [added] +7/-0 (excluded: default_path)~~
~~- `src/test/kotlin/shop/OrderServiceTest.kt` [added] +12/-0 (excluded: default_path)~~

Thirteen of the 17 files would be reviewed, and the answer came in 0.038 seconds. Leaving out the lockfile is sensible. Leaving out both test files is a choice you should know about: a pull request that removes an assertion is not looked at. The other surprise is build.gradle.kts. It matches the Kotlin pattern, so the build script is reviewed with 683 words about coroutines and data classes, while a Groovy build.gradle gets a 30-word rule about dependencies.

The configuration we tested

The project’s documentation describes the way out, and it works. Put your own checklist in the repository and merge it into every file’s rule:

{
  "rules": [
    { "path": "**/*", "rule": ".opencodereview/security.md", "merge_system_rule": true }
  ],
  "include": ["**/src/test/**"]
}

Our security.md held four lines: bound parameters in SQL, validation of input that reaches a path, a command or a redirect, an access check on every endpoint, and no secrets or personal data in logs. With this file in place the Java checklist grows from 461 to 530 words, the built-in part stays, and the preview reports 15 of 17 files because the tests are back:

src/main/java/shop/PriceCalculator.java | **/* | 530 | 1
src/main/kotlin/shop/OrderService.kt | **/* | 752 | 1
src/test/kotlin/shop/OrderServiceTest.kt | **/* | 752 | 1

The flag matters. It defaults to false, and without it a matching entry replaces the built-in rule. We ran that as well: the same entry without merge_system_rule cut the Java checklist from 461 words to 60. You would have added security and silently removed the thread-safety and logic checks.

What we would do

We would try this tool on a Kotlin or Java codebase only with a project rule file from the first day, and we would not count its default review as a security review. Out of the box, for JVM code, it is a reviewer for correctness, performance and concurrency. That is useful, and it is not what the repository description suggests.

Two things would change this view: a security section in java.md and kotlin.md, or published recall figures per language and per category of defect.

Limits of this test

We measured the instructions, not the findings. No model was called, so we cannot say how many security defects the default setup misses in practice. One version (1.13.11, commit 0f5a284), one sample project of 17 files we made up, and a keyword count of six terms over the rule texts. The benchmark numbers in the README are the vendor’s own and we did not reproduce them.

Sources

  1. alibaba/open-code-review
  2. system_rules.json at v1.13.11: which rule file each path gets
  3. system_rules.go at v1.13.11: first match wins, merge_system_rule
  4. kotlin.md at v1.13.11: the built-in Kotlin checklist
  5. java.md at v1.13.11: the built-in Java checklist
  6. Review Rules, the project's documentation
  7. GitHub Trending, daily, 17 November 2026
  • AI security
  • code review
  • Kotlin
  • Java

Lees dit artikel in het Nederlands

Does this apply to you too?

In a 30-minute intro call Kevin or Pascal looks at what this means for your systems.

Book an intro call