# We asked an AI code reviewer what it checks in Kotlin. Security was missing

> Open Code Review 1.13.11 reviews Kotlin with a 683-word checklist that never mentions security, while a SQL file gets four security questions. One config entry fixes it.

2026-11-17 · Mardi Gras Developers

The only thing Alibaba's AI code reviewer is told about building strings in Kotlin is to use string templates. Its Kotlin checklist is 683 words long and ends like this:

```console
$ ocr rules check src/main/kotlin/shop/OrderService.kt
Pattern: **/*.{kt,kts}
[…]
- **Immutability**: Prefer `val` over `var`.
- **String handling**: Use string templates (`"Value: $value"`) instead of concatenation.
```

Nowhere in those 683 words does it say that a template inside a SQL string is how an injection starts. There is no line about input validation, access checks or secrets either. In the same project, the Dockerfile is reviewed with the question "Is permission validation complete?".

**0 of 683** words about security in the checklist an AI reviewer gets for a Kotlin file

## What that means in a pull request

Take two versions of one function. This is our own illustration, not output of the tool:

```kotlin
// Ticks every box on the built-in Kotlin checklist: no !!, no var, a string template.
fun findOrders(status: String) =
    jdbc.queryForList("SELECT * FROM orders WHERE status = '$status'")

// What a security line would ask for: a bound parameter.
fun findOrders(status: String) =
    jdbc.queryForList("SELECT * FROM orders WHERE status = ?", status)
```

The first version puts request input straight into the query. We did not run a model on it, so we do not know whether the reviewer would flag it anyway. What we do know is that nothing in its instructions asks it to, and that the one instruction about strings points the other way.

## Why we looked

[Open Code Review](https://github.com/alibaba/open-code-review) gained 951 stars on 17 November and 3,056 that week, according to [GitHub's trending list](https://github.com/trending?since=daily). It is a Go command-line tool under the Apache-2.0 licence, and its README says it started as Alibaba's internal reviewer. The repository describes it as having a "built-in multi-language ruleset (NPE, thread-safety, XSS, SQL injection)".

Half of the tool is ordinary code. Before any model is called, it decides which changed files get reviewed and which rule text goes with each file. That half runs without an API key, so it can be measured. We build Kotlin and Java backends, so we wanted to know what a team like ours gets on day one.

## What we did

We downloaded the 1.13.11 release binary (59 MB, checksum verified) into a container and created a 17-file sample project: a small Kotlin and Java order service with tests, a MyBatis mapper, a SQL migration, a Dockerfile, Terraform, a Flutter screen, a Vue page and a GitHub workflow. Then we asked the tool three things with `ocr delegate preview` and `ocr rules check`: which files it would review, which checklist each file gets, and what a project rule file changes. No model was called. The script and the full output are in this site's repository.

## Two files, two checklists

| The reviewer is asked about | `OrderService.kt` | `V2__add_status.sql` |
| --- | --- | --- |
| Checklist | Kotlin rule, 683 words | Fallback rule, 101 words |
| SQL injection and XSS | No | Yes |
| Handling of sensitive information | No | Yes |
| Permission validation | No | Yes |
| `val` instead of `var` | Yes | No |
| `GlobalScope` in coroutines | Yes | No |
| `@JvmStatic` for Java callers | Yes | No |

The Kotlin checklist is a good one for what it covers: null safety, coroutines, resource handling with `use`. It reads like a style and correctness guide, and that is what the model receives. Java is in the same position with 461 words on typos, dead code, logic errors, performance and thread safety.

For every checklist we counted the lines that contain one of six terms: injection, XSS, security, vulnerab, sensitive, permission.

_Figure: A SQL migration gets four security lines in its checklist. A Java or Kotlin file gets none._

The files the tool knows least about get the fallback checklist, and that one has a security section:

```console
$ ocr rules check src/main/resources/db/migration/V2__add_status.sql
Pattern: default
[…]
#### Security
Are there security vulnerabilities such as SQL injection or XSS?
Is sensitive information handled correctly?
Is permission validation complete?
```

Across all 66 built-in rule files, 23 contain none of the six terms. That number is a keyword count and it undercounts: the C rule has a section called "Buffer Overflow Protection" and matches nothing. For Java and Kotlin we read the full text, so those two zeros are real.

## Why a language rule removes the security questions

The mapping lives in [`system_rules.json`](https://github.com/alibaba/open-code-review/blob/v1.13.11/internal/config/rules/system_rules.json): 67 glob patterns, each pointing at one rule file, plus `default.md`. The lookup in [`system_rules.go`](https://github.com/alibaba/open-code-review/blob/v1.13.11/internal/config/rules/system_rules.go) walks the patterns in order and returns the first match. The default is used only when nothing matches. A language rule therefore replaces the default instead of extending it, and a language rule only asks about security if its author wrote that in. The Go, PHP and TypeScript rules have such lines. The Java and Kotlin rules do not.

_Figure: The first matching rule wins, so a Java file never sees the fallback checklist that asks about injection._

The resolved text is pasted into the prompt under the heading "Review Checklist". The system prompt of the main review task in that release does not contain the word security either. A model can still report an injection it happens to notice, and the planning prompt defines high severity as "May cause security vulnerabilities, data loss, system crashes, or critical functional failures". But the README's own argument for the design is that rule matching keeps "the model's attention sharply focused", and it states that recall is lower than a general-purpose agent's, "a deliberate trade-off favoring precision over noise". A checklist that points the model at spelling and dead code points it away from everything else.

## What else the preview showed

```console
$ ocr delegate preview --commit HEAD
[…]
~~- `docs/README.md` [added] +2/-0 (excluded: unsupported_ext)~~
[…]
~~- `package-lock.json` [added] +1/-0 (excluded: default_path)~~
[…]
~~- `src/test/java/shop/PriceCalculatorTest.java` [added] +7/-0 (excluded: default_path)~~
~~- `src/test/kotlin/shop/OrderServiceTest.kt` [added] +12/-0 (excluded: default_path)~~
```

Thirteen of the 17 files would be reviewed, and the answer came in 0.038 seconds. Leaving out the lockfile is sensible. Leaving out both test files is a choice you should know about: a pull request that removes an assertion is not looked at. The other surprise is `build.gradle.kts`. It matches the Kotlin pattern, so the build script is reviewed with 683 words about coroutines and data classes, while a Groovy `build.gradle` gets a 30-word rule about dependencies.

## The configuration we tested

The project's [documentation](https://github.com/alibaba/open-code-review/blob/v1.13.11/pages/src/content/docs/en/review-rules.md) describes the way out, and it works. Put your own checklist in the repository and merge it into every file's rule:

```json
{
  "rules": [
    { "path": "**/*", "rule": ".opencodereview/security.md", "merge_system_rule": true }
  ],
  "include": ["**/src/test/**"]
}
```

Our `security.md` held four lines: bound parameters in SQL, validation of input that reaches a path, a command or a redirect, an access check on every endpoint, and no secrets or personal data in logs. With this file in place the Java checklist grows from 461 to 530 words, the built-in part stays, and the preview reports 15 of 17 files because the tests are back:

```console
src/main/java/shop/PriceCalculator.java | **/* | 530 | 1
src/main/kotlin/shop/OrderService.kt | **/* | 752 | 1
src/test/kotlin/shop/OrderServiceTest.kt | **/* | 752 | 1
```

The flag matters. It defaults to false, and without it a matching entry replaces the built-in rule. We ran that as well: the same entry without `merge_system_rule` cut the Java checklist from 461 words to 60. You would have added security and silently removed the thread-safety and logic checks.


Run `ocr rules check` on one file per language in your repository and read what comes back. It needs no API key and shows exactly what the reviewer is asked to look for.


## What we would do

We would try this tool on a Kotlin or Java codebase only with a project rule file from the first day, and we would not count its default review as a security review. Out of the box, for JVM code, it is a reviewer for correctness, performance and concurrency. That is useful, and it is not what the repository description suggests.

Two things would change this view: a security section in `java.md` and `kotlin.md`, or published recall figures per language and per category of defect.

## Limits of this test

We measured the instructions, not the findings. No model was called, so we cannot say how many security defects the default setup misses in practice. One version (1.13.11, commit 0f5a284), one sample project of 17 files we made up, and a keyword count of six terms over the rule texts. The benchmark numbers in the README are the vendor's own and we did not reproduce them.

## Sources

- [alibaba/open-code-review](https://github.com/alibaba/open-code-review)
- [system_rules.json at v1.13.11: which rule file each path gets](https://github.com/alibaba/open-code-review/blob/v1.13.11/internal/config/rules/system_rules.json)
- [system_rules.go at v1.13.11: first match wins, merge_system_rule](https://github.com/alibaba/open-code-review/blob/v1.13.11/internal/config/rules/system_rules.go)
- [kotlin.md at v1.13.11: the built-in Kotlin checklist](https://github.com/alibaba/open-code-review/blob/v1.13.11/internal/config/rules/rule_docs/kotlin.md)
- [java.md at v1.13.11: the built-in Java checklist](https://github.com/alibaba/open-code-review/blob/v1.13.11/internal/config/rules/rule_docs/java.md)
- [Review Rules, the project's documentation](https://github.com/alibaba/open-code-review/blob/v1.13.11/pages/src/content/docs/en/review-rules.md)
- [GitHub Trending, daily, 17 November 2026](https://github.com/trending?since=daily)

---
Source: https://mg-devs.nl/en/blog/ai-code-reviewer-java-kotlin-checklists-skip-security/
