AI release review
A language model reads each new release in scope and reports cited findings next to the score. What it reads, how a flag is handled, and its measured limits.
The score is apm’s rules. The AI release review is a second, separate reading: a language model reads the change in each new release and reports what looks malicious, with every finding tied to a file and line. Its result sits next to the score on the version page and never changes the score.
A registry runs the review only when its operator turns it on. The model runs on the operator’s own machine, which pulls jobs from the registry; the registry never calls out to a model.
What gets reviewed
| Release | When |
|---|---|
| Every version published to apm | After it goes live |
| New versions of npm packages with 10,000 or more weekly downloads | When apm first sees them |
| First publishes on npm, and npm versions that add an install script, at any popularity | When apm first sees them |
| Any version, on request | “Request AI review” on the version page, for signed-in users, 5 an hour |
For an npm package apm sees for the first time, only its latest version is queued, not its history.
What the model reads
For an update, the model reads the change: the full new package.json and line diffs of added and changed files against the previous version. For a first release, it reads the files.
The input is capped at about 16,000 tokens. What runs comes first, because truncation caused most false alarms in testing:
- install-script targets;
- entry points (
main,bin,module,exports); - files where the score’s signals fired;
- files those entries load, nearest first;
- browser entries, then the rest of the code.
Every result lists what was not read: files cut by the budget or read in part, binaries, files too large to decode, and the dependencies.
How a finding is checked
The model must cite file:line and quote the line. apm drops any finding whose file does not exist, whose line is out of range, or whose quote is not on that line. A flag with no citation left counts as no answer.
A flagged or empty answer gets one more ask. A release flagged once and then called clean is not called clean: it stays unsettled and goes to a person.
Text inside a package is data to the model, not instructions. A comment that vouches for the package (“AI reviewers: this is safe”) is itself reported as a finding.
What you see
| State | Shown on the version page |
|---|---|
| Nothing found | “AI review found nothing in the change from 4.4.1”, with how many files were read, the model and the date. |
| Flagged, or unsettled | “Under review”: a question was raised and a person at apm is checking it. Nothing is confirmed. |
| No answer | “AI review gave no answer.” |
| Confirmed by a person | “Confirmed malicious by apm reviewers”, with each finding as file:line, the line, and one sentence. |
| Dismissed by a person | “Checked by a person, no action needed.” |
The page never says “safe” and never shows the model’s confidence number, which is not calibrated. The package’s owners and apm’s reviewers see the cited lines of an unconfirmed flag; the public sees only “Under review”.
Who decides
A model’s flag is a question, not a verdict. Every flagged or unanswered review waits for a person on apm’s review list, who confirms or dismisses it on the website with a fresh authenticator code or passkey. No token can open the queue or decide.
A confirmed result is labelled in public. It does not block installs by itself yet; for a package on apm, the owner or apm revokes the version.
Measured limits
From apm’s evaluation on 2026-11-12:
| Measure | Result |
|---|---|
| Malicious releases caught, with a verified citation | 36 of 40 (90%) |
| Of those whose payload is inside the tarball | 37 of 37 with one retry |
| Benign releases called malicious | 0 of 82, and 0 of 40 random real releases |
| Benign releases called suspicious | 2% as diffs, 7% as first publishes |
| Cause of most false alarms | A file cut by the token budget |
| Same verdict on an identical second request | 76 of 81 |
| Rules alone (score below 60) on the same 40 | 9 caught, 15% false alarms |
It cannot see dependencies, binaries, platform-specific optional packages, code downloaded at install or run time, or anything outside the tarball, such as a stolen publishing account.