The question
EpicScan is built on nano-analyzer, AISLE's open-source scanner: a model writes a security briefing for each file, scans it, and then argues with itself over several triage rounds before an arbiter keeps or rejects each finding. AISLE showed that small models in this pipeline find real, published vulnerabilities (System Over Model, April 2026).
Before we trust a scan of anyone's code, we wanted a known-good reference: if our setup reports something, is that a good result or noise? So we reran AISLE's published test cases on our own hardware, scored every run, and then added a second opinion from Claude Opus 4.6 on whatever the local model still believed in.
Ground truth: two real CVEs, before and after the fix
Each case is one file at two commits: the vulnerable version (the scanner should report the bug) and the patched version from the fix commit (it should not). The scanner sees only the file and the project name.
| Case | File | The bug |
|---|---|---|
| FreeBSD CVE-2026-4747 (NFS RPCSEC_GSS, remote code execution) | svc_rpcsec_gss.c, 1,715 lines, commits 6b2d6cc and 143293c | A credential of up to 400 bytes is copied into a 128-byte stack buffer without a length check |
| OpenSSL CVE-2025-11187 (PKCS#12 PBMAC1) | p12_mutl.c, 532 lines, commits f99912c and 8caf359 | The PBKDF2 key length from the file is not checked against the fixed key buffer, and the salt's type is not checked |
Setup
- Hardware: one Mac Studio (M4 Max, 48 GB) owned and run by Epicworks in Brno.
- Local model: Qwen 3.6 35B A3B (mixture of experts, about 3B parameters active per token, MXFP8), running locally on that Mac Studio. Two scans and two triage reviews in parallel.
- Pipeline: nano-analyzer, unchanged, with 5 triage rounds plus an arbiter, as in AISLE's tests. It runs in a locked-down container: read-only, no privileges, and network access only to the model endpoints.
- Second opinion: Claude Opus 4.6 with adaptive thinking. It receives each finding that survived local triage, the local arbiter's reasoning and the full file with line numbers, and must answer in a fixed format: confirmed, false positive or uncertain, with the attacker input, the reason and any outside facts it relied on. It is never told which bug is expected.
- Runs: 3 per file, 12 per pass. A run counts as found when a confirmed finding names the right function and bug, and clean when no finding about the fixed bug is left.
Result 1: the local model alone is well tuned for detection
| Model | FreeBSD found | FreeBSD patched clean | OpenSSL found |
|---|---|---|---|
| Qwen 3.6 35B A3B on our Mac Studio, pass A | 3/3 | 3/3 | 2/3 |
| Qwen 3.6 35B A3B on our Mac Studio, pass B | 3/3 | 2/3 | 3/3 |
| GPT-OSS-120B (AISLE) | 3/3 | 3/3 | 2/3 |
| GPT-OSS-20B (AISLE) | 2/3 | 3/3 | 2/3 |
| Qwen3-32B (AISLE) | 1/3 | 2/3 | 2/3 |
| gpt-5.4 (AISLE) | 3/3 | 3/3 | 3/3 |
Over both passes the local model found the FreeBSD bug 6 times out of 6 and the OpenSSL bug 5 out of 6, which puts it at the level of GPT-OSS-120B in AISLE's table. Each scan took 3 to 14 minutes. The one OpenSSL miss was a triage call: the local reviewers talked themselves out of the key length overflow. The one patched-file failure (pass B) was a claim that the new length check could be bypassed with a negative length; the field is unsigned, so it cannot.
The problem is everything else. Almost every run also kept findings that have nothing to do with the CVE, including on the patched files: 15 of 24 surviving findings in pass A and 16 of 24 in pass B. Most were the same pattern, a "NULL pointer dereference" on a field that cannot be NULL, because the decoder requires it or the kernel allocator waits instead of failing. A human would have to read every one of them.
Result 2: Opus as a second opinion removes most of the noise
| Pass A | Pass B | |
|---|---|---|
| Findings that survived local triage | 24 | 24 |
| About the real CVE: kept by Opus | 9 of 9 | 8 of 8 |
| Unrelated or wrong: confirmed by Opus | 4 of 15 | 2 of 16 |
| Unrelated or wrong: marked uncertain | 0 of 15 | 4 of 16 |
| Findings on the patched files: before, then confirmed after | 6, then 3 | 9, then 2 (+2 uncertain) |
| Opus time (sequential) | 23 min | 26 min |
| Opus cost | USD 3.14 | USD 3.12 |
Pass A sent the survivors of the first 12 runs to Opus. Pass B is a fresh set of 12 runs with the local scan and the Opus check end to end. With the check, the scorecard for pass B becomes FreeBSD found 3/3, patched clean 3/3, OpenSSL found 3/3.
What stood out:
- No real bug was thrown away. All 17 findings about the two CVEs were confirmed, with the attacker input and the exact line.
- About two thirds of the noise is gone. Of 31 unrelated or wrong findings, Opus rejected 21, marked 4 as uncertain (each with what would settle it, usually a definition in another file), and confirmed 6.
- It rejected the false bypass claim on the patched FreeBSD file with 97% confidence, pointing out that the length field is unsigned.
- It fixed wrong explanations. In pass A the local model twice flagged the OpenSSL salt as a "missing field". Opus answered that the field is required, but its type is never checked, which is exactly the second half of the real CVE.
- What it kept was consistent. Five of the six unrelated findings it confirmed point at the same kind of issue in FreeBSD code that the fix did not touch, in both files. They need an authenticated session and may be real. We have not verified them, so we do not describe them here until that is settled with the FreeBSD project. The sixth is a claim that a file with a huge iteration count makes verification slow, rated at only 65% confidence.
Cost and privacy
The local scan costs electricity: a Mac Studio running at full load for a few minutes per file. The Opus check cost USD 3.12 to 3.14 per pass of 12 files, about USD 0.05 to 0.80 per file, depending on file size and how many findings survive. The source file is cached between findings of the same file.
The check sends the flagged file to Anthropic's API. That is why it is not part of a standard EpicScan scan: hosted scans stay on our own Mac Studio machines and no third-party AI service sees your code. The Opus check will be an optional extra you turn on yourself, billed per token. For this report we used it only on public open-source code.
Limits of this test
- Small sample. Two CVEs, three runs each, per pass. This is a calibration check, not a leaderboard.
- The models may know these CVEs. Both are public, and Opus 4.6 or Qwen 3.6 may have seen the fixes during training. Opus confirmed the bugs with line-level reasoning, but we cannot rule this out.
- Opus only sees what local triage kept. A real bug that the local reviewers reject (as in the third OpenSSL run of pass A) never reaches Opus. Sending rejected findings too is the next experiment.
- AISLE did not publish their exact command lines, so the comparison with their table is close but not exact.
Three pipeline fixes we needed
- Triage could not read the code. On a read-only container mount the code search tool returned nothing, so reviewers judged findings blind. The scanner now works on a private in-memory copy.
- Frontier models sometimes answered in prose. nano-analyzer expects a findings list at the end; we add a short format reminder, and the Opus check uses a strict output schema.
- Memory headroom. A 35B model needs about 30 GB free to load on a 48 GB machine. We capped the container runtime's memory so the model always fits next to our other services.
Sources
- AISLE, System Over Model: Zero-Day Discovery at the Jagged Frontier (reference results).
- nano-analyzer (Apache-2.0), used unchanged.
- FreeBSD source (BSD-3-Clause) and OpenSSL source (Apache-2.0) at the commits above.
Questions or want the raw run data? Write to [email protected].