Half the Findings Were the Scanner's Fault

We have a rule about diagnostics: never sell one you haven’t survived. When we productized the AI Codebase Debt Audit, the first repo it scanned was not a client’s. It was marrow, our own open-source memory engine, a codebase we ship, run in production, and had every reason to believe was clean.

The first scan said otherwise. Thirty-six findings. Eight high severity. A paydown estimate of $3,150 at a blended $150 an hour, with every finding traced to a file and line and the arithmetic shown in the report.

Then we did to our own audit what we do to every audit: we verified the findings before believing them. That pass is where this stops being a marketing story.

Eighteen of the thirty-six findings were the scanner’s fault. Three distinct bugs, each one invisible unless you check the diagnostic against ground truth you actually know:

First, the secrets probe flagged five high-severity leaks that were our own privacy-test fixtures: synthetic credentials planted on purpose so the test suite can prove the real leak detector catches them. The scanner’s history mode ignored the annotations that mark them as intentional. A real audit would have billed a client for rotating secrets that were never secrets.

Second, thirteen duplication findings pointed at applied database migrations. Migrations are append-only history: a redefined function is supposed to repeat its predecessor, and nobody sane refactors a shipped migration. That was nearly half a ledger of work no one should ever do. And it would have appeared on every client’s ledger too, because every production repo has migrations.

Third, the static-security probe had silently produced nothing at all, and the harness read the silence as a clean pass. A scan that returns zero findings and a scan that never ran are different events, and a diagnostic that cannot tell them apart will eventually sell someone a clean bill of health it never earned.

So we fixed the scanner first. Secrets scanning now honors intentional-fixture annotations and separates committed secrets from a developer’s local files, which are not repo exposure. Migrations, lockfiles, and build artifacts are excluded from refactorable debt. The security probe now records its own errors loudly, so a file it failed to parse is surfaced instead of counted as clean.

That left the real debt, and the real debt was real. Six dependency vulnerabilities, two of them high, sitting in the chain of a repo we tell people to run. Fixed and shipped the same day, advisories cleared to zero. The most expensive duplication finding was four ingest sources each carrying its own copy of the same scan-and-dedupe logic, drifted just enough to be dangerous. We extracted one shared path, deleted a net 32 lines, and proved the change behavior-preserving before merging: all 70 tests green, public CI green on both commits.

The final ledger: 10 findings, zero high severity, $675. Duplication fell from 3.4 percent to 1.42%. The ten that remain are all medium, all named, and all still on the ledger, because not everything is worth paying down at once and an honest ledger says so.

Two things came out of that day. A cleaner repo, which is the small one. And a scanner that earned the right to be pointed at someone else’s code, which is the large one. Every automated diagnostic overstates. The vendors selling scan reports know this and ship the inflated number anyway, because a bigger number closes deals. Our audit’s verify pass exists because the first draft of our own report lied to us, and we would rather find that out on our own repo, for free, than have a client find it out after paying.

That is what the audit fee buys: findings that survived being checked against a codebase whose ground truth we knew cold, run by people who published the part where their own tool was wrong. The before and after are public. The commits are public. The CI runs are public. When we hand you a ledger with a dollar figure on it, that figure has already been through the process you just read.