Ten read-only agents and a critic: auditing a billing system before touching it

The most dangerous code you will ever change is billing code you didn’t write. Before I touch a system like that, I want a map — not the map the docs claim, the map the code actually implements. Recently I needed one for a subscription system that had grown across four codebases: a mobile app, a shared models-and-services package, and roughly 17 independently deployable cloud-function codebases, each deployed on its own schedule, each old enough to have opinions.

The question that mattered was embarrassingly simple: when the app decides whether a user is entitled to the product, what does it actually check?

The answer turned out to be: it depends which part of the system you ask.

Four sources of truth is three too many

Reading through the code, entitlement was being answered by at least four competing signals:

• A database flag on the user document, written by several different code paths.
• A client-side inference based on which platform the subscription was purchased through.
• The payment provider’s live subscription state, checked in some flows and not others.
• A daily sweep job that reconciled some of the above, on its own schedule.

None of these was the source of truth. All of them were a source of truth, depending on which screen or function you were in. That’s not a bug you fix; it’s an architecture you have to understand completely before you fix anything, because every one of those signals has some code depending on its exact current behavior.

Understanding it completely is the problem. Four codebases and 17 deployables is more reading than one person does carefully in a reasonable time. It is, however, a very good shape for parallel AI agents — as long as you set the rules correctly.

The fleet: ten readers, zero writers

I wrote one structured audit prompt and fanned it out to about ten agents running in parallel, each assigned a slice of the system: this group of cloud functions, that part of the shared package, the purchase flows in the app, the webhook handlers, the reconciliation jobs.

Two rules made it work.

Rule one: read-only, enforced, not requested. Every agent ran with write access disabled. This isn’t paranoia about the agents going rogue — it’s about what read-only does to the output. An agent that can edit will drift toward fixing what it finds, and a half-fixed billing system mid-audit is strictly worse than an untouched one. An agent that can only read produces the thing I actually wanted: findings. Each agent’s deliverable was a report — every place its slice reads or writes an entitlement signal, quoted with file and line.

Rule two: someone’s job is to find what everyone missed. After the ten reports came back, I ran one more agent — a completeness critic — whose only instruction was to look at the combined findings and hunt for what wasn’t covered. Which codebases got shallow treatment? Which function directories appear in the deploy config but in nobody’s report? Where do the reports contradict each other?

The critic is the part most people skip, and it’s the part that earns its keep. Parallel agents have the same failure mode as parallel humans: everyone assumes the boring file is someone else’s slice. The critic’s whole job is to be annoying about that.

What the audit found that I wouldn’t have

The headline finding was the four-way entitlement split above. But the details are where the value was:

• Serialization drift. The shared package’s toMap/fromMap pairs had drifted in places — fields written by one codebase that another codebase silently dropped on read. In a billing context, a silently dropped field is a future incident with a date on it.
• Write paths nobody remembered. The database flag wasn’t written by two code paths, as assumed — it was written by more, including one inside a function that, by its name, had nothing to do with subscriptions.
• The sweep as load-bearing duct tape. The daily reconciliation job wasn’t a safety net over a correct system; it was the mechanism that made an incorrect system appear correct within 24 hours. Any plan that removed it without fixing the underlying writes would have surfaced a week of latent inconsistencies at once.

That last one changed the plan. The migration order I’d sketched before the audit would have consolidated the read side first. The audit made it obvious the write side had to be unified first, with the sweep kept running as a guardrail until write-side telemetry went quiet.

Why this is a fractional-CTO deliverable, not a coding trick

I’ve come to think of this as architecture archaeology, and it’s now a standard early step when I take on an existing system — the same way the first thing I do with a new client’s infra is read it before I deploy to it.

The economics are what changed. A careful manual audit of a system this size is days of senior-engineer reading, which means in practice it gets sampled instead of done — you read the “important” files and extrapolate. The parallel-agent version compresses the reading into hours and moves the human work to where it belongs: writing the audit prompt precisely, choosing the slices, judging the findings, and deciding what they mean for the plan. The agents did the coverage; the critic covered the coverage; I made the calls.

The output artifact matters too. The audit produced a written map of every entitlement signal, its readers, and its writers — which is exactly the document the next engineer (or the next agent) needs. The system didn’t have that document before. Most systems don’t.

The shape, if you want to reuse it

1. One question. Not “audit the billing system” — “find every place that decides whether a user is entitled, and what it checks.” Vague prompts produce vague coverage.
2. Explicit slices. List the codebases and directories per agent. Don’t let agents self-assign; that’s how the boring file gets skipped.
3. Read-only, mechanically enforced. Findings, not fixes. Quoted evidence with file paths, so you can verify any claim in seconds.
4. A completeness critic. One agent whose only job is gaps and contradictions in the combined output.
5. A human verdict. The reports are input. The migration order, the risk calls, what to tell the client — that part doesn’t delegate.

The plan I executed afterwards was different from the plan I had before the audit, and better in ways I can point to. That’s the test. An audit that doesn’t change your plan was either unnecessary or wasn’t listened to.