On this page
Most systems accumulate third-party credentials the way garages accumulate boxes: OAuth tokens for calendar integrations, session material for messaging channels, API keys users hand you for their own accounts. Early on, storing them as plaintext rows feels fine — the database is private, the disk is encrypted, what’s the threat? Then the platform grows, the blast radius of a database leak grows with it, and one day “encrypt stored tokens at rest” moves from the someday list to the sprint.
On an AI-agent platform I work on as fractional CTO, we did that retrofit on a live system — tokens in Postgres and in Cloudflare KV, users active the whole time. This is what the exercise actually involves, including the part where a CLI’s update banner nearly corrupted the verification.
One secret should do one job
The first design decision wasn’t about ciphers. It was about key hygiene: the temptation, in any codebase that grew fast, is to have one shared secret moonlighting as several things — an encryption key here, an HMAC secret there, a bearer credential somewhere else. Untangling that matters more than the encryption itself, because a key that does five jobs can never be rotated without breaking four other things.
We split to per-surface encryption keys: each category of stored credential gets its own key, named for the surface it protects. A compromise or rotation of one surface no longer touches the others.
Version the keys on day one
Every encrypted value we write carries a key version. The env holds the current version number and the keyed material for each version. Decryption reads the version off the stored value; encryption always uses the current version.
This costs almost nothing to add at the start and is miserable to add later. It’s what makes rotation a real operation — introduce key N+1, flip the current version, re-encrypt lazily or by sweep — instead of a migration you keep postponing because it would require touching every row atomically.
The backfill is the actual project
New writes are the easy half: wrap the write path, done. The existing data is the work. We wrote one-time backfill scripts for both stores:
- Postgres: select plaintext rows, encrypt, write back, in batches, idempotently — a row already carrying a key-version prefix gets skipped, so the script can die and re-run safely.
- KV: enumerate keys, read each value, encrypt, write back. Same idempotency rule.
Two properties matter more than speed. First, the script must distinguish “already encrypted” from “plaintext” by inspecting the value, never by trusting a progress marker. Second, it must run while live traffic reads and writes the same rows — which the key-version prefix also solves, because the read path handles both formats during the transition.
Verify by decrypting, not by counting
A backfill that reports “12,000 values processed” has proven nothing. The verification pass has to read values back and actually decrypt them — confirming the ciphertext round-trips to sane plaintext — rather than counting writes.
Which is where the best bug of the project lived. The KV verification shelled out to wrangler to read values, and partway through, the audit started failing with:
plaintext value is not valid auth-context JSON (head="There is a n")
“There is a n…” — the opening of wrangler’s “There is a newer version available” banner, printed to stdout, where the script expected the stored value. The tool’s upgrade nag was being parsed as production data. I wrote up the general lesson in a short TIL: pin your CLI versions in scripts, and validate the shape of anything you read through one before trusting it. If the verification had counted successes instead of validating payloads, that banner could have been silently “verified” as data.
Process honesty: the early merge
One more thing worth admitting, because polished writeups always omit it: partway through, the branch got merged before it was ready — an accidental early merge that had to be reverted and then re-landed cleanly once the backfill tooling was actually done. No user impact, but a reminder that the process risks on a change like this are as real as the technical ones. Encryption retrofits touch auth paths; the merge discipline around them should be tighter than normal, not looser.
The shape of the retrofit
If you’re planning the same exercise:
- Split shared secrets so each surface has its own key — before encrypting anything.
- Version every encrypted value so rotation is an operation, not a rewrite.
- Wrap the write path first, so the plaintext population stops growing.
- Backfill idempotently, detecting encryption state from the value itself, safe to re-run, safe under live traffic.
- Verify by round-tripping — read back and decrypt, and validate the shape of everything a CLI hands you.
- Treat the merge as sensitive as the code.
None of it is cryptographically exotic. All of it is operational care — which is where encryption-at-rest projects actually succeed or fail.