← Writing

The Worker was being killed 1,800 times a day. Here's why.

A production postmortem from Cloudflare Workers: 11,058 out-of-memory kills in a week, three wrong hypotheses, and a root cause that turned out to be a single React dependency array.

On this page

For a week, the production API of an AI-agent platform I work on as fractional CTO was dying with Worker exceeded memory limit — over and over. A seven-day log census eventually counted 11,058 out-of-memory kills, running at ~66 per hour and peaking at 178 per hour in the hours before we shipped the fix.

Most of them traced back to a single dependency array in a React component.

This is the full story: the mechanism, the wrong turns, the fix, and the rules we wrote down afterwards so we never build this bug again.

Why a Worker runs out of memory

Each Cloudflare Worker isolate gets 128 MB. Two things consume it.

First, a fixed floor. Ours was ~14.7 MB before a single request arrived: an 11.16 MB bundle plus a 3.5 MB Prisma query-compiler WASM module.

Second — and this is the part that reframed everything — ~3.81 MB per concurrent database client. The schema has grown past 180 models, and every request built its own Prisma client, which compiles that schema inside WASM. Two properties make this dangerous:

  • The compile is synchronous. It blocks the isolate thread while it runs.
  • WASM memory never shrinks. memory.grow() is one-way. An isolate’s footprint is set by its historical peak concurrency and stays there for life.

Do the arithmetic: 10 concurrent clients × 3.81 MB = 38.1 MB, plus the 14.7 MB floor, and you’re at ~53 MB before any real work happens. A burst that briefly pushes concurrency higher raises the floor permanently.

The tell we almost missed

Before we understood the mechanism, the logs held a clue that didn’t fit any of our theories: an endpoint that queries a 30-row table took 4,992 ms. Twelve of twenty-three “cheap” requests exceeded two seconds — while the database slot queue sat empty at zero.

Cheap endpoints taking five seconds with nothing queued isn’t a slow database. It’s the isolate thread blocked, synchronously compiling schemas for a pile of concurrent clients. Once you know that, the memory numbers and the latency numbers are the same story.

The root cause: one object in a dependency array

One page of the dashboard has a chat. Subscribing to its realtime channel requires an HTTP call to the Worker for channel authorization — and on Workers, that call costs a Prisma client.

The subscription effect depended on an object that was rebuilt inline on every render:

// rebuilt every render → new identity every render
const activeContact =
  conversationData?.contact ?? contacts.find(…) ?? null;

useEffect(() => { client.subscribe(channel) },
  [activeContact, queryClient, channelName]);
// React compares by identity → teardown + resubscribe, per render

Every render tore down the subscription and re-subscribed, firing a fresh authorization POST. We measured 12 of them in 167 milliseconds, and up to 64 kills in a single second. Not a reconnect loop — a render loop.

Here’s the finding worth framing: those requests return roughly 200 bytes each. They still killed the isolate. The driver is concurrent client construction, not response size. On Workers, a request is not cheap because its response is small.

The other half: one click, twenty requests

The second cause has nothing to do with bugs. Opening a detail view in the app fires about twenty independent requests within a few milliseconds. They land on one isolate, each takes a database client, and the isolate dies partway through.

Worse, the view is a master–detail layout on a single route, so selecting an item is a query-param change, not a navigation — every click repeats the burst. Triaging ten items means ten full bursts. Three of those twenty calls fetched a paginated list just to extract a single value.

What we changed

Four changes shipped together:

  1. Stabilized the subscription effect. Memoized the object, and made the effect depend on its userId — a primitive — rather than the object itself. This alone removed the render loop behind ~34% of all kills.
  2. Made channel auth cheaper. The authorization answer was derivable from the session row already loaded, instead of two extra user lookups.
  3. Stopped redundant refetching. React Query staleTime on the projects list went from 0 → 60 s; notifications from 15 s → 5 min. Notification freshness is push-driven anyway — the server publishes an event and the client invalidates, which refetches regardless of staleTime. The short window bought no liveness; it just refetched on nearly every navigation.
  4. Lowered the client ceiling from 16 → 10 concurrent Prisma clients. Pure headroom — it caps the worst case around 53 MB.

The result

WindowHoursKillsPer hour
7-day census, before16711,05866.2
Hours immediately before release7.11,264177.9
After release18.8180.96

A 99.5% reduction, confirmed the next day. Kills now touch 0.14% of invocations — and that’s an overestimate, because the success-count denominator was itself truncated by an API row cap. The surface that produced thousands of kills went silent, and traffic was healthy throughout, so the quiet is a change in behaviour, not an absence of load.

The house rules we wrote down

The incident is the cheap part. The expensive part is that a Worker is a shared runtime with a hard cap — you cannot add memory to it — and features get built as if it were a server. These are now written rules on the project:

  • Count the fan-out, not the endpoint. “Is this endpoint fast?” is the wrong question. The right one is: how many fire at once, and what else fires alongside them? If a page already makes 15 requests on mount, the 16th is not free — it raises the peak concurrency that permanently sets the isolate’s memory floor.
  • Never fetch a paginated list to extract one value. Return the derived value from the parent endpoint.
  • Frontend fetching decisions are architecture. An unstable useEffect dependency is a capacity bug, not a rendering nit. Create long-lived clients once in a []-scoped effect; only the subscription goes in the dependent one. Depend on primitives.
  • Stream, never buffer. new Response(r2Object.body) is correct; await r2Object.arrayBuffer() pulls the whole file into a 128 MB budget. Size-gate before reading.
  • Don’t fix memory by raising a limit. A cap being hit means the work is misshaped. Raising a client ceiling or --max-old-space-size converts a visible failure into a slower, less obvious one.

The one sentence to remember

On Workers, a request is not cheap because its response is small. Every database-touching request costs memory that is never reclaimed, so the number that matters is how many run at once — which makes burst width an architectural concern that lives as much in the frontend as the backend.

If your serverless backend is dying mysteriously under a healthy-looking load, start with the story of how I found this one — including the three wrong root causes I confidently reported first. That’s its own post.

← All writing Book a call →
Book a call → WhatsApp