When to retreat from an upgrade

Production calendars broke with this:

Location with the name "US/Alaska" doesn't exist

The cause was an upgrade of the Dart timezone package — part of a larger round of dependency updates. The new version’s timezone database no longer resolved a location name that real user data depended on. Appointments that had scheduled fine for years suddenly couldn’t be displayed.

The fix seemed obvious: revert the timezone upgrade. That’s when the actual problem revealed itself.

The three-way standoff

timezone wasn’t a leaf dependency. Two other packages in the app — the local-notifications plugin and the calendar UI library — had both released versions that pinned the new timezone major version. Reverting timezone meant un-upgrading both of them too. Un-upgrading those meant giving back the fixes that had motivated the update round in the first place, and re-checking their compatibility with everything else.

This is the shape of the modern dependency problem: you don’t upgrade packages, you upgrade cliques. Three packages that must move together will also only retreat together, and the version solver will happily spend your afternoon proving it to you.

Deciding to retreat

There were two ways forward:

1. Push through — keep the new timezone, and chase the US/Alaska failure: migrate stored location names, patch resolution, hope no other legacy names lurk in production data.
2. Retreat — revert the clique to the last known-good set, ship the fix today, and schedule the migration as deliberate work instead of incident response.

I retreated. Not because pushing through was impossible, but because the failure was in production data resolution, and I couldn’t yet enumerate how many other stored values would break the same way. An upgrade whose blast radius you can’t state is not an upgrade you should be finishing under incident pressure.

The part that made the retreat cheap: the broken behavior was discovered quickly, the last-good versions were known, and nothing yet depended on the new APIs. Every one of those advantages decays with time. A week later, with new code written against the upgraded packages, the same retreat would have been surgery.

Retreat is a skill, not a failure

Nobody writes blog posts titled “we rolled it back.” The incentive structure of engineering culture treats retreat as an admission that you shouldn’t have tried. I think that’s exactly backwards. The teams that upgrade fearlessly are the ones that retreat well — because a cheap, practiced rollback makes attempting the upgrade low-risk in the first place.

What retreating well looked like here:

Know your rollback before you upgrade. Before starting a dependency round, note the current resolved versions (your lockfile is the record — commit it, and know which commit was last-good). The moment something breaks, “go back” should be a checkout, not an investigation.

Retreat the whole clique. Partial rollbacks are where the version solver eats afternoons. Identify the set of packages that moved together and move them back together.

Write the postmortem for the future battle, not the past one. The most valuable artifact from this incident wasn’t the fix — it was a written record of the standoff: which three packages are coupled, which version pair breaks US/Alaska resolution, what needs to be true before the next attempt (audit stored location names first, migrate legacy identifiers, then upgrade). The next person to try this upgrade — probably me, six months later — starts from that document instead of from the same error message.

Distinguish “retreat” from “never.” A retreat with a written re-entry plan is planning. A retreat with nothing written down is how apps end up frozen on three-year-old dependencies, which is its own slow-motion incident.

The bigger upgrade this sat inside

This standoff happened inside a broader framework upgrade round — the kind that touches the build toolchain (Gradle version bumps, JDK confusion between what the IDE, the shell, and CI each resolve) and interacts badly with over-the-air patching, which has its own opinions about what constitutes a compatible release.

The lesson scales up with the upgrade: the larger the round, the more valuable it is to be able to partially land it. Landing five safe upgrades and retreating from one coupled clique beats holding the whole round hostage to its hardest member — but only if you tracked which packages are independent, which is itself work worth doing before you start.

The rule I keep now

Before any dependency upgrade that touches production behavior, I want written answers to three questions:

1. What is the exact last-known-good state, and how fast can I be back on it?
2. Which packages are coupled to this one and must move (and retreat) together?
3. What user-visible behavior would tell me the upgrade is failing — and would it tell me quickly?

If I can’t answer all three, I’m not ready to upgrade. And if the answers exist but the upgrade still goes wrong, retreating isn’t the failure case. Retreating is the plan working.