The change that passes every test and silently breaks a customer

A webhook run is "fine" on your side. The receiver is remote, async, and never sends you the error back. Here is the versioning discipline that makes a breakable contract actually hold — no new tool required.

---

Reader: You are a solo founder or a 1–3 person team shipping a SaaS, an API, or a webhook that your own customers build on. You are somebody's integration provider whether or not you have ever said the words "we are a platform."

What you get from this piece: after reading, you can (1) tell an additive change from a breaking change at a glance, (2) ship a breaking change as a shadowed parallel migration — new version, dual delivery, then drop the old — without silently nuking a consumer, and (3) put a real, dated deprecation window on it plus the smallest amount of telemetry that lets you see the integrations you cannot currently see. No new tool. No weekend spent building a schema-diff product.

---

The silent one: a deploy that ships green while a customer's parser drops every event

Here is a real failure, small in scale, enormous in shape.

Someone who builds on SendCloud's webhook reported a change: a field in the parcel-status-changed event — a subentity called the contract — had silently gone from "an integer" to "an object" carrying id and type. Their webhook handler was doing JSON deserialization against the old shape. The new shape broke it. Overnight. And — quoting the issue — "this is not documented" anywhere on the producer's dev site.

The person affected didn't learn it from an alert. No deliver signal went out. They filed a GitHub issue in another developer's library, where the library's maintainer replied, essentially, "wait, what changed exactly?"

Two facts in that small story matter. First: on the producer's side, whatever changed was a normal deploy — their tests passed, their 200s flowed, their dashboard was green, and nothing on their seat said "you just broke someone." Second: the consumer did not report back. Asynchronous receivers have no channel for "your payload no longer deserializes." They just... stop working. The event is received, parse fails, nothing happens, and the customer who depends on the result screams a few hours later.

A "clean deployment" and a "just broke every consumer" deployment are visually indistinguishable from the seat of the person who ships them.

---

The person you never hear from: the observability asymmetry

Everything you can check about a webhook change measures your side of the interface.

  • Your tests run your serializer, your fixtures, your team's understanding of the payload shape.
  • Your 200s are responses your server chose to send.
  • Your dashboard, your logs, your uptime page — all report on the thing you operate.

Every consumer is somewhere you cannot see, is asynchronous (so a semantic mismatch produces silence, not an exception you'd catch), and does not transmit its failure back to you. The interface between your business and their business has an observability asymmetry built into it: as the producer you only ever measure your own half.

Stripe spells out exactly why that asymmetry bites. The structure of objects sent to your webhook endpoints is governed by the producer's account default API version — not by what the consumer's code expects — and "if an endpoint has an explicit version set, it always uses that version" (Stripe API upgrades). Upgrading that account version, Stripe warns, "switches the version used to render objects sent to your webhooks." So one producer-side action silently re-renders every consumer's payloads — unless that consumer went out of its way to pin a version.

That is the whole failure condensed: the thing that tells you a change is correct is the side that never breaks, and you cannot see that side.

(Caveat, stated plainly: there is no measured statistic for how often indie SaaS ships a silent breaking webhook change, or what it costs. That number does not seem to exist publicly. The "you only find out when their customers scream" pattern is grounded in real events like the one above and in the way the giant producers structure their docs, not in a published figure. Treat the frequency claim as a working assumption that matches experience, not a datum.)

---

Why "just call it v2 / just use semver" does not hold

The obvious answer — put a version number on it — sounds watertight and does almost nothing. Here is why.

A version number only helps if the old version stays sealed. If you ship v2 as your payload but you are allowed to silently re-type or rename a field in what you now label v1, then the number is theater. The consumer who never migrated reads "v1" — the version they know — and their deserializer still breaks, because you changed what v1 means.

Semver is even worse detachment from the actual failure surface. Semver names the kind of change: MAJOR, MINOR, PATCH. It says nothing about the three things that actually break a consumer — the migration path, the parallel run, the sunset window their migration deadline. "Bump the major" tells the reader the change is breaking and does not deliver a usable way for the consumer to move. No consumer was ever saved by a version number; a consumer was saved by an additive-only v1 that stayed true and a dated window to leave it.

So the load-bearing rule is not "version things." It is additive-only. GitHub, in its classification of breaking vs additive REST changes, makes this the spine of everything it does (GitHub breaking changes):

  • Breaking: removing an operation, removing or renaming a response field, changing a type, adding a required parameter, making an optional one required, removing enum values.
  • Additive (available in all supported versions): adding an operation, adding an optional parameter, adding an optional header, adding a response field, adding an enum value.

The contract survives only because within a version you take the additive bucket only. The phrase that makes it usable is the rule: a rename, a re-type, or a removal is a breaking change by definition — never do it in place.

If you let people rename or re-type in place, "versioning" buys nothing. Versioning is just the shell that keeps the "never edit a contract in place" promise.

---

The discipline that actually holds — as the people who run it at scale do it

Four pieces, each cheap, none requiring a platform team:

1. Additive-only inside a version.

New optional fields, new enum values, new events — all fine, ship them any week. A rename, a re-type, or a removal forces you into step 2. Do not spend time negotiating with yourself about whether the consumer window is fine; classify it, then follow the steps.

2. A breaking change is a NEW version, never an in-place edit.

Pin it on the endpoint or in the payload. The version becomes the consumer's escape hatch: they can sit on the old version's contract for as long as your window allows, on your deploy cadence, because you have given them a thing to migrate to rather than a moving target behind them.

3. Run the migration in parallel — the shadowed rollout.

Make the new contract real while the old one is still live. Deliver both, process the new, dry-respond the old, then kill the old. This is Stripe's documented pattern for webhooks below, and it is the single most important mechanism in this piece.

4. Register a dated sunset and announce it in-band.

Publish Deprecated and End-of-Life dates and put the warning somewhere the consumer runs into it — the changelog, the health surface, the API surface itself — not in a silent deploy. Give a minimum window on the giants' scale (GitHub ~24 months; Shopify 12-month support, 9-month overlap — set your floor, do not shave theirs).

5. Measure what you cannot see.

Log which consumers are still pinned to the old version and flag them as EOL approaches. You cannot see their deserializers, but you can see their receiver hits and their pinned version. That is your only window onto them — use it.

---

Worked example A — a breaking webhook upgrade done without a silent break (Stripe's rollout)

This is the concrete, step-by-step version that Stripe documents for handling webhook versioning (Stripe — handle webhook versioning). You can run this on a plain webhook today.

You have a webhook at https://your-app.com/webhooks. New breaking version is 2026-04-10.

  • Stand up the new contract as a second endpoint, pinned to the new version.** https://your-app.com/webhooks?version=2026-04-10. Create it disabled first — that keeps it from receiving real events while it's still empty.
  • Enable both endpoints.* From here, events go to both*, old format and new format, side by side. (Stripe's account version switch determines which "render" each gets.)
  • Update your handler to process the NEW one and ignore the OLD one.** In Stripe's terms: if the query parameter is the new version, handle the event; if the query parameter is the older version, ignore it and return a 200 anyway — a 200 prevents Stripe from retry-restreaming the old event. You're not dropping the migration; you're holding it.
  • Flip priority.** When you are confident, invert the rule: process the new, and on the old return a 400 so Stripe retries the event against the new endpoint. That keeps the old stream alive as a safety net — if you need to roll back, the events you would have lost are being retried to the newer endpoint, and your older code path can still take them.
  • Disable the old endpoint.** Once the migration is done, turn the old one off. If you leave it returning a 400 forever, integrations that expect a clean 200 break in a new way — the shutdown is part of the contract.
  • Keep the rollback in the back pocket.** Stripe lets you roll back an account version for 72 hours after upgrade, and webhooks that were sent with the new structure and failed get retried with the old structure (Stripe API upgrades).

The shape you want: new version real, both live, old drains, new takes the load, then old goes dark — each stored until the migration completes. The consumer never saw a moment where their events vanished.

---

Worked example B — how the discipline is published as a contract consumers can bank on

Then, when the parallel migration is done, you advertise the contract so consumers can plan on it. The scale operators all publish the same overall promise, each with its own clockwork:

  • GitHub** classifies every change as breaking or additive, and the breaking ones only ever appear inside a new date-versioned release (e.g. 2026-03-10). Additive changes are available in every supported version. And the clockwork: "When a new REST API version is released, the previous API version is supported for at least 24 more months following the release" (GitHub breaking changes).
  • Shopify* releases a version a quarter, names it by date, and promises each stable version is "guaranteed not to change for its supported lifetime" — a minimum 12-month support with at least 9 months of overlap between consecutive versions. Deprecations are announced in-band* across all supported stable versions via the developer changelog, the API health report that lists resources needing changes, and in-tool deprecation warnings. And if a consumer targets a version that's retired, Shopify falls forward to the oldest accessible stable version and tells you in the X-Shopify-Api-Version header rather than failing silently (Shopify — versioning).
  • Auth0 runs a public lifecycle page that prints both a Deprecated date and an End of Life** date for each contract change, so there is an unambiguous, dated sundown window. E.g. the legacy Universal Login UI was deprecated 2024-08-23 with EOL 2025-07-31; the upcoming 10KB user-profile limit was deprecated 2026-08-04 with EOL 2027-03-04 (Auth0 — Deprecations and Migrations).

What all three are doing is the same thing: telling the consumer exactly how long their current version will keep working, so their migration has a runway they can schedule against. The specific dates age; the shape of the promise is the durable lesson — which is why this piece lives on the discipline, not on the version numbers.

And the migration choreography is not exclusive to Stripe. Chromatic's own docs for its webhook upgrade show the twin: add a mandatory version field to the payload (old format had none, so it disambiguates), run in support-both mode while the new version is rolled out, update integrations, and only then drop the old format (Chromatic — upgrading webhooks). That is the versioning discipline above — a version marker plus a parallel run — made concrete in another vendor's docs.

---

Reusable object: the outbound-contract rule in five steps

Keep this pinned in your repo's CONTRIBUTING.

  • Additive-only inside a version.** New optional fields, params, events, enum values. A rename, re-type, or removal is a breaking change by definition. Never edit a contract in place.
  • A breaking change is a NEW version, never an edit.** Pin it on the endpoint query (?version=2026-04-10) or a payload version field — whatever makes the old contract a stable place consumers can stay on.
  • Run the migration in parallel.** Create the new contract, deliver both, lightly respond 200 to the old and process only the new, then disable the old once moved. (Stripe's shadowed two-endpoint pattern.)
  • Register a dated sunset and announce it in-band.** Print a Deprecated date and an End-of-Life date; surface the warning in the changelog, the health/API surface, and the tooling consumers actually look at. Set your window on the giants' scale — GitHub ~24 months, Shopify 12-month support / 9-month overlap. Do not shave it to "a few weeks."
  • Measure what you cannot see.** Log which consumers are still pinned to the old version, and flag them as EOL hits. Their version header + event count is the only telemetry you will ever get from their side of the interface — treat it as sacred.

---

What to do first on your actual webhook today

Not "read more about versioning." One concrete move:

Pick the single webhook (or API route) with the most consumers, and add the version data you would need to run a shadowed migration on it tomorrow. If you have no version marker, add a payload version field now (additive, safe) or pin a version query param on the endpoint. Then, once you have declared a version, the next change to that contract — before you rename or re-type anything — runs the five-step rule above instead of being an in-place edit.

Corollary you can run today without shipping anything: run the classification over every change you already shipped. The rename from last quarter that's still silently live? That is the one to go back and version — it is already the consumer-facing break.

Two caveats to file next to this:

  • The big-producer discipline is uniformly documented (Stripe, GitHub, Shopify, Auth0), so a naive reading says "this is solved." The real story is a gap the docs do not measure: small operators running barely-SaaS webhooks rarely adopt what the giants enforce, and no data source measures that gap. This piece is the "bridge the gap" version, grounded in real first-party docs and one real operator failure — the potential silent-break frequency itself is not an invented stat, because no such stat is public.
  • If you are the one on the receiving end, the defense is the same discipline mirrored: pin a version, treat a missing version field as active, and log reads against the version claim so that when a producer silently re-types, you are the one with the receipt that is a few lines away from an upgrade path, not the one with a heap of dead events.

---

Sources

(This piece is a rewrite of operator best practice, not a reproduction of the daily-brief concept it started from. The product-ish frame in the brief — "build and sell a schema-diff tool" — was deliberately not the point; a competent team self-ships a JSON Schema diff in a weekend, so the new unit is the decision rule.)

---

X entryway list (for the distribution stage)

The angle brief named five entryways. Not yet written or posted; these are the assets for distribution once the draft is approved. Each is ordered as: hook type | the one claim/mechanism it would carry | link decision.

  • Sharp claim** — "You can break a customer with a deploy that passes every test. A webhook run is fine on your side — the receiver is remote, async, and never sends you the error back. 100% green CI is compatible with breaking every consumer." · Links to article (the claim earns the read; the article delivers the version + sunset discipline).
  • Specific failure moment** — "A webhook silently went from returning an integer to returning an object. It broke a consumer's deserializer overnight — and wasn't documented anywhere. The finding was a GitHub issue, not an alert the producer ever sent." · Links to article (the unmistakable failure is the hook; the article explains the invisibility + the discipline that prevents it).
  • Mechanism** — "The asymmetry you measure YOUR side of an interface — tests, 200s, dashboards — nobody sends you the consumer's error, and one that returns nothing looks 'done.' A clean deploy and a broke-everyone deploy look identical from a deployment seat." · Links to article (the mechanism stands alone; the article is the discipline that closes the gap).
  • Build sequence** — "Rename a field in a webhook and it's a breaking change. To ship it without nuking consumers: add a new version, deliver both in parallel, 200 on the old, drain on the new, set a dated sunset, then kill the old — the shadowed migration Stripe documents." · Links to article (the build is the actionable unit; the article is the why-it-holds plus the sunset contract).
  • Contract** — "GitHub supports a prior REST API version at least 24 months past the current one. Shopify gives stable versions a 12-month floor with 9 months overlap. Stripe prints a Deprecated date and End-of-Life for each contract change. 'Version your API' is only real once it has a dated window." · Links to article (the concrete contract sets the bar; the article turns it into the operator's five-step rule).

Note: per the repurposing rule, none of these is a shortened article summary with the same hook — each picks one distinct entry point (claim, failure, mechanism, build, contract) and the link is a bonus, not the whole point. Do not post until the user approves this draft.