Resilience

One bad apple never poisons the tree

Before: one bad input stopped everything. A pair of broken test rows froze staging attribution and purchases for 10 days. Now the bad item is set aside with its last good numbers, Slack names it, and everything else keeps publishing. Only a real outage (ClickHouse down, lease lost, schema missing) still stops the pipeline.

BEFORE one bad item → nothing after it publishes10 days frozen on staging
NOW the bad item is held aside · the rest flows0 items held in production today
…and it costs nothing. Six speed gates, develop vs. the change
GateBeforeAfter
Extra queries on a healthy run874874✓ 0 added
400× speed (95% within)13.62 s13.58 s✓ same
400× with a bad item every minute13.58 s13.63 s✓ +0.4%
One bad item holds…—exactly 1 key✓ never a page
1,000 items already held13.58 s13.49 s✓ flat
Acquisition reconciler31.63/s31.52/s✓ same
gitops CI stays fast: ads-egress check duration
Before journey work
~5 min
After it (my slowdown)
10–14 min
Now (#1354)
4.6 min
The slow journey suites now run once, in parallel, and only when journey code changes. Deploy commits trigger none of them.