Built the webhook backend for our Zapier integration. The interesting decisions weren't about Zapier at all.
Shipped the webhook delivery layer that'll sit behind our Zapier integration this week, per-Zap HMAC-signed subscriptions, retry, fan-out to multiple subscribers per event. Haven't submitted to Zapier yet (want real users running live Zaps against it first), but building the plumbing forced a few decisions I didn't expect to be hard.
Retry vs. give up. A webhook can fail for two really different reasons. Either "the server's having a bad minute, try again in five" or "this is set up wrong, and it'll never work no matter how many times you try." I split it by status code: a 404 or 401 means give up right away, no retry, something's actually broken on their end, not just slow. Everything else gets retried, spread out over 30 minutes. Get this wrong either way and it's bad: retry too hard and you're hammering someone's server for half an hour over a typo'd URL. Give up too fast and a real hiccup gets treated as dead, and their Zap just... stops, with nobody noticing.
Checking the URL again right before sending, not just when it was first saved. Almost missed this one. When someone sets up a webhook, you check the URL isn't pointing somewhere sketchy, like an internal address. But that check happens once, at setup. DNS can change after that, so a URL that looked fine when saved could point somewhere it shouldn't by the time it's actually delivered, especially if that delivery got retried 20 minutes later. So now it checks again, every single time, right before it sends.
A delivery budget per organization. One viral link can generate a lot of clicks very fast, and if someone has a "notify on every click" Zap wired up, that's now a lot of outbound webhook calls in a short window, enough to look like abuse to Hangfire (our job queue) or to whatever's on the receiving end. Capped it at 1,000 deliveries/hour per org; past that, new events get dropped with a log line instead of the queue backing up.
None of this shows up in any Zapier onboarding doc, because none of it is actually about Zapier, it's just what "reliably deliver something to a URL you don't control, that might be broken, malicious, or misconfigured" turns into once you sit down and build it properly.
Here's the actual fork I haven't resolved: the delivery budget. Capping it protects my infrastructure and the receiving server from a viral-link flood, but from the customer's side, it means their Zap can silently stop firing mid-spike with zero warning, at the exact moment their data would matter most (a launch, a viral post, whatever caused the spike in the first place). Cap it and you're protecting the system by quietly failing the user when they need it working. Don't cap it and one customer's traffic spike becomes everyone's incident.
I picked "cap it, log it, move on", but I'm not convinced that's right, and I don't think "just notify them" fully answers it either, since the whole point of the spike is they're too busy to watch a dashboard. If you've built rate limits or budgets on top of someone else's usage pattern (not just your own infra), where did you land: protect the system and eat the occasional silent drop, or protect the user's expectation and risk taking the hit yourself?
Good write-up. The budget decision is the right place to be uncomfortable.
I would avoid treating “over budget” as a successful non-delivery. A pattern that has worked well for webhook-like systems is to make it a separate state: delivered, failed, deferred, capped. The cap still protects your infra, but the customer gets an audit trail and a replay window instead of an invisible gap.
For high-volume spikes, I’d probably separate events by consequence too. Analytics-style events can be sampled or dropped with clear counters. State-changing events, especially anything tied to billing, fulfillment, account changes, or customer-visible automation, should be delayed or moved into a recovery queue before being silently dropped.
The user notification can be async, but the system of record should know exactly what was skipped and why. That gives you a better support/debug story without letting one customer turn into everyone’s incident.
Yeah, this is the fix I keep circling back to. Right now "capped" and "failed" look the same to the customer, just a Zap that stopped firing. Splitting them into real states (delivered/failed/deferred/capped) plus a replay queue is way better than what I have.
The billing/state-changing vs analytics split is a good call too. I don't think I need to sample-drop anything yet at our volume, but tagging events by consequence so the important ones get a recovery path instead of just disappearing into the cap, that's worth building now rather than after someone's Zap silently breaks.
Going to add a status field per delivery attempt instead of just log-and-drop. Thanks for this, appreciated.
Nice, that status field will pay for itself in support/debugging.
One implementation detail I’d add: keep both a normalized delivery status and the raw attempt history. For each attempt, store status_code/error, retry_after or next_retry_at, and capped_reason if the budget stopped it.
That lets you answer the important question quickly: did we not send it, did they reject it, did DNS/security block it, or did we cap/defer it? I’d also show capped/deferred counts per endpoint, not only per org, so one noisy destination doesn’t hide a broken but important integration.
Same, adding per-endpoint counts too so one bad destination doesn't get lost in the org total. Only thing I'm still figuring out: when something gets capped, right now the customer doesn't see "deferred" until the retry actually happens, which could be up to an hour later. Thinking I should just flag it as deferred right away, then let the retry happen quietly in the background, so people aren't left wondering if it's stuck.
Yes, I would flag it as deferred immediately. I would separate the customer-visible state from the delivery attempt lifecycle:
That way the UI can say "deferred due to delivery budget; next retry scheduled" instead of looking stuck. The retry should append a new attempt, not overwrite the deferred record, because support will need to explain why the delivery was delayed.
I would also avoid retrying forever. After a bounded window, move it to needs_review or expired and make replay an explicit action. That protects your infra without making the customer guess whether the Zap is dead.
I really appreciate this, it's been super helpful. I'm on board with almost everything, one thing that would be great is to have a clear status for each attempt, like whether it was delivered, failed, deferred, or capped, rather than just logging and dropping it.
We're going to go ahead and flag something as "deferred" right away, as soon as it reaches the limit, instead of waiting to see if it needs to be retried. This way, people won't be left wondering for a whole hour if the Zap has stopped working.
Also splitting counts per-endpoint, not just per-org, so one noisy destination doesn't bury a real problem.
Let's put this on hold for now and come back to it later.
Thanks for saving me from sending out the wrong version, I really appreciate it.