I built a CLI to catch Stripe webhook regressions using real event fixtures — looking for 2–3 developers to try it
I’ve been building a small open-source CLI called Spoke Hooks.
The problem I’m trying to solve is pretty simple:
When a Stripe webhook breaks after a code change, the interesting failures often come from real event payloads — retries, duplicate deliveries, odd payload shapes, or business-specific data — not from the clean test payload you wrote six months ago.
Spoke Hooks lets you keep those events as regression fixtures.
The workflow is:
npm install -D [@spoke](/spoke)-labs/hooks
npx spoke-hooks init
npx spoke-hooks add stripe-event.json
npx spoke-hooks baseline
npx spoke-hooks test
baseline records how your current webhook handler responds.
After changing your code, test replays the same event. If the HTTP status or response body changes, it fails with a non-zero exit code, so it can run in CI.
Right now the scope is intentionally small: Stripe + Node.js + Express, local-only, with no account, hosted backend, dashboard, or cloud storage.
GitHub:
https://github.com/Spoke94/spoke-hooks
npm:
npm install -D [@spoke](/spoke)-labs/hooks
What I’m looking for right now is not feature ideas from people who haven’t used it yet.
I’d like to find 2–3 developers who already maintain a Stripe webhook handler and are willing to try it on a real or realistic project.
If you do, I’d especially like to know:
where the workflow feels awkward,
whether baseline + replay is actually useful,
and what stops you from keeping real webhook events as regression fixtures today.
I’m trying to validate the workflow before building anything bigger.
Completely agree with this perspective. Keeping things simple early on really helps avoid over-engineering. Thanks for sharing!
One subtle failure mode is fixture drift: a real payload captured against one Stripe API version can change meaning later. I’d store the event/API version and fixture schema version in baseline metadata so stale fixtures are flagged—are you planning that kind of freshness check?
One adoption detail I’d measure is the time from a production incident to a safe fixture: if redaction or naming takes longer than writing a regression test, the workflow may be abandoned. A small manifest recording event type, API version, expected side effects, and why the fixture exists could keep the collection understandable after a few months. I’d also treat duplicate delivery and async completion as separate assertions, since an HTTP 200 alone won’t catch those failures.
The pain is real. I migrated my payment webhook last week (Paddle,
not Stripe) and the scary part wasn't broken payloads — it was
silently wrong ones: 200 OK, half-processed state, nobody notices
for days.
Event fixtures would've caught what my unit tests didn't. Happy to
be one of the 2–3 — my stack is Next.js on Vercel, webhook handles
subscription state and credit top-ups, so it sees the full event
zoo including retries.
The real event fixtures angle is the right call.
Mocked payloads for webhook testing is exactly how most regressions get missed — the edge case lives in a real payload shape from a real customer flow, not the happy path fixture you wrote 8 months ago. The gap between "tests pass" and "production works" in webhook-heavy systems is almost always a payload schema problem that only shows up on a Monday morning.
Have you thought about a shared fixtures library as a longer-term direction? Teams with similar Stripe integrations often have near-identical webhook shapes and would genuinely pay to not rebuild this themselves. The person who curates the canonical real-world fixture set for common Stripe events has something competitors can't easily replicate.
What's the most surprising edge case you caught with real fixtures that the mocked version would have missed?
The scope discipline here is right, but I think the weak point is the oracle rather than the fixtures. Most Stripe handlers return 200 with an empty body no matter what happens internally, so status plus response body will stay green through exactly the regressions people care about: the subscription row that stopped being updated, the receipt email that now fires twice, the idempotency key that quietly changed shape. If you let the project register an optional assertion hook that returns a serializable snapshot of side effects, or let people plug in a recorder for outbound calls, then the diff starts covering the failures that actually cost money, and it does that without you needing a dashboard or a hosted backend. Two smaller things I would want documented before trying it on a real handler: how you deal with signature verification on replay, since a stored real event has a stale stripe-signature and timestamp and most handlers reject it outright, whether you re-sign with the local test secret or expect people to bypass the check in test mode; and whether replaying the same fixture twice in a row is a first-class mode, because duplicate delivery is the most common production bug in this area and you would get that assertion almost for free. Which direction are you leaning on the assertion surface, staying strictly HTTP-level to keep the tool trivial to adopt, or going after side effects and accepting the config cost?
Testing webhooks properly is always a pain point during integration, especially with edge cases. Clean approach with event fixtures! Wish you luck with the beta tests.
The real-event fixture approach is a practical way to catch webhook regressions that clean mocks miss, especially around retries and duplicate deliveries. I have seen similar issues when automation and coding agents replay stale payloads—how are you thinking about fixture versioning when the handler contract changes?
The ask for 2-3 real users before building anything else is the part most tool builders skip. Framing it as "not looking for feature ideas from people who haven't used it" is unusually disciplined.
We build an SEO auditing tool and the failure mode is similar: a code change that passes every test but silently produces different output on real data. Synthetic test cases cover the obvious paths. The edge cases that actually break production come from data shapes you didn't know existed until a user hit them.
The question I'd push on: once someone has twenty fixture files, do they ever update the baseline? The danger with snapshot-style testing is that the baseline becomes a sacred artifact nobody touches, and "just update the baseline" becomes the new "just skip the test." PII was mentioned in another comment — I'd add that fixture rot is the other adoption barrier.
This is a great approach. The biggest pain with Stripe webhooks is definitely the weird edge cases in live payloads that the official mock events never cover.
To answer your question on what stops me from keeping real events as fixtures today: sanitizing PII.
Pulling a real event means I have to manually scrub customer emails, names, or sensitive metadata before committing the JSON to my repo. It's usually too much friction.
If your CLI eventually adds a way to auto-sanitize or redact sensitive fields when saving the fixture, that would be a massive game-changer.
I maintain a few Node.js SaaS apps. I'll try to give it a spin locally this weekend and let you know how the baseline workflow feels on a real codebase. Keep it up!
Thanks — this is exactly the kind of feedback I was hoping to get.
And yes, the PII/sanitization point has now come up repeatedly. The current release does not auto-sanitize production fixtures yet, so please don’t commit unsanitized real customer data.
If you do try it this weekend, even using a manually sanitized or realistic fixture is enough for me to learn from the workflow.
I’d be especially interested in where init → add → baseline → test feels awkward on a real codebase, and whether the baseline/replay model feels useful enough that you’d actually leave it in CI.
Thanks for giving it a real try — I’d genuinely value the feedback.
Hi, I do QA on small SaaS apps, and webhook handlers are where I find the most expensive bugs, so this caught my eye. Two cases I'd add next to duplicates: events arriving out of order (invoice.paid before the subscription.created your handler expects), and the same event under a newer API version, since the payload shape follows the endpoint's pinned version. One practical worry: real fixtures carry customer emails and addresses, so a scrub step before they land in the repo would help teams say yes. What I can't judge from the README is how it behaves with async handlers that ack first and process later. Thanks for keeping the scope this tight.
Thanks — this is very useful.
The duplicate/idempotency case has now come up from several people, and your point about async handlers adds an important constraint to the side-effect assertion design.
You’re right that an immediate assertion after the HTTP response is not enough if the handler acknowledges first and processes later. I’m looking at the assertion layer as something that may need to wait/poll for application state rather than assume the side effect is synchronous.
The scrub step is also moving from “nice to have” to a V0 requirement based on the feedback so far.
I’m keeping out-of-order delivery and API-version drift in scope as validation scenarios for now rather than expanding the implementation too quickly.
For async handlers, would a configurable assertion timeout/poll command be enough in practice, or would you expect Spoke Hooks to understand the queue/job system directly?
The retry that still returns 200 is the case that got us once: a duplicate event applied twice while every clean test payload passed. One thing I'd want before wiring this into CI is a way to assert side effects (row counts in a test DB), not just status/body diffs. Is that on the roadmap?
Yes — this is exactly the kind of case that makes status/body checks insufficient.
A duplicate delivery that still returns 200 but applies the same side effect twice would currently slip through V0.
I’m now looking at side-effect assertions as the next focused step, but I want to keep the first version generic rather than baking in a specific database.
Would something like a post-replay assertion hook be enough for you?
For example, after replaying the fixture, Spoke Hooks could run a user-provided command/script that checks your test DB state — row counts, balances, idempotency records, etc. — and fails CI if that command exits non-zero.
If that would be enough for you to wire it into CI, that’s very useful to know.
Response diffs will miss the bug this is built for: a retry that still returns 200 while applying the event twice. I'd want baseline to capture side effects, not just status and body, and fail the replay when the same event.id is processed again. Is the check intentionally response-only for now?
Yes — the response-only check is intentional in the current V0. It was the smallest useful regression signal for proving the replay workflow, but the feedback here is making it clear that it is too weak for idempotency failures.
I agree with the invariant you’re pointing at: receiving the same event.id again should be allowed, but replaying it should not produce an unexpected second side effect.
I’m now looking at the next step as a generic side-effect assertion layer rather than a database-specific integration. For example, after replaying an event, Spoke Hooks could run a user-provided assertion command that checks DB/application state and fails CI if the state is wrong.
Would that be enough for your use case, or would you specifically want Spoke Hooks itself to understand and baseline database state?
The workflow is concrete, but the real test is whether developers keep using real-event fixtures after the first setup. Does baseline + replay become part of CI naturally?
Exactly — I think that’s the more important test than whether the first setup works.
I know the baseline + replay workflow works technically in CI, but I don’t have external evidence yet that teams keep maintaining the fixture corpus over time. That’s one of the things I’m trying to validate before expanding the product.
My hypothesis is that it only sticks if adding a new production event is cheap enough that developers treat it like adding a regression test after a bug — not like maintaining another test system.
Have you used any fixture/snapshot-based workflow that actually became part of the team’s CI habit? I’d be interested in what made people keep it up versus abandon it after the first setup.
And if you maintain a Stripe webhook handler yourself, I’d be very interested in having you try the current workflow and tell me whether it feels like something you’d actually leave in CI.
That “leave it in CI” test is exactly the signal I’d be interested in seeing. What’s the best email to reach you on?
Absolutely — you can reach me at spoke9412@gmail.com.
I’d really appreciate it if you try the current version in a real or realistic Stripe/Node project, especially with the idea of leaving spoke-hooks test in CI rather than just running it once.
I’m most interested in where the workflow becomes annoying, what would stop you from keeping it in CI, and whether maintaining the real-event fixtures feels natural over time.
Feel free to email me whenever you get a chance to try it.
Thanks! I’ve just sent it over.
Looking forward to hearing your thoughts whenever you have a chance.
The thing that decides whether a team can adopt this is redaction at the add step. Real Stripe events carry emails, names, addresses and card metadata, so spoke-hooks add is effectively asking developers to commit customer data into git, which is an instant no from anyone who has to pass a security review. Ship a default scrub of known PII fields with an allowlist for the business fields people actually assert on, and this goes from a personal tool to something usable inside a company.
That’s a fair point, and I agree this is a real adoption blocker for teams with security review requirements.
Right now, spoke-hooks add imports the event JSON as-is — there is no built-in redaction step yet. So today the developer is responsible for sanitizing production events before committing them.
A default scrub of known PII fields, with an explicit allowlist for fields the team wants to preserve, sounds much closer to the workflow I’d want for company use as well.
I’m deliberately trying not to add features before I know they unblock real usage, so I’d like to ask one thing: if spoke-hooks add redacted known PII by default and let you keep selected business fields, would that be enough for you to actually try it in a real project?
Also, are there any fields you would definitely want preserved by default for webhook regression testing?
Same problem from the reconciliation side. When you replay real events, do you also keep fixtures for payout., refund., and balance_transaction.* — not just charge.*? Those are the events where a payload shape change can quietly break the books: fee breakdowns shift, disputes land after a payout settles, and suddenly the CSV does not tie out. Curious whether you've seen that in the wild.
That’s a really good point.
Spoke Hooks isn’t limited to charge.* events — the fixture/replay flow works with generic Stripe event payloads, so payout.* and refund.* shaped events can be preserved and replayed as well.
What I haven’t validated yet is the reconciliation-specific failure mode you’re describing. V0 currently compares the webhook handler’s HTTP status and response body, so if the handler still returns the same response but silently produces the wrong ledger/reconciliation state, that would not be caught yet.
I also haven’t personally seen that exact payout/reconciliation breakage in the wild yet, so this is exactly the kind of case I’m trying to learn from.
If you have a sanitized payout/refund example, I’d be very interested to know what output you would actually want to assert on — ledger rows, fee breakdowns, exported reconciliation data, or something else. That would help me understand whether the useful regression boundary is the HTTP response or the downstream accounting effect.