I built a CLI to catch Stripe webhook regressions using real event fixtures — looking for 2–3 developers to try it

That’s helpful — “the final bank-facing reconciled number” is a much clearer regression boundary than the raw event itself.

It also reinforces the direction I’m leaning toward: Spoke shouldn’t need to understand every accounting model internally. A project-defined assertion hook could return that reconciled value, and baseline/test could compare it just like any other side-effect snapshot.

Appreciate the concrete example.

Yes — that’s exactly the trust problem I’m trying to solve next.

The current release does not yet have a pre-save redaction flow, and I don’t want to pretend that raw production events are safe to commit as-is.

The direction I’m leaning toward is for add to make the saved fixture explicit before persistence: show what will be stored, apply/default redaction rules, and let the developer review the result before it lands in the repo.

I also want to keep a fast path for already-sanitized or synthetic fixtures so the common case doesn’t become an interactive wizard.

Your point about this deciding whether someone uses a real event versus falling back to another clean sample is especially useful — that’s exactly the adoption behavior I want to avoid.

This is exactly the distinction I’m converging on from the recent feedback.

The response is only one part of the oracle — the expensive regressions are usually in state.

The direction I’m leaning toward is:

keep HTTP response checks as the zero-config baseline
add an optional project-defined assertion hook that can run SQL, call a function, or return some serializable state
store that state in the baseline and compare it after replay
make duplicate replay a first-class scenario so the second delivery can explicitly assert “no additional side effect”

I also agree on signature verification — I want the replay path to preserve it rather than bypass it, likely by re-signing the stored payload with a local/test secret at replay time.

On Fastify: current V0 is deliberately scoped around Stripe + Node + Express, so I don’t want to claim Fastify support before I’ve tested it properly. The replay layer is HTTP-oriented though, so I’d like to keep the design framework-agnostic where possible rather than baking Express assumptions into the core.

Really useful feedback — especially the “would adopt if…” criteria.

Absolutely — still have room, and this is exactly the kind of test I’m looking for.

Current V0 is Stripe + Node and the flow is:

npm install -D @spoke-labs/hooks
npx spoke-hooks init
npx spoke-hooks add <event.json>
npx spoke-hooks baseline
npx spoke-hooks test

One important warning before you use a production-derived event: automatic PII redaction isn’t in the current release yet, so please sanitize any real customer data before committing the fixture.

What would help me most is feedback from the real handler itself — where init → add → baseline → test feels awkward, whether the replay catches anything your existing tests miss, and whether you’d actually leave it running in CI afterward.

Thanks for offering to give it a real run.

Thanks — that’s exactly the kind of problem I’m trying to keep the first version focused on.

API-version drift has come up repeatedly now, so I’m leaning toward storing version/freshness metadata with each fixture rather than treating the payload as timeless.

I also like the “compare expected behavior between releases” framing. If a baseline changes intentionally, the tool should make that change visible rather than silently replacing the old expectation.

And I agree on the early-user point — I’d rather learn deeply from a few real integrations than optimize for broad reach before the workflow is proven.

This is a really useful distinction.

The “fresh process” point is especially important — a handler can look idempotent in development simply because in-memory state survives between events, while a fresh deploy exposes that nothing was actually persisted.

I’m starting to think duplicate delivery and retry-after-5xx should be modeled as sequences rather than isolated fixture replays, with the option to reset the app/process between runs.

And agreed on signature verification: I want the replay path to stay as close to production as possible rather than teaching users to bypass verification.

Thanks — this adds a useful process-lifecycle angle to the idempotency problem.

Thanks — appreciate the context and the kind words. I’m keeping the current validation focused on direct developer usage and feedback for now.

Thanks — agreed on treating webhook input as untrusted and keeping signature verification in the real replay path.

Idempotency and out-of-order delivery have both come up repeatedly in the feedback as well, so they’re definitely part of what I’m validating now.

Appreciate the checklist reference.

Thanks — those are exactly two of the things I’m trying to keep simple.

The current V0 doesn’t hard-code a specific Stripe event type; it stores and replays the event JSON fixture against your local webhook handler, so the same workflow can be used for different event types.

For CI, the goal is intentionally boring: install the npm package, keep the fixtures in the repo, run spoke-hooks test, and let the exit code gate the pipeline. No hosted backend or dashboard is required.

I’ve already got the basic flow running in GitHub Actions, but the part I’m validating now is whether that setup stays frictionless on real codebases rather than just in the demo project.

This is a strong point, and you’re the second person to call out signature verification specifically, so I’m treating it as a real compatibility requirement now rather than just a docs issue.

I agree that replaying the originally captured Stripe-Signature is the wrong behavior — it can fail for timestamp freshness instead of application behavior and teaches people to disable verification in tests.

My preferred direction is to keep the exact stored payload, then re-sign it at replay time with a configured local/test webhook secret so the real verification path still runs.

I’d rather preserve end-to-end behavior than bypass signature checking by default.

And yes, stripe_api_version should live next to the fixture metadata as part of the same replay contract.

Thanks — that’s exactly what I’m trying to protect right now.

The temptation is to keep adding features once people start suggesting good ideas, but I’d rather prove the smallest useful workflow with real users first.

Yes — this kind of lightweight freshness check is starting to look increasingly justified.

I don’t want to build a full fixture-versioning subsystem, but storing metadata like Stripe API version, capture/import time, and a fixture schema version feels like a useful minimum.

That would at least let Spoke distinguish “this fixture is old/stale relative to the current contract” from “this regression is real.”

The part I haven’t decided yet is how strict the tool should be.

Would you expect an API-version mismatch to emit a warning and let the test continue, or should CI fail hard until the fixture is explicitly reviewed/re-baselined?

That’s a useful way to frame it — “time from incident to safe fixture” may actually be one of the better adoption metrics for this workflow.

If capturing, sanitizing, naming, and understanding the fixture takes longer than writing a normal regression test, I can see teams abandoning it quickly.

I also like the lightweight manifest idea more than jumping straight to a full fixture-versioning system. Event type, API version, expected side effects, and a short “why this fixture exists” note would probably make the corpus much easier to maintain over time.

And agreed on duplicate delivery vs async completion — I’m starting to think those need separate assertion semantics rather than being collapsed into one generic HTTP check.

Would you expect that metadata to live inside each fixture JSON itself, or in a separate manifest file for the whole fixture set?

Thanks — that 200-OK-but-partially-applied-state failure is exactly the kind of case the recent feedback has been pointing toward.

I really appreciate the offer to be one of the 2–3 testers.

One important constraint though: the current V0 is intentionally Stripe-only, so I don’t want to waste your time by pretending Paddle is supported yet.

Your example is still very useful because it reinforces that the real regression oracle has to cover side effects, not just the HTTP response.

If you also maintain any Stripe webhook handlers, I’d be very interested in having you try the current version. Otherwise, I’d be happy to keep you in the loop for a later Paddle/Next.js pass once the Stripe workflow is proven.

And the Next.js/Vercel setup is useful context too — I’m seeing that the same problem exists beyond long-running Express handlers.

Thanks — I agree the shared fixture library could be an interesting longer-term direction, especially if the events are canonicalized and scrubbed rather than treated as raw production payloads.

I’m deliberately not expanding into that yet though. Right now I’m trying to prove the local workflow with real users first and solve the more immediate problems around sanitization, side-effect assertions, and fixture lifecycle.

On the edge-case question, I don’t want to invent a production story I haven’t earned yet.

So far, Spoke Hooks has caught intentional regressions in the local proof, while several developers here have described real failures around duplicate delivery, idempotency, out-of-order events, and payload/version drift.

Getting the first real user to come back and say “this fixture caught something my normal tests missed” is actually one of the signals I’m trying to reach next.

This is exactly the direction I’m leaning now.

I want the HTTP-only path to stay zero-config, but the repeated feedback is making it clear that the useful regression oracle has to be able to observe side effects.

The smallest version I’m considering is an optional project-defined assertion hook that returns a serializable snapshot of application state. baseline would store that snapshot, and test would compare it after replay. That keeps Spoke Hooks database-agnostic while still covering things like row counts, idempotency records, balances, or outbound-call summaries.

The signature-verification point is important too. I don’t want to pretend stale production signatures will work on replay. I’m going to inspect and define that behavior explicitly before claiming real-handler compatibility.

My current instinct is that replay should be able to re-sign the stored payload locally using a configured test webhook secret, rather than asking users to disable verification.

And yes, replaying the same fixture twice should probably become a first-class idempotency scenario rather than something users have to script manually.

Would a generic snapshot/assertion hook plus local re-signing be enough for you to try this on a real handler, or would you expect outbound-call recording to be part of the first usable version too?

Thanks — appreciate it!

Webhook edge cases are exactly what pushed me toward using real event fixtures instead of relying only on clean mocks.

I’m keeping the beta intentionally small for now and trying to learn from real usage before expanding it.

That’s a good question.

Right now, V0 intentionally keeps fixtures as plain JSON in the repo, so there’s no built-in fixture versioning beyond Git history.

I’m starting to see a broader lifecycle problem emerge though: fixtures can become stale when the handler contract or upstream payload shape changes, and blindly re-baselining them can hide the very regressions the tool is supposed to catch.

I don’t want to jump straight into a versioning subsystem yet, so I’m curious what the minimum useful behavior would be for you.

Would recording metadata like the Stripe API version / capture time and warning when a fixture looks stale be enough, or would you expect explicit fixture versions tied to handler contract changes?

That’s a really good point.

I’ve been thinking mostly about fixture capture and replay friction, but fixture rot / baseline drift is a different failure mode entirely.

If teams can respond to every failure with “just update the baseline,” the regression signal eventually loses its value.

I don’t want to jump straight into building baseline-management features from one comment, so I’m curious: what would make baseline updates feel trustworthy to you in practice?

Would you expect explicit per-fixture diffs, a separate approval step, or something else that forces the developer to review exactly what changed before accepting a new baseline?

Thanks — this is exactly the kind of feedback I was hoping to get.

And yes, the PII/sanitization point has now come up repeatedly. The current release does not auto-sanitize production fixtures yet, so please don’t commit unsanitized real customer data.

If you do try it this weekend, even using a manually sanitized or realistic fixture is enough for me to learn from the workflow.

I’d be especially interested in where init → add → baseline → test feels awkward on a real codebase, and whether the baseline/replay model feels useful enough that you’d actually leave it in CI.

Thanks for giving it a real try — I’d genuinely value the feedback.

Thanks — this is very useful.

The duplicate/idempotency case has now come up from several people, and your point about async handlers adds an important constraint to the side-effect assertion design.

You’re right that an immediate assertion after the HTTP response is not enough if the handler acknowledges first and processes later. I’m looking at the assertion layer as something that may need to wait/poll for application state rather than assume the side effect is synchronous.

The scrub step is also moving from “nice to have” to a V0 requirement based on the feedback so far.

I’m keeping out-of-order delivery and API-version drift in scope as validation scenarios for now rather than expanding the implementation too quickly.

For async handlers, would a configurable assertion timeout/poll command be enough in practice, or would you expect Spoke Hooks to understand the queue/job system directly?

Yes — this is exactly the kind of case that makes status/body checks insufficient.

A duplicate delivery that still returns 200 but applies the same side effect twice would currently slip through V0.

I’m now looking at side-effect assertions as the next focused step, but I want to keep the first version generic rather than baking in a specific database.

Would something like a post-replay assertion hook be enough for you?

For example, after replaying the fixture, Spoke Hooks could run a user-provided command/script that checks your test DB state — row counts, balances, idempotency records, etc. — and fails CI if that command exits non-zero.

If that would be enough for you to wire it into CI, that’s very useful to know.

Yes — the response-only check is intentional in the current V0. It was the smallest useful regression signal for proving the replay workflow, but the feedback here is making it clear that it is too weak for idempotency failures.

I agree with the invariant you’re pointing at: receiving the same event.id again should be allowed, but replaying it should not produce an unexpected second side effect.

I’m now looking at the next step as a generic side-effect assertion layer rather than a database-specific integration. For example, after replaying an event, Spoke Hooks could run a user-provided assertion command that checks DB/application state and fails CI if the state is wrong.

Would that be enough for your use case, or would you specifically want Spoke Hooks itself to understand and baseline database state?

Exactly — I think that’s the more important test than whether the first setup works.

I know the baseline + replay workflow works technically in CI, but I don’t have external evidence yet that teams keep maintaining the fixture corpus over time. That’s one of the things I’m trying to validate before expanding the product.

My hypothesis is that it only sticks if adding a new production event is cheap enough that developers treat it like adding a regression test after a bug — not like maintaining another test system.

Have you used any fixture/snapshot-based workflow that actually became part of the team’s CI habit? I’d be interested in what made people keep it up versus abandon it after the first setup.

And if you maintain a Stripe webhook handler yourself, I’d be very interested in having you try the current workflow and tell me whether it feels like something you’d actually leave in CI.

That’s a fair point, and I agree this is a real adoption blocker for teams with security review requirements.

Right now, spoke-hooks add imports the event JSON as-is — there is no built-in redaction step yet. So today the developer is responsible for sanitizing production events before committing them.

A default scrub of known PII fields, with an explicit allowlist for fields the team wants to preserve, sounds much closer to the workflow I’d want for company use as well.

I’m deliberately trying not to add features before I know they unblock real usage, so I’d like to ask one thing: if spoke-hooks add redacted known PII by default and let you keep selected business fields, would that be enough for you to actually try it in a real project?

Also, are there any fields you would definitely want preserved by default for webhook regression testing?

That’s a really good point.

Spoke Hooks isn’t limited to charge.* events — the fixture/replay flow works with generic Stripe event payloads, so payout.* and refund.* shaped events can be preserved and replayed as well.

What I haven’t validated yet is the reconciliation-specific failure mode you’re describing. V0 currently compares the webhook handler’s HTTP status and response body, so if the handler still returns the same response but silently produces the wrong ledger/reconciliation state, that would not be caught yet.

I also haven’t personally seen that exact payout/reconciliation breakage in the wild yet, so this is exactly the kind of case I’m trying to learn from.

If you have a sanitized payout/refund example, I’d be very interested to know what output you would actually want to assert on — ledger rows, fee breakdowns, exported reconciliation data, or something else. That would help me understand whether the useful regression boundary is the HTTP response or the downstream accounting effect.