Building v2 of my payments backend in Go after a year on Node. Here is what v1 actually taught me.
I run 2Settle, a product with one stubborn goal: make crypto easy to spend on everyday things. Not trading, not wires. The coffee, the subscription, the bill you owe right now, paid with money you already hold.
v1 shipped on a Node stack and taught me what the product actually is. v2 is a ground-up build in Go. I want to share the reasoning, because "we rewrote it in a faster language" is the lazy version of the story and it is not what happened.
Why the language changed
The honest driver was not raw speed. It was the shape of the problem. A payment is not a request you can retry casually. It is real, final, and there is a person standing at a checkout waiting on it. The moment you take everyday spending seriously, your backend is doing many small, independent, time-sensitive things at once, and each one has money attached.
Go's model for concurrency, thousands of things happening at the same time without the code turning into a callback maze, maps almost exactly onto that. In v1 I was constantly working around the runtime to get predictable behavior under load. In v2 the runtime works with me. That is the whole trade.
Three things v1 taught me that shaped v2
- Version one is a research project you get to charge for. You do not know what the product is until real people use it in ways you did not plan. Everything I thought was a core feature on day one looked different once money moved through it.
- Boring and predictable beats clever. In payments, the exciting path is the one you never want to hit. v2 is deliberately less clever than v1 in a lot of places, and it is far easier to reason about at 2am.
- Correctness is a product feature, not a tax. Users do not see your concurrency model, but they feel it the first time a payment does exactly what they expected, instantly. That trust is the product.
What I am not doing this time
Not chasing every chain and token on day one. Not adding features v1 never validated. Not calling it done. It is still building, still being tested, shipped, and tested again.
Happy to answer anything about the Node to Go decision or building payment infrastructure the second time. What I will not get into is anything live-system specific, for obvious reasons.
"Boring and predictable beats clever"—especially when building payment infrastructure. Moving from Node to Go for better concurrency primitives and predictable behavior under load is a super solid move. Great breakdown on why correctness is a core product feature!
Thank you
"Boring and predictable beats clever"—especially when building payment infrastructure. Moving from Node to Go for better concurrency primitives and predictable behavior under load is a super solid move. Great breakdown on why correctness is a core product feature!
"Boring and predictable beats clever"—especially when handling money. Moving from Node to Go for better concurrency primitives and predictable behavior under load is a great architectural choice for a payments backend. Solid insights on built-in correctness building trust!
Worth designing against explicitly while v2 is cheap: goroutines make it easier to lose money, not harder.
Two traps that look fine in testing. If the ctx you thread into money-moving work descends from the inbound HTTP request, a client hanging up cancels your payment mid-flight — cancelled leg, no recorded outcome, and you don't know whether the provider saw it. And a bare go doThing() that panics takes the process down with whatever it was halfway through. Money work wants a context you own and a supervised worker off a durable queue.
Which is what "boring and predictable" cashes out to: persist intent, perform the side effect, record the outcome — three writes, so a crash leaves a row instead of a gap.
One thing I'd make a v2 feature rather than a v3 chore: reconciliation against the provider's own record. The failure you can't see from inside your own system is the one where you believe you succeeded and the counterparty disagrees.
Great point @jorvexis.
That challenge is exactly what the core infrastructure is built around. Idempotency keys on every transaction mean no double writes. But it goes beyond db rows: the single source of truth in my system is the ledger.
Every payment taken and every payout is registered and resolved there, and the ledger reconciles every one of them. So no guessing, even on retries.
The "not casually retryable" point resonates from a different domain — I build offline-first mobile, where the client goes dark mid-write and retries are the normal case, not the exception. The lesson that took me longest: the language matters less than making every write carry its own identity, so a retry is provably the same operation rather than a new one. Once the client generates an idempotency key per intent and the server dedupes on it, "did that go through?" stops being a guess — the retry is safe by construction. I learned this the hard way when forked client state produced duplicate operations that were each individually valid and collectively wrong. Does your v2 push idempotency keys all the way to the client, or resolve finality server-side? For payments the client half seems like where the real ambiguity lives.
The forked-state case is the interesting one, because it's also where client-generated keys quietly stop working. The key has to be persisted with the intent before the first send rather than minted at send time — if the client generates a fresh UUID on retry after a restart, you get two writes that are each individually valid and the server has no way to tell they were the same intent. Which is the exact failure you described, one layer down.
So the key ends up being part of the intent record, not part of the request.
Great framing @Studio81Labs , and you're pointing right at the seam. In v2 I split this into two layers on purpos
At the API boundary, there's a client-supplied idempotency key, scoped per client. The first request claims it; a duplicate either gets rejected while the first is still in flight, or replays the cached response once it finishes. That is the part that would anser "did that POST land" question, and yes, it sits with the client - that is the first layer.
But finality is not the client's to decide. Every real money movement is claimed server-side in the ledger with a key derived from the event itself, never from anything the client sends. The ledger is the only write path, and the claim is an atomic insert: if that key already posted, the caller aborts instead of paying out again - no double payouts.
So the client key protects the request, the ledger key protects the money. Your forked-state case is exactly why it's built this way: two individually valid calls can't settle twice, because the second loses the claim on an event key the client never owns. The client half is where the ambiguity lives, so I moved the one thing that must be unambiguous, finality, off it.
Rewriting in Go may improve reliability, but the valuable part is that v1 exposed the real domain model before you committed to a cleaner architecture. Payments backends usually fail at state transitions, retries, reconciliation, and observability, not because Node is inherently incapable.
I agree with you @SongTrailer,
The gaps in v1 are not attributed to the node per se, but there are a few things go handle better than node at this scale, and that is why it was an obvious choice for v2.
Even at the initial design, special attention was given to state transitions, retries, and reconciliation, but speed at scale is where Go's concurrency model makes a better choice for what we look at
The boring beats clever line is the one I would frame on the wall. The rewrite risk that scares me most is not picking Go over Node. It is ledger drift showing up two months later from an edge case v1 never hit. When I move something that touches money to a new stack, I run the old and new versions side by side on the same webhook stream for a while and diff the ledgers line by line before trusting the new one alone. That kind of trust is invisible until you actually compare outputs instead of assuming the new code behaves the same, and the reconciliation problem the other commenter raised sounds like the real test of whether this rewrite held up.
It is great advice you have given @eddzsh,
And I would work with that, especially since, being a mere human, one cannot foretell all possible edge cases.
Thanks for sharing your experience.
The transition from Node to Go for a crypto payments platform is a masterclass in aligning runtime behavior with the unforgiving, concurrent nature of real-world capital, proving that architectural rewrites should be driven by the shape of the problem rather than raw benchmark vanity. Acknowledging that an early version of a product is essentially a research project funded by paying users reframes technical debt, turning messy lessons into a blueprint for simpler, more predictable systems where correctness is treated as a core product feature. Yet, while Go makes the synchronous request path cleaner, the real monsters in payment infrastructure still lurk in the async underworld of late webhooks, duplicate payloads, and unexpected blockchain reorgs. Moving forward, two critical questions remain: first, how does v2 deterministically handle a transaction rollback when a network reorg invalidates a payment your system already treated as final? And second, have you fully committed to deriving user balances directly from an append-only event stream, or are you still maintaining a parallel table that risks drifting under high-concurrency pressure?
Two good ones, and they deserve straight answers instead of hand-waving.
On the reorg: I won't pretend v2 has a deterministic auto-rollback for a payment already settled. It doesn't, and I'd be suspicious of anyone claiming one, because once fiat has left for the beneficiary, rolling back is a real-world clawback, not a database operation. What v2 does is make that case vanishingly rare by construction: nothing reaches settled until confirmations clear a per-asset threshold, and I compute confirmations from the chain tip myself rather than trust a provider's count. A reorg shallower than the threshold never settled in the first place. A reorg deeper than it is a genuine loss event that lands in an admin queue for a human, and by design a reversal compensates only the fiat leg, it never tries to unwind the crypto side. So deterministic prevention up front, explicit human-driven correction for the rare tail. I'd rather admit the tail is a business problem than dress it up as an automatic transaction.
On balances: the append-only entries are the source of truth. Every value movement is an immutable double-entry row, and one Post path is the only way to write them. I do keep a balances table, but it's a derived cache, not a parallel ledger. It's updated inside the same database transaction as the entries, as an atomic balance = balance + delta under the row lock, so concurrent posts serialize instead of racing and a partial post can never be observed. It can't drift from the write path because there is no separate write path. The only drift it can take is external, a manual DB edit or a bug writing these tables directly, and a reconciliation job re-sums the entries against the cache on a timer and flags any mismatch to admins rather than silently trusting the number. So not pure recompute-on-read event sourcing, but the exact failure you're pointing at, a parallel table quietly diverging under concurrency, is what the same-transaction atomic update is there to rule out.
Moving a payments backend specifically (not just any service) from Node to Go is a meaningful decision given how sensitive that part of the stack is. Was the switch driven more by performance/concurrency needs, or was it more about type safety and fewer runtime surprises in a domain where bugs are expensive?
The move was driven mainly for perfomance gains.
For a payment solution handling meaningful concurrent transactions, Go's speed makes a better choice to solve the challenge at hand
Concurrency at the transaction level is a fair reason, though I'd have expected the win to show up as predictability rather than raw speed. Node handles I/O-bound work well enough, and a payments backend is mostly waiting on other people's networks — so the throughput ceiling usually isn't the language.
Where I'd expect Go to actually pay off is tail latency. GC pauses landing in the middle of a settlement window are the kind of thing that looks fine in averages and terrible at p99, and payments is a domain where p99 is what gets escalated.
Did the improvement show up more in the average or in the tail? And separately — a year on v1 means you'd also learned where the money bugs were. Did any of the rewrite come from wanting stricter types around amounts and currencies, or was that genuinely not part of it?
Hi, Nice to meet you
It is my absolute pleasure, we should link up to keep in touch
yeah the schema tax is real. what's kept it manageable for me: treat event shapes like migrations, never delete or rename a field, tag every event with a version, upcast old ones on read so replay always sees the current shape. and past a certain volume full replay gets slow too, periodic balance snapshots plus replay-from-snapshot is what keeps the 5 minute read honest
hi marc, I hope you are well.
the idempotency point below is the right worry for correctness. the one that got me on similar flows was afterward: a support ticket lands and someone needs to know why the balance is what it is. a current-balance table gives you the what, not the why. an append only trail of every state change makes that a 5 minute read instead of a log-diving afternoon
Agreed, and here's the sharp version: the append-only trail has to be the source of the balance, not a log sitting next to it. The moment it's a parallel audit table, it drifts from the balance exactly when you need the two to agree, because whatever bug skipped writing the balance correctly usually skipped or mangled the audit row too. Fold the events to derive the balance and "why is it this" and "what is it" can't disagree by construction.
Event sourcing is heavier up front, and for money that trade usually pays: the 2am job turns from "reconcile the audit log against reality" into "replay the events." The tax you take on is versioning your event schema forever, which never fully goes away.
"Correctness is a product feature" is the line I'd frame on the wall. One push though: the correctness that bites at 2am usually isn't the concurrent request path Go just cleaned up for you. It's the async reconciliation after the payment. A crypto payment is final on-chain, but your backend hears about it through confirmations and webhooks that land late, out of order, or twice, and every so often a reorg unwinds one you'd already treated as done. Go makes the in-flight handling clean; it can't tell you whether your ledger's view of a payment is eventually consistent with the chain's.
When we built commission tracking on Stripe, every money-correctness bug lived in that same async layer: refunds clawing back a payout, a webhook firing twice, events arriving out of order. Never the request handler. So the v2 question I'm most curious about: what's your idempotency and reconciliation model for an event that arrives twice, or never? That's where "boring and predictable" earns its keep.
This is the exact layer I care most about too, and you're right that Go doesn't help here at all. The whole async side is built on one rule: every event is applied through a write that's safe to run twice and safe to run out of order. Nothing trusts that it's seeing an event once, in sequence, or at all.
Arrives twice: duplicates collapse at the database, not in app logic. A redelivered payout webhook hits a compare-and-set on the expected status, so the second one updates zero rows and returns a no-op. A re-seen on-chain deposit inserts with on-conflict-do-nothing on (reservation, tx), so re-polling the same tx forever changes nothing and fires no second event. And the money post itself carries a server-derived idempotency key, so even if two triggers race, the payout lands once.
Out of order: state isn't assembled from event sequence. Confirmations get recomputed from the chain tip on every poll, and every transition is a compare-and-set on the from-status, so a late or stale webhook can't walk settled back to settling. It just misses and gets dropped.
Never: this is where pull beats push. Deposits are polled to begin with, so a missing webhook isn't fatal, the next poll finds it. A payout whose confirmation never lands trips a timeout job that pages admins and, importantly, does not auto-retry, because the money may be in flight. It stays settling until a human verifies with the provider. Separately, a reconciliation job re-sums the ledger entries against the cached balances and flags any drift instead of silently trusting the cache.
Reorgs: the defense is upfront. Nothing settles until confirmations clear a per-asset threshold, and I compute confirmations myself rather than trust each API's count.
Where I'll be honest about the edges: I don't yet have an automated unwind for a reorg that reverses a deposit after it already settled, that path is admin-triggered today, and the reconciliation I run is ledger-internal, not a full ledger-versus-chain reconciler yet. That second one is the next thing I want to build, and your Stripe experience is exactly the reason: the bug always lives in the async layer, so the async layer is where the boring, provable machinery has to go.
I'm curious what v2 has forced you to unlearn from v1. Was there a decision you were convinced was fundamental in the first version that you now see as solving the wrong problem?
Well,
2settle started out as a payment and remittance platform that would enable locals to spend or accept crypto as though it were fiat,
But as we dug deep, we saw that there was a gap that was completely unattended; no infrastructure helped others to solve the seemingly easier problem of allowing spending of crypto as fiat with ease,
So the v1 taught us that there is a bigger gap than just fiat-to-crypto conversion.
Thanks for taking the time to explain it. I'd enjoy continuing the conversation outside the thread if you're open to it. What's the best email to reach you on?
It is my greatest pleasure @aryan_sinh, you can reach me on x- @mosnyik, insta - @mosnyiks, LinkedIn- @mosnyik and if you prefer email- [email protected]
Thanks! I’ve just sent it over.
Looking forward to hearing your thoughts whenever you have a chance.
The v1 to v2 leap you describe is exactly where most solo founders stall — they get stuck refactoring instead of shipping. I ran into this too. What helped was a hard rule: no rewrite until the existing code actively costs more in maintenance than the rewrite would take. Saved months.
The 'only rewrite when the cost of maintenance exceeds rewriting' principle is a good approach, but we started out as a solution that allows users to easily spend crypto in a day-to-day level of transaction, but we quickly discovered that there is an even larger gap that can be filled and we have a first-mover advantage in the market - infrastructure that settles the crypto payment, so we adapted the solution into a payment engine, then we started to see the gaps with the initial inherited stack - node and Go offers a fair solutions.
Challenges like the boring predictability, speed at scale just to mention a few.