UPI-style payments, in the US

Asking a neighbour to download an app and link a bank account in order to hand over fifteen dollars is where the flow dies. So the payer never installs anything. The kid shows a QR code, the payer lands on a page that already knows the gig and the amount, and a phone number and a card finish it. About a minute, start to finish.

What made it work was treating the payer as a different kind of account rather than a normal user with fewer permissions. Signup is a phone number, and a session created that fast can only reach payment endpoints. If that person later installs the app with the same number, the account is upgraded in place, with their card, history and schedules intact and no second account to reconcile.

The alternative, which is what you build if you are not careful, is a normal account with a flag on it. Then every endpoint in the product has to remember to check the flag, and the interesting question about a security bug becomes which endpoints forgot. Making it a different kind of session moves that question from every endpoint to one place.

A double tap on Pay, a browser retry, the same webhook delivered twice, the scheduler running on two servers at once. Every one of those is an ordinary Tuesday and every one can charge somebody twice.

The answer in each case was to make the database refuse it rather than check for it in code. A request carries a key and a unique constraint claims it, so checking and claiming are one atomic operation instead of two with a race between them. Two people paying the same QR contend for a row lock. The same rule covers the recurring charges a schedule produces, where the key comes from the schedule and the occurrence number, so a given occurrence can only be charged once however many times a job retries.

The scheduler gets the same treatment. Two instances waking up at the same moment take different batches of due schedules rather than both taking the first one, which is a property of how the rows are claimed rather than an agreement between the instances not to overlap.

The subtle one is a timeout. It does not mean the payment failed, it means nobody knows, and treating unknown as failed and charging again is exactly how people get charged twice. Those are left for a reconciliation job to resolve against the provider a few minutes later.

Payments fail constantly and most of the failures are boring. What is not boring is that the right response is completely different depending on which kind it is, and that the wrong response to one of them charges a customer twice.

So every error, from the provider or from our own code, was sorted into one of four categories, and the category decided what happened rather than a piece of handling written next to whichever endpoint hit it. A timeout or a provider error is retryable, with backoff, using the same key so a retry cannot become a second charge. A declined or expired card needs the payer to do something, so it fails and tells them what. An expired link or a mismatched amount is invalid and is rejected without a retry, because retrying it will never work.

The fourth category is the one worth having the argument about: unknown. The request may have reached the provider before the connection dropped. Nothing is retried, nothing is marked failed, and the record is left for reconciliation to settle against the provider. Every double charge I have seen described anywhere started with somebody putting this case in the first category.

The original endpoint did everything in one database transaction: take the lock, write the row, call the banking provider, update, commit. That provider call is a network round trip, and for its entire duration the transaction held a row lock and a connection from the pool.

The median looked fine and the tail did not, because under load every request queued behind whichever provider call happened to be slow. Splitting it into two short transactions with the network call in between fixed it, and that is only safe because the attempt is already written down before the provider is called. That plus some index work took p95 down by about 30%.

The general version of this is worth carrying around. A slow endpoint and an endpoint that makes everything else slow are different problems with different fixes, and an average hides the difference between them. Nothing here got faster. It stopped being done while holding something somebody else needed.

The interesting part of payments is not moving money, it is everything built around the assumption that any step can fail halfway through and nobody will tell you.

The other lesson took two more years. The banking provider underneath all of this went bankrupt, and people using apps built on it lost access to their own money for months. Nobody in 2022 was designing for that. Your own ledger has to be the thing you trust, reconcilable against the provider at any time, and the provider belongs behind an interface narrow enough that replacing it is a project rather than a rewrite.