Webhooks and idempotency: a guide for payment teams

Most payment bugs that cost real money are not in the payment logic. They sit in the plumbing around it: a webhook processed twice, a late “pending” event that overwrites “paid”, a payout request resent after a timeout that had in fact gone through. Each one is rare, and at a few thousand payments a day, rare happens every week.

This article covers the two tools that prevent most of them: webhooks, which tell your server what happened, and idempotency keys, which stop your own retries from making it happen twice. It explains how providers actually deliver webhooks, how to write a handler that survives that, how to use idempotency keys, and the mistakes we see most often.

Why are webhooks the source of truth for payments?

Because the browser cannot be trusted to come back. A customer who pays with a card may close the tab during the 3-D Secure step. A customer who pays by bank transfer, voucher or instant payment may finish minutes or days later, on another device. The redirect to your “thank you” page tells you where the browser went, not what the provider decided.

The webhook is a server-to-server message from the provider (or your orchestration layer) to your backend, sent when the payment changes state. Stripe describes webhooks as the way to respond to asynchronous events such as a bank confirming a payment, a dispute or a recurring payment. For local methods common in our markets, such as PIX, OXXO and SPEI, asynchronous confirmation is the normal case, not the exception.

How do providers actually deliver webhooks?

At least once, with retries, and not necessarily in order. The details differ by provider:

Provider	Delivery promise	Retries	Ordering
Stripe	Endpoints may receive the same event more than once	Up to three days with exponential back-off in live mode	Not guaranteed; track event IDs, not timestamps
Checkout.com	At least once	Up to eight retries, at 5, 10, 15 and 30 minutes, 1, 4, 12 and 12 hours	May arrive out of sequence
Adyen	Duplicates share the same eventCode and pspReference	Queued for retry if no success response within 10 seconds	Check the timestamp; some events carry a sequence number
Omno	Process idempotently, keyed on the transaction	Up to 10 retries, every attempt in the delivery log	Never downgrade a final status

Two consequences follow. Your endpoint will see duplicates, so processing the same event twice must be harmless. And your endpoint will see a “pending” after a “succeeded”, so the order of arrival cannot decide the state.

How should a webhook handler be built?

A handler that survives real traffic does five things, in this order:

1. Verify the signature. Stripe signs each event with an HMAC using SHA-256 in a signature header; Checkout.com sends an HMAC in its Cko-Signature header. Reject anything that fails. Stripe’s libraries also reject events older than a default tolerance of 5 minutes, which limits replay attacks.
2. Store the raw event. Write it to a table or queue with its event ID. Stripe warns that frameworks which modify the raw body break signature checks, so verify before parsing.
3. Acknowledge immediately. Return a 2xx response before any business logic. Adyen says to acknowledge before applying business logic because errors in that logic otherwise cause failing webhooks.
4. Process from the queue. A worker reads the event, checks whether the event ID has been seen, and applies the change.
5. Apply a state machine, not an overwrite. A payment can move from created to pending to succeeded, or to failed, or later to refunded. A transition backwards, such as succeeded to pending, is ignored and logged.

Stripe’s advice on spikes is worth repeating: with synchronous processing, a burst of deliveries, for example when many subscriptions renew at the start of the month, can overwhelm the endpoint. A queue lets you process at the rate your system can take.

What is an idempotency key, and when do you need one?

An idempotency key is a unique value your system sends with a request that changes something, such as creating a payment, a refund or a payout. If the request is repeated with the same key, the server returns the original result instead of doing the work again.

You need one because timeouts are ambiguous. When a request times out, it may have failed, or it may have succeeded with the answer lost on the way back. Without a key, the safe-looking retry can charge a customer twice or pay a seller twice.

Stripe’s rules show what to expect from a well-behaved API. Keys can be up to 255 characters, and Stripe suggests version 4 UUIDs or another random string with enough entropy. The first result is saved and returned to retries, including errors. A reused key with different parameters is rejected. Keys may be removed once they are at least 24 hours old, after which the same key starts a new request. GET and DELETE requests do not need keys because they are idempotent already.

There is also an IETF Internet-Draft describing an Idempotency-Key HTTP header for making POST and PATCH requests fault-tolerant. It has expired without becoming a standard, so each provider’s own documentation remains the rulebook.

How do idempotency keys and webhooks work together?

They protect opposite directions. The key protects requests you send; the webhook handler protects messages you receive. A safe flow for one payout looks like this:

• Create a payout record in your own database with your reference and a new key.
• Send the request with the key. On a timeout, resend with the same key.
• Mark the payout as sent, not paid.
• Wait for the webhook, or look up the status, before marking it paid or failed.
• Only when it is confirmed failed may a new attempt, with a new key, go to another route.

Example figures, for illustration only. A marketplace sends 30,000 seller payouts a month. Its payout call times out 0.4% of the time, or 120 payouts. If a third of those had actually gone through and the system resends without a key, 40 sellers are paid twice. At an average of USD 180, that is USD 7,200 to chase back each month, plus the support load. With keys and status checks, the figure is zero.

What are the common mistakes?

• Crediting on the redirect. The browser redirect is a user-interface event. Ledger changes wait for the webhook or a status lookup.
• A new key on every retry. If the retry loop generates a fresh key, the key does nothing. Generate it once, when the operation is created.
• Using the key across providers. A key means nothing to a provider that has never seen it. Before cascading a timed-out payment elsewhere, confirm the first attempt failed. Our article on cascading and decline recovery covers which declines to move.
• Ordering by timestamp alone. Stripe notes that distinct events can share a timestamp in seconds. Use event IDs for deduplication and a state machine for order.
• No reconciliation job. Some webhooks will be lost after all retries. A daily job that looks up every payment still open past its normal window closes the gap.
• Ignoring test traffic. Make sure transactions from test providers can never credit a real balance.

How do you test it before go-live?

Replay the same event twice and check that the balance changes once. Send “succeeded” then “pending” and check the order stays paid. Make the endpoint return errors for an hour and confirm the retries are processed cleanly when it recovers. Kill the payout call mid-request and confirm the retry returns the original result. These four tests catch most of the problems above.

How does Omno handle webhooks and idempotency?

Omno sends a webhook for each transaction status change, for Pay-In and Pay-Out, in one format whichever provider processed the payment, and the webhook together with the Get Transaction Info endpoint is the source of truth for final statuses. Failed deliveries are retried up to 10 times, with every attempt visible in the delivery log, and the webhooks guide sets out the rules: respond quickly, process idempotently, never downgrade a final status, and ask the API when in doubt. The payment orchestration page shows how this fits the rest of the platform.