Payment outage playbook: detect, reroute, communicate and reconcile

The first sign of a payment outage is rarely an announcement. It is a dip on a dashboard, a support message, or a colleague asking why the checkout feels slow. The next hour decides how much it costs, and most of the damage comes from confusion: nobody knows who decides, what has failed, or what to tell customers.

This article is a runbook for that hour and the days after it: detect, confirm, reroute, communicate, reconcile. Our article on single PSP risk makes the case for having a second provider. This one assumes the incident has started and you need to know what to do next.

What do real payment outages look like?

They vary more than most plans assume. These public incidents show the range.

Date	Incident	What was reported	What it teaches
June 2018	Visa, Europe	Partial failure of authorisations for around six hours, says the Payment Systems Regulator	Many payments still work, so a partial failure is easy to miss
September 2023	Square	A firewall change and DNS upgrade overloaded internal DNS. Square says its troubleshooting tools were affected too	Recovery can take far longer than the first fix
November 2024	Worldline, Italy	Worldline said gas works damaged a supplier’s cables and network. Nexi and Bancomat services were affected	Your provider’s supplier can be the failure
February 2025	ECB, T2 and T2S	The ECB says a suspected database fault was really a storage hardware failure	The first diagnosis can be wrong
April 2025	Adyen, Europe	A DDoS attack came in three waves. Adyen says it caused intermittent outages and some failed or delayed payments	Recovery may not be final
June 2026	Worldpay, UK	Worldpay cited a third-party power disruption during England’s World Cup match. Pubs told customers cash only	Outages can land on your busiest evening
August 2026	Mastercard	Mastercard said a scheduled update caused transactions to be declined	A second provider cannot fix a network

The sources do not all report the same detail, and some root causes are the provider’s own account. Even so, the pattern holds: failures are often partial, the cause is often upstream of your provider, and the picture changes during the incident.

How do you detect an outage early?

From your own data. Square’s timeline shows services unavailable at 1:54 PM ET and a status incident created at 2:47 PM ET. That is 53 minutes in which a merchant relying on the status page would not have known. Status pages confirm a problem. They rarely reveal it first.

Watch these signals per provider, method and country, and compare each with the same hour a week earlier.

Signal	What it can mean
Approval rate falls on one provider only	Provider or acquirer problem
Timeouts or 5xx errors rise	Provider or network problem, outcomes may be unknown
Latency rises before errors do	Early stage of a failure or overload
One card brand falls on every provider	Card network problem
One issuer or card range falls	Issuer problem
Payments stuck pending, webhooks slow	Status delivery problem, even if payments work
Checkout completions fall, no provider errors	Your own site, checkout or a release

Our guide to the decline dashboard covers the data and alert levels. Add a human route as well: give support a one-line way to report “customers can’t pay” to whoever is on call.

Then declare the incident. Google’s incident response guidance says to declare incidents early and often, because in its experience managed incidents are resolved faster. Declaring costs little, and you can stand down in ten minutes.

How do you confirm what has failed?

Name three roles first. Google’s model has an incident commander to coordinate, a communications lead to handle updates and an operations lead to apply the fix. In a small team, one person may hold two of these, but everyone should know who decides.

Then sort the failure, because the right response depends on where it sits.

What failed	How it looks	Will another provider help?	First action
Your checkout or integration	All providers fall together, often after a release	No	Roll back
One provider	Errors or timeouts at one provider	Yes	Move affected traffic
One method or acquiring route	One method or country at one provider	Yes, for that segment	Move that segment only
Card network	One brand fails across providers	Not for that brand	Promote other methods
Issuer	One bank or card range fails	No	Suggest another card or method
Bank rails or settlement	Payouts or settlement are late	No	Tell finance, pause what depends on them

Do not wait for certainty on small, reversible steps. The ECB’s report shows why a first diagnosis needs checking: a suspected database fault led to a decision not to fail over straight away, and the true cause was found later that day. Your rule can be simple: if a change can be undone in a minute and affects only the failing segment, make it now, and keep investigating.

How do you reroute without making things worse?

Cascading moves payments when a provider fails, but it should not be your whole response. Every payment still has to wait for the failing provider first, so take that provider out of the route once you are confident.

1. Move the smallest group that fixes the problem. One method or country is safer than everything.
2. Check the destination. Confirm that the next provider supports those payments and has room under its limits. A large spill onto a quiet provider can look like a risk event.
3. Override sticky rules. Returning customers tied to the failing provider need to move too, as our article on sticky routing explains.
4. Treat a timeout as unknown. Stripe’s documentation says that after a connection failure you may not know whether the request was received or executed, and that a 500 response is indeterminate because the operation may still have had an effect. Our guide to webhooks and idempotency covers the mechanics.
5. Hold what you cannot confirm. Do not ship an unconfirmed order. Do not mark it failed either, since it may have been paid. Put it in a queue for review.
6. Keep customers out of retry loops. Show another way to pay rather than inviting them to press the button again. Repeated attempts also count towards card network retry limits.
7. Promote what works. In a card network outage, Apple Pay and Google Pay may not help, because they usually carry a card underneath. Move other card brands, bank payments and local methods up the page.
8. Log every change with the time. You will need to undo each one later.
9. Return in steps. Adyen’s incident came in three waves, so the first good reading may not be the end. Give the recovered provider a small share and watch it before you restore the normal split.

What do you tell support and customers?

Something useful, early and often. The Payment Systems Regulator’s direction on Visa’s 2018 incident records EY’s finding that communications during it were not timely, regular or actionable. Those three words make a good standard. Square also acknowledged that merchants felt its communication frequency and support response fell short. The ECB’s report notes that participants would have liked earlier advice to stop sending messages, which came only at 13:00. Telling people what to do, or to stop doing, is often the most valuable update.

Audience	What they need	How often
Support team	What is affected, what to say, what not to promise	First update within minutes, then every 30 minutes
Customers at checkout	That the payment did not go through, and another way to pay	As soon as the checkout message can change
Customers who may be charged	Whether a charge is confirmed, and when you will update them	When you have checked
Finance	Which providers and dates are affected	At the start and the end
Marketing	Whether to pause paid traffic or a campaign email	At the start, and when it ends
Provider’s account manager	Your evidence: time, method, country, sample references	At the start

The update intervals above are suggestions, not a standard. A sample message for support could read: “Some card payments are failing because of a problem with one of our payment providers. Please ask the customer to try another method. Do not ask them to retry the same card. If they see a pending charge, take the order number and we will confirm by [time].”

Two cautions. Never tell a customer they have not been charged unless you have checked. And do not name a provider publicly until the cause is confirmed, because the fault may be upstream.

How do you reconcile after the outage?

The incident is not over when the graphs recover. Square’s timeline shows offline payments still finishing more than eight hours after service was restored, and late results are normal. Stripe retries undelivered webhook events for up to three days in live mode and does not guarantee their order, and Adyen retries for up to 30 days. Stripe also says that during incident recovery it attempts to reconcile failed operations and send webhook events for any objects created, so your systems may receive payments they never saw an API response for.

1. Mark the window. Set start and end times per provider, with a margin on both sides.
2. Export every attempt in that window, with the merchant reference and provider status.
3. Get the final status from the provider for each unknown payment. Do not rely only on webhooks.
4. Find the three problem groups. Paid at the provider with no order, order marked paid with no provider record, and two approvals for one order.
5. Resolve each group. Fulfil the order or reverse the payment. Return duplicates through the provider that took them, and tell the customer.
6. Match settlement files when they arrive. Our article on reconciliation across PSPs sets out the method.
7. Undo temporary changes. Restore routing, splits, caps and checkout order, and confirm each one against your log.
8. Run a second pass after the webhook retry windows have passed.

What should you do after the incident?

Hold a short review within a week, and write it down. Record the time to detect, time to decide and time to reroute, the failed payments, the amount recovered and the amount lost. Ask the provider for its incident report. Adyen, for example, promised a detailed post-incident review, and Visa’s was carried out independently by EY. Check your contract for service credits.

Then prepare for the next one:

1. Assign roles and deputies, with contact details outside your main tools. Square says many services used for troubleshooting were themselves affected.
2. Keep a provider contact sheet, including an out-of-hours route.
3. Write message templates for support, checkout and email.
4. Keep a second route active with a daily share of real traffic.
5. Offer more than one way to pay, so a card network failure is not a full stop.
6. Rehearse. Move a small share deliberately, and time how long the runbook takes.

How does Omno help during a payment outage?

In Omno, if a provider is unavailable or rejects the request, the payment moves to the next provider in the rule until one accepts. Your team can change routing rules and turn a provider off from the dashboard, and the hosted checkout builder changes method order and categories with no release. The transaction trail shows every attempt, cascade step, provider response and webhook, which gives you the record you need to reconcile afterwards. The smart routing and cascading page explains how the rules work.