Summary

On April 1, 2026, between 9:32 AM and 10:13 AM Eastern Time, several secondary services that support PaymentExpress functionality experienced a disruption during a routine failover of our production environment. Core authorization and settlement systems remained available, however a subset of payment workflows were impacted as a result of issues with these secondary services.

The most notable impacted services were those that support login, transmission of emails, transmission of payment postbacks (to notify third-party systems of record of a successful payment), and the storage of payment receipt history. The full impact of these issues is described in the following section.

Impact

Post Payment Actions

• For some completed payments, follow-up actions which are typically performed immediately after the successful payment were delayed. Most notably, this impacted:

  • Postbacks of successful payment data to third-party systems of record. This temporarily prevented PaymentExpress and systems of record from remaining in sync.
  • Transmission of emailed receipts to customers.

• These delayed actions were completed later in the day after service was restored, ensuring systems of record ultimately were brought back in sync, and customers received their receipts as expected.

Login

• During the period of impact, staff were unable to log in to PaymentExpress Back-Office Reporting, and customers were unable to log in to Online Payment sites (those which require login). Staff or customers who were already logged in still had access to those sites.

In-Person Payments

• Due to the login issues, some staff members may have been temporarily unable to access PaymentExpress, preventing them from cashiering payments, or accessing reports.
• Certain in-person payment workflows that require a real-time response from the system of record during the payment process failed.
• Due to the postback issues, cashiers may have had confusion about the success of payments, leading to unnecessary reattempts of successful payments.

Online Payments

• Online payment workflows requiring login (such as creation of scheduled payments, saved payment methods) were temporarily unavailable.
• Online payment workflows that require a real-time response from the system of record during the payment process failed.
• Due to the postback issues, customers may have had confusion about the success of payments, leading to unnecessary reattempts of successful payments.

Root Cause

The incident was triggered by a problem in our data center failover process for these secondary services.

During a routine change in how traffic was routed between our data centers, an unexpected race condition in our automated failover procedure left the database cluster powering these services in an unhealthy state, resulting in errors and timeouts for the services that depend on it.

To restore service, our support team:

1. Moved traffic back to the original data center.
2. Repaired and stabilized the affected database cluster.
3. Began cleanup work to resend delayed messages, restore missing receipts, and correct any duplicate or missing records.

Action Items

To address the underlying issue and reduce the risk of similar incidents in the future, we are taking the following steps:

1. Armor our failover process against the race condition which impacted this database. We have identified the race condition, which will be fixed and thoroughly tested prior to any future data center failover.
2. Improve monitoring around failover events. While our monitoring immediately identified issues with the failover, we have identified changes which can provide our team better clarity earlier in the process, and improve our time to resolution for similar issues.
3. Improve resilience and recovery for postbacks and integrations. We are refining how we detect and recover from delayed post-payment actions so that any backlog is cleared more quickly, reducing how long affected payments remain in a confusing or incomplete state for customers and clients.

‌

We sincerely apologize for the disruption this incident caused to your operations. Ensuring the reliability and availability of PaymentExpress is our highest priority. We are committed to learning from this event and strengthening our infrastructure and procedures to better protect your services going forward.