Billing engine partial outage

Incident Report for Subbly

Postmortem

Summary

On April 15, 2024, between 3:00 AM and 9:00 AM UTC, a critical issue was identified in our billing engine affecting approximately 3.9% of our merchants. This error led to unintended multiple charges on some customer subscriptions during the renewal process. We have contacted everyone who was affected already.

Technical details

The problem originated from an SSL networking issue within a pod on one of our servers. This anomaly prevented the pod from connecting to the necessary database, causing job processing errors where jobs were incorrectly re-attempted. The issue was traced back to a specific data center anomaly affecting the said node.

Resolution steps

Upon detection of the issue, our engineering team promptly intervened with a series of corrective measures:

Pro-active Communication: We contacted all affected merchants about the issue with the steps we were taking.
Immediate Refunds: All duplicate charges were identified and refunded to the affected customers.
Order Management: Corresponding orders linked to the duplicate charges were archived and removed from our administrative platforms to prevent further confusion.
Fee Credit: We issued credits for both Subbly and Stripe application fees involved with these transactions, which are typically non-refundable. Please note that the credit process should be completed by the following morning (16th April).

Preventive measures

In response to this incident, we have implemented a robust solution to prevent the recurrence of this specific error. Enhancements to our network and database interactions have been deployed to fortify the stability of our job processing routines.

Statement from our team

We sincerely apologize for the inconvenience this may have caused and deeply value your trust and partnership. We are committed to ensuring the reliability of our services and upholding the quality our merchants expect. Should you need further assistance or have any concerns, our support team is on standby to assist you.

We appreciate your understanding and continued support as we move forward from this incident.

Contact Information

For additional information, follow-ups, or immediate concerns, please do not hesitate to reach out to our customer support team directly.

Posted Apr 15, 2024 - 16:52 UTC

Resolved

All extra charges have been refunded. Root cause has been permanently patched. And we are working through crediting the Stripe and Subbly transaction fees which will be finished by tomorrow morning. 3.9% of merchants were affected by this issue. A postmortem will be published shortly.

We apologize for any inconvenience caused.

Posted Apr 15, 2024 - 16:32 UTC

Update

We are continuing to clean up the incorrect charges. ETA 2-3 hours from now.

Posted Apr 15, 2024 - 11:52 UTC

Identified

When the issue was identified we shut the troublesome worker/job down stopping the issue in its tracks, soon after an immediate permanent fix has been put in place.

Another more robust permanent fix is being put into place as well.

The initial scope of the damage has been identified as well and we're now working on refunding extra charges.

We will be in contact with anyone who was affected soon after with a breakdown of impact, and what we're doing about it. We will also update here with a postmortem.

Apologies to anyone affected, we understand this can be very disruptive and stressful.

Posted Apr 15, 2024 - 11:20 UTC

Investigating

We are currently having problems with some of the customer's renewals being charged multiple times. We're investigating the root of the problem and we will update you shortly on the outcome. Please stand by and sorry for any inconveniences.

Posted Apr 15, 2024 - 10:12 UTC

This incident affected: Core Subbly Services (Billing Engine).