Network connectivity issues

Incident Report for Subbly

Postmortem

Yesterday our data centre had issues with network connectivity which affected connections between servers. This may have led to data loss and duplicate data issues.

We’re still reviewing the impact of our data centre networking issues, we will reach out to anyone who we believe was affected. Please let us know if you notice anything that doesn’t seem right.

We’re also reviewing ways in which we can avoid impact if this situation were to happen again.

Apologies for the inconvenience.

Best wishes,
Subbly Team

Posted Apr 07, 2023 - 11:02 UTC

Resolved

All our services are currently working properly.

Our team has been working tirelessly to ensure that our services remain stable and reliable, and we will continue to monitor the situation closely to maintain our high standards of service quality. If you encounter any issues or have any concerns, please do not hesitate to contact our support team. We are always available to assist you and provide the necessary support.

Posted Apr 07, 2023 - 03:25 UTC

Investigating

We're experiencing an incident with externally managed Kubernetes clusters, customers may see increased error rates when using the website. We're investigating and will be back to you shortly. Sorry for any inconvenience, we're on the case.

Posted Apr 06, 2023 - 18:52 UTC

This incident affected: Core Subbly Services (Admin, Checkout, Billing Engine, Legacy API, Feeds, Webhooks, Automations & Jobs, Emails & Communication).