Checkout facilities and admin analytics downgraded
Incident Report for Subbly

On Friday around 9am GMT the checkout and admin services were affected for approximately 30 minutes. As soon as the issue was brought to our attention we reverted the cause of the issue.

First of all I’d like to personally apologise on behalf of the Subbly team, this is less than ideal timing given its the busiest time of the year.

The cause of the issue was due to a minor deployment that included a slight change to some middleware that led to requests not having the correct information. More specifically there was a misconfiguration on the production servers which led to this undetected behaviour until it was live on production.

Our devops team are already working on a plan for the next iteration of our server architecture to handle continual scaling, a part of this plan includes streamlining deployment processes as well as ensuring the same environment state for all including local and staging. We have elevated the priority on the deployment processes and state management as we continue to work on the plan for this.

In addition to this we have update the configuration on the production servers and the deployment was re-released successfully on Monday at approx 11am GMT.

Apologies again and thanks for choosing Subbly.

Posted Dec 07, 2020 - 12:13 UTC

Incident has been resolved. Checkouts and admin analytics are back up operational. Apologies for the inconveniences!

We will be writing a postmortem explaining what happened soon.
Posted Dec 04, 2020 - 11:09 UTC
The issue has been identified and a fix is being implemented.
Posted Dec 04, 2020 - 10:58 UTC
We're currently experiencing some problems with the admin analytics and checkout facilities due to latest deployments. Working on a quick resolution and reverting to the previous state.
Posted Dec 04, 2020 - 10:55 UTC
This incident affected: Core Subbly Services (Checkout).