On Friday around 9am GMT the checkout and admin services were affected for approximately 30 minutes. As soon as the issue was brought to our attention we reverted the cause of the issue.

First of all I’d like to personally apologise on behalf of the Subbly team, this is less than ideal timing given its the busiest time of the year.

The cause of the issue was due to a minor deployment that included a slight change to some middleware that led to requests not having the correct information. More specifically there was a misconfiguration on the production servers which led to this undetected behaviour until it was live on production.

Our devops team are already working on a plan for the next iteration of our server architecture to handle continual scaling, a part of this plan includes streamlining deployment processes as well as ensuring the same environment state for all including local and staging. We have elevated the priority on the deployment processes and state management as we continue to work on the plan for this.

In addition to this we have update the configuration on the production servers and the deployment was re-released successfully on Monday at approx 11am GMT.

Apologies again and thanks for choosing Subbly.