Resolved -
And we're back to 100% online. API and checkout included.
We will continue working on the more robust solution to mitigate a similar situation happening again.
The issue was related to either physical hardware failure by our supplier, or an overloaded node on the server.
Sorry for any inconvenience caused.
Nov 11, 19:04 UTC
Update -
Good news hotfix is underway, we should have the checkout and API fully back online in 5 minutes
Nov 11, 18:56 UTC
Update -
Unfortunately the hot fix is proving tricky, we're unable to successfully upgrade the server directly, we believe the issue stems from a physical hardware issue. We've started working on the permanent fix immediately and will continue to try finish the hotfix.
New ETA is 1 hour.
Apologies for any inconvenience.
Nov 11, 18:49 UTC
Update -
Putting into maintenance mode now. ETA 25 minutes from now.
Nov 11, 18:20 UTC
Identified -
Our team has identified one of our servers which handles a critical job queue is underperforming. This job queue is required for the API which the checkout depends on is underperforming.
We're performing a hotfix which will require 15 minutes of downtime affecting the checkout. We're preparing for this downtime just now. We will update here when the checkout is officially offline.
We have already planned a permanent fix for this essential componenet to ensure stable performance for BFCM and the holidays, which will be actioned after the hotfix.
Sorry for any inconvenience. We'll be right back.
Nov 11, 18:03 UTC