And we're back to 100% online. API and checkout included.
We will continue working on the more robust solution to mitigate a similar situation happening again.
The issue was related to either physical hardware failure by our supplier, or an overloaded node on the server.
Sorry for any inconvenience caused.
Posted Nov 11, 2024 - 19:04 UTC
Update
Good news hotfix is underway, we should have the checkout and API fully back online in 5 minutes
Posted Nov 11, 2024 - 18:56 UTC
Update
Unfortunately the hot fix is proving tricky, we're unable to successfully upgrade the server directly, we believe the issue stems from a physical hardware issue. We've started working on the permanent fix immediately and will continue to try finish the hotfix.
New ETA is 1 hour.
Apologies for any inconvenience.
Posted Nov 11, 2024 - 18:49 UTC
Update
Putting into maintenance mode now. ETA 25 minutes from now.
Posted Nov 11, 2024 - 18:20 UTC
Identified
Our team has identified one of our servers which handles a critical job queue is underperforming. This job queue is required for the API which the checkout depends on is underperforming.
We're performing a hotfix which will require 15 minutes of downtime affecting the checkout. We're preparing for this downtime just now. We will update here when the checkout is officially offline.
We have already planned a permanent fix for this essential componenet to ensure stable performance for BFCM and the holidays, which will be actioned after the hotfix.
Sorry for any inconvenience. We'll be right back.
Posted Nov 11, 2024 - 18:03 UTC
This incident affected: Core Subbly Services (Checkout) and API.