We would like to apologise for any inconvenience caused from the downtime this morning.
At around 4:30am GMT the core Subbly services came offline (website hosting was unaffected). Our team were immediately notified and sprung into action. We identified a failing API request to a third party which had come offline.
Our engineering team implemented a temporary hot fix as quickly as possible by reducing the timeout for the offending api call and this brought services back online but with slow loading times.
We identified that the third party had been offline for an entire 6 hours due to a major data centre outage. It didn’t affect us until 5+ hours later due to caching which eventually expired as expected (it’s almost unheard of for something to be down for 6 hours so we hadn’t planned for this).
We decided to refactor the code for the API call and moved into an asynchronous process which will never affect the uptime of the platform in future. This issue has now been considered permanently resolved.
Thank you for choosing Subbly, and apologies again for any inconvenience or stress caused.
Keep rocking on.
-The Subbly Team