2019-07-17T15:48:48.590203+00:00
Incident impacting Checkout

A permanent fix has been implemented to prevent this happening again in the event of a node failure.

If you have been affected we will be in contact about the next steps to resolve any affected customers or subscriptions.

Status changed to
Fully Operational
2019-07-17T22:56:08.138725+00:00

After diving deeper, affected customers appear to be 26 in total and affected orders in the region 50.

We will be following up shortly with news on permanent solution and route to resolution for the above - whether manual or programmatic.

Status remaining
Degraded Performance
2019-07-17T17:04:47.377054+00:00

Issue has been resolved with temporary fix, but please stand by for fix / recommendation for affected subscriptions/orders. Permanent fix is being implemented.

Status remaining
Degraded Performance
2019-07-17T16:16:23.488778+00:00

Upon further investigation this issue was specifically relating to rendering of emails. This means some orders being marked as shipped by old order management system and ShipStation may have failed. Estimated impact below.

Total potentially affected new subscriptions/checkouts: 89

Total potentially affected ShipStation/batch orders marking orders as shipped : 24

Recommended and decided resolution steps and actions for the affected to be confirmed.

Status remaining
Degraded Performance
2019-07-17T16:03:55.321722+00:00

In the middle of last night (2am BST) a node had to have its power cycled due to it crashing, this is pretty routine, and nothing to be alarmed about.

However when it re-booted there was a new issue which caused some issues with a percentage of checkouts showing an error when it had actually succeeded. Some customers due to thinking the checkout had failed re-attempted multiple times resulting in multiple subscriptions.

The server has been pulled out of rotation and will be diagnosed and fixed in the coming hours.

We apologise for any inconvenience for your or your customers.

Status changed to
Degraded Performance
2019-07-17T15:48:48.590229+00:00
Back to Current Status