2018-09-22T23:17:30.475070+00:00
Incident impacting Admin

Script has finished running, marking as resolved.

Status changed to
Fully Operational
2018-09-24T19:57:09.674558+00:00

Had to reset the data and we're re-running the script just now. We discovered a slight flaw in the logic due to some edge cases if you changed the product name. ETA 4.5 hours for full clean up.

Status remaining
Degraded Performance
2018-09-24T02:39:32.414813+00:00

We're currently running a script to fix any polluted orders. This should take about 4 hours. Will update here when it's finished!

Please note: some orders may still have duplicate line item charges (shipping or tax). We have added a notice to the order view to help you identify which orders are affected. You will need to do a partial refund to your customers who were affected. Again we apologise for this. This ONLY affects any orders generated during the server downtime both Subbly and Stripe experienced which was for a few hours over the last month.

Status remaining
Degraded Performance
2018-09-23T21:05:49.249649+00:00

We discovered an issue with order line items from when we rolled out the new cart functionality. We have since patched the cause and are working on a fix for the data pollution for orders. This is a separate issue where customers were possibly charged for items multiple times for specific items during the outages experienced by us and Stripe (separately).

Personal note: I just want to apologise for the inconvenience caused, and also want to thank you for your patience with us. We're working on improving things always, and sometimes these things go wrong especially when we're trying to move fast, but just know we're working on mitigating these issues and we're always here to support you. Thank you again for being a part of this journey with us - Stefan, CEO.

Status changed to
Degraded Performance
2018-09-22T23:17:30.475106+00:00
Back to Current Status