tag:status.subbly.co,2005:/historySubbly Status - Incident History2024-03-28T02:50:42ZSubblytag:status.subbly.co,2005:Incident/202664162024-03-15T15:37:06Z2024-03-15T15:37:06ZBilling Engine - Might delay renewals<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>15:37</var> UTC</small><br><strong>Resolved</strong> - Incident has been resolved, impact was minimal. All services are running normally and optimally.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>12:53</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>12:38</var> UTC</small><br><strong>Investigating</strong> - We're currently investigating an incident with the Billing engine, there might be delays to subscription renewals. If you're facing any major issues, please contact support to report it.</p>tag:status.subbly.co,2005:Incident/199392092024-02-09T17:22:50Z2024-02-09T17:22:50ZKnown minor issues due to a major upgrade<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>17:22</var> UTC</small><br><strong>Resolved</strong> - All notable minor bugs have been squashed. If you come across anything else please do let our friendly support team know. Thanks for your patience. <br /><br />P.s. we're very excited to roll out part 2 next week!</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>14:56</var> UTC</small><br><strong>Update</strong> - Fixes are nearly in place for the minor issues. Scope of impacted services and components changed for accuracy, renewals and API/feeds were unaffected. Will close this incident by end of the day. Thanks for your patience, and please do keep reporting any bugs you come across (as always).</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>17:49</var> UTC</small><br><strong>Identified</strong> - Fixes are under way.<br /><br />The one-time product inventory issue affecting some edge cases, will require some temporary workarounds (using Automations) until we finish rolling out the rest of the upgrade early next week. You will have received communication about this already if it's affecting you.<br /><br />Please report any bugs or issues you come across.<br /><br />Thanks for your patience and cooperation, this will be worth it.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>17:45</var> UTC</small><br><strong>Update</strong> - Website builder removed from scope, this issue has been resolved.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>17:35</var> UTC</small><br><strong>Update</strong> - Updated scope of impact, for accuracy.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>17:34</var> UTC</small><br><strong>Update</strong> - Updated scope of impact.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>13:23</var> UTC</small><br><strong>Investigating</strong> - We're currently deploying a huge upgrade to the platform (improving the product creation login and adding facilities for creating bundles and 'subscribe and save' features to the system). During the deployment, we've noticed some inconsistencies with existing feature functionality as a result of the deployment and we're working hard to get the new features backwards compatible as we speak. <br /><br />Problems with the following functionalities have been identified so far: <br /><br />- minor problems with inventory management for one-time products<br />- billing/shipping cadence anomalies for anchored products (the ones with the set billing date and a set cut off date)<br />- minor problems with backend product changes not fetching through to the frontend if your site is hosted on Subbly<br /><br />Please report any issues you come across to the support team (use the chat widget in the admin). <br /><br />Hope that you'll get plenty of benefit from the new upgrade and brand new features!</p>tag:status.subbly.co,2005:Incident/199326372024-02-08T07:00:56Z2024-02-08T07:00:56ZService maintenance<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>07:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>06:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>18:02</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>18:02</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>Scheduled</strong> - We will be deploying some improvements and new features to the platform, downtime of around 1 hour is to be expected.</p>tag:status.subbly.co,2005:Incident/197357152024-01-17T09:51:13Z2024-01-18T14:34:43ZCart widget unresponsive<p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>09:51</var> UTC</small><br><strong>Resolved</strong> - Problems with server load have been identified and fixed. Cart widgets and checkouts are back and operational.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>09:26</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing downgraded performance with cart widget functionality. We're working on investigating the root cause of the issue and on deploying a fix for it. Sorry for any inconveniences caused.</p>tag:status.subbly.co,2005:Incident/189533702023-10-30T16:33:14Z2023-10-30T16:33:14ZCustomer support chat and email is unavailable<p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>16:33</var> UTC</small><br><strong>Resolved</strong> - Chat widget is back and working as usual. Sorry for the inconveniences caused.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>15:08</var> UTC</small><br><strong>Monitoring</strong> - Our support chat is now coming back online and we will be working to catch back up on tickets. We sincerely apologize for he inconvenience caused, we've already implemented measures to prevent this happening again, we know how important swift responses are to you and we strive to uphold a high quality and high speed response time. We are also going to review our backup channels and processes in case this does ever happen again. Thank you for your patience.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>13:53</var> UTC</small><br><strong>Update</strong> - We are currently facing technical issues with our customer support chat and emails, we are working hard to bring them back online and expect it to take up to 24 hours. <br /><br />👉 Got an urgent question or issue? Please use the dedicated chat on our facebook group: https://m.me/ch/AbZTz1gnOkU_HlcZ/ or our Discord chat: https://subb.ly/discord whichever you prefer.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>13:52</var> UTC</small><br><strong>Identified</strong> - We are having issues with out support channels. Sorry for the inconvenience, we're working hard to bring them back online.</p>tag:status.subbly.co,2005:Incident/189552212023-10-28T21:00:53Z2023-10-28T21:00:53ZSome automations and jobs not running properly<p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>21:00</var> UTC</small><br><strong>Resolved</strong> - Fix has been deployed, all jobs are back and running. Jobs and automations that have been clogged up in the queue because of the incident were processed successfully.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>20:20</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>20:20</var> UTC</small><br><strong>Identified</strong> - We've identified problems with some of the jobs not being ran properly in the last 24 hours. As a result of that, some of the jobs haven't been or have been partially executed (most notably, customer's renewals). Problem has been diagnosed and we're actively working on fixing it.</p>tag:status.subbly.co,2005:Incident/187660222023-10-12T14:45:56Z2023-10-12T14:45:56ZCheckout facilities not working<p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>14:45</var> UTC</small><br><strong>Resolved</strong> - Fix was deployed.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>14:45</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>14:35</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>14:35</var> UTC</small><br><strong>Investigating</strong> - We're having problems with the checkouts affecting the small subset of Enterprise customers. Checkouts are currently not working for Enterprise customers due to the infrastructural changes.</p>tag:status.subbly.co,2005:Incident/186043782023-09-27T09:00:27Z2023-09-27T09:00:27ZDatabase Maintenance<p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>08:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>25</var>, <var data-var='time'>20:06</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing necessary maintenance during this time. These maintenance periods are how we continually improve Subbly’s reliability. This will take around 1 hour.</p>tag:status.subbly.co,2005:Incident/184792582023-09-12T10:30:00Z2023-09-12T11:49:23ZMain Services (Checkout & Admin) degraded performance<p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>10:30</var> UTC</small><br><strong>Resolved</strong> - We've experienced a downtime of the core features for ~12 minutes due to high database CPU usage. Cause of this was migrating old database events to the new structure (improvement that will improve logging on the admin). Problem was immediately isolated and we rectified the problem by splitting the script into smaller chunks to prevent throttling of the CPU. Problem has been fixed at 12.31PM UCT time.</p>tag:status.subbly.co,2005:Incident/172920052023-05-22T00:00:18Z2023-05-22T00:00:18ZWebsite Builder Edit Mode<p><small>May <var data-var='date'>22</var>, <var data-var='time'>00:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>00:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>15:10</var> UTC</small><br><strong>Scheduled</strong> - Edit mode of the website builder will go down for 15-45 minutes this weekend, May 20 and 21, 2023 due to planned server maintenance (you won't be able to work inside the website builder on changing/building your sites). <br /><br />The public URLs will not be affected and will be available for site visitors (your sites will still be public and unaffected).<br /><br />Our team will do its best to guarantee a fast and smooth maintenance!<br /><br />Thank you for your patience and apologies for any inconvenience that this may cause.</p>tag:status.subbly.co,2005:Incident/167727122023-04-07T03:25:37Z2023-04-07T11:02:25ZNetwork connectivity issues<p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>03:25</var> UTC</small><br><strong>Resolved</strong> - All our services are currently working properly.<br /><br />Our team has been working tirelessly to ensure that our services remain stable and reliable, and we will continue to monitor the situation closely to maintain our high standards of service quality. If you encounter any issues or have any concerns, please do not hesitate to contact our support team. We are always available to assist you and provide the necessary support.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>18:52</var> UTC</small><br><strong>Investigating</strong> - We're experiencing an incident with externally managed Kubernetes clusters, customers may see increased error rates when using the website. We're investigating and will be back to you shortly. Sorry for any inconvenience, we're on the case.</p>tag:status.subbly.co,2005:Incident/165123732023-03-15T15:34:32Z2023-03-15T15:34:32ZSlower response times of the admin<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>15:34</var> UTC</small><br><strong>Resolved</strong> - Issue has been fixed now. We had an unintentional DDoS attack which caused the problem, solution has been delivered for it. All systems are up and operational. Apologies for slower response times, you might have been experiencing problems logging in as well. For any further questions, please check in with the support team.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>15:24</var> UTC</small><br><strong>Identified</strong> - We found the source of the issue, working on fixing it right now.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>14:37</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>14:37</var> UTC</small><br><strong>Investigating</strong> - We're having some difficulties with the response times of the admin after the recent admin rollout. We're investigating and will be back with you shortly. Sorry for any inconvenience, we're on the case.</p>tag:status.subbly.co,2005:Incident/159673542023-01-29T04:00:00Z2023-01-30T15:03:33ZCheckout/admin not working due to hosting provider outage<p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>Resolved</strong> - We've been experiencing some downtime as a result of hosting server outages. Checkouts and main dashboard were offline for around 12 minutes around 5AM GMT on the 29th of January. Fix has been put in place since then and all systems are operational again. If you are still experiencing some problems please contact the support team. Apologies for any inconveniences that this may have caused.</p>tag:status.subbly.co,2005:Incident/130764272022-11-14T19:57:33Z2022-11-14T19:57:33ZLogin and dashboard services downtime<p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>19:57</var> UTC</small><br><strong>Resolved</strong> - This incident has now been resolved, a permanent patch will be in place by close of business tomorrow.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>19:38</var> UTC</small><br><strong>Monitoring</strong> - Service is back online, we’re monitoring and working on a permanent fix</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>19:27</var> UTC</small><br><strong>Investigating</strong> - Hey everyone! We're experiencing downtime with the Login and account main dashboard but working as fast as we can to normalize the service. <br /><br />Will keep you posted.</p>tag:status.subbly.co,2005:Incident/128910362022-11-13T01:00:49Z2022-11-13T01:00:49ZNetworking maintenance<p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>01:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>03:57</var> UTC</small><br><strong>Scheduled</strong> - During this window, our server provider will be performing network maintenance that may impact the response time of the platform.</p>tag:status.subbly.co,2005:Incident/126271672022-10-26T19:56:17Z2022-10-26T19:56:17ZRenewals delayed<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>19:56</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>19:37</var> UTC</small><br><strong>Investigating</strong> - We have momentarily stopped our subscription renewal workers to investigate an issue. We will be back to normal in a while and restart the queue.</p>tag:status.subbly.co,2005:Incident/106583292022-07-20T11:00:00Z2022-07-21T05:33:15ZFilters not applied on orders batch actions<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>11:00</var> UTC</small><br><strong>Resolved</strong> - Due to the recent deployment of "Stacked filters" feature, orders batch actions performed between 10:01 am and 03:07 pm UTC with a previous cached session using the old filters could have not applied any filtering to the results, causing the action to be effective on every order.<br /><br />A hotfix was deployed afterward to exclude the usage of old filters in the new functionality and prevent any misbehavior.</p>tag:status.subbly.co,2005:Incident/104537892022-07-02T03:52:46Z2022-07-02T03:55:06ZWebsite hosting affected<p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>03:52</var> UTC</small><br><strong>Resolved</strong> - The incident was fully resolved. Apologies for any inconvenience.</p><p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>03:01</var> UTC</small><br><strong>Monitoring</strong> - Mitigations have been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>02:20</var> UTC</small><br><strong>Update</strong> - We’ve identified the cause and are deploying mitigation measures. Performance may be intermittent.</p><p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>01:58</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>01:34</var> UTC</small><br><strong>Identified</strong> - The server cluster is working slower due to abnormally high traffic. We are proactively working on mitigating the performance deficiency.</p>tag:status.subbly.co,2005:Incident/99838882022-05-13T16:32:36Z2022-05-13T16:32:36ZRenewals not processing<p><small>May <var data-var='date'>13</var>, <var data-var='time'>16:32</var> UTC</small><br><strong>Resolved</strong> - Issue is resolved now. Renewals are processing again.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>14:23</var> UTC</small><br><strong>Monitoring</strong> - Issue has been identified and fix has been put in place. We're monitoring the behavior and renewals are intentionally still paused. We can expect everything to go back to normal in 1-2 hours.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>13:51</var> UTC</small><br><strong>Investigating</strong> - We're currently experiencing some intermittent problems with some customer's renewals. Renewals are currently not processing while we investigate the issue.</p>tag:status.subbly.co,2005:Incident/99664522022-05-12T07:11:22Z2022-05-12T07:11:22ZDatabase maintenance<p><small>May <var data-var='date'>12</var>, <var data-var='time'>07:11</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>05:15</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>05:07</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>13:10</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing necessary maintenance during this time. These maintenance periods are how we continually improve Subbly’s reliability. We are planning 2 or 3 more maintenance periods over the next 2 months, and will make sure to inform you well in advance.</p>tag:status.subbly.co,2005:Incident/96585322022-03-30T07:00:24Z2022-03-30T07:00:24ZDeployment of multi-currency project<p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>07:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>05:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>14:35</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>14:22</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>14:20</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.subbly.co,2005:Incident/96065412022-03-23T02:05:56Z2022-03-23T02:05:56ZWebsite hosting networking issues<p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>02:05</var> UTC</small><br><strong>Resolved</strong> - The incident was fully resolved. Sorry once again for the hiccup.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>00:56</var> UTC</small><br><strong>Monitoring</strong> - Sites are online again but the team is continuing to monitor.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>00:38</var> UTC</small><br><strong>Identified</strong> - Google Cloud is facing some networking issues that is affecting the website hosting (See here: https://status.cloud.google.com/). <br /><br />We hope to resume normal service ASAP. Sorry for any inconvenience.</p>tag:status.subbly.co,2005:Incident/91856472022-01-28T11:00:00Z2022-01-28T16:24:17ZDegraded performance of the checkouts<p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>11:00</var> UTC</small><br><strong>Resolved</strong> - Checkouts for some of the merchants were affected due to gift-voucher changes in front-end. Problem has been diagnosed and fix has been pushed to production within 2 hours from when the incident happened. We are sorry for the inconveniences caused but we're back again.</p>tag:status.subbly.co,2005:Incident/86996372021-11-27T01:35:05Z2021-11-27T04:59:28ZAdmin and checkout services downtime<p><small>Nov <var data-var='date'>27</var>, <var data-var='time'>01:35</var> UTC</small><br><strong>Resolved</strong> - Service is back online and fully operational. The issue has been permanently patched. We sincerely apologise for the inconvenience.</p><p><small>Nov <var data-var='date'>27</var>, <var data-var='time'>01:04</var> UTC</small><br><strong>Identified</strong> - Issue identified and working on deploying the fix.</p><p><small>Nov <var data-var='date'>27</var>, <var data-var='time'>00:43</var> UTC</small><br><strong>Investigating</strong> - Hey everyone! We're experiencing downtime with the Admin & Checkout but working as fast as we can to normalize the service. <br /><br />Will keep you posted.</p>tag:status.subbly.co,2005:Incident/82834072021-10-21T16:30:32Z2021-10-21T16:30:32ZSSL issues on Subbly.me domain<p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>16:30</var> UTC</small><br><strong>Resolved</strong> - This has been fixed now, sorry for any inconvenience.</p><p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>16:28</var> UTC</small><br><strong>Identified</strong> - We are aware of an issue with the subbly.me SSL, we are working on a fix just now.</p>