tag:trust.cloudcheckr.com,2005:/historyCloudCheckr Status - Incident History2024-03-28T08:51:19-04:00CloudCheckrtag:trust.cloudcheckr.com,2005:Incident/175636542023-06-13T17:05:47-04:002023-06-13T17:05:47-04:00AWS US-East issues impacting CloudCheckr<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>17:05</var> EDT</small><br><strong>Resolved</strong> - AWS reports that all systems are back to normal operations.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>15:44</var> EDT</small><br><strong>Identified</strong> - AWS is experiencing increased error rates in N. Virginia, resulting in 404 errors for customers trying to reach app-us.cloudcheckr.com. We are monitoring the situation.</p>tag:trust.cloudcheckr.com,2005:Incident/126391792022-11-01T07:54:13-04:002022-11-01T07:54:13-04:00Some people are receiving an error message when attempting to login to CloudCheckr CMx with Chromium-based browsers<p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>07:54</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>13:18</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>18:04</var> EDT</small><br><strong>Update</strong> - Investigation is continuing. There are some indications that going directly to the appropriate regional login URL (e.g. app-us.cloudcheckr.com) rather than through the login link available on our commercial web site may avoid the issue.</p><p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>15:58</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating reports of some people receiving a 403 error when attempting to login to CMx. This is impacting multiple CloudCheckr regions.</p>tag:trust.cloudcheckr.com,2005:Incident/126353932022-10-27T10:15:30-04:002022-10-27T10:15:30-04:00CMx Outage in CloudCheckr Federal<p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>10:15</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>09:58</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>09:42</var> EDT</small><br><strong>Update</strong> - The issue has been identified and resolved. We will continue to monitor.</p><p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>09:33</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating a CMx outage in the CloudCheckr Federal environment.</p>tag:trust.cloudcheckr.com,2005:Incident/106618142022-07-21T14:24:29-04:002022-07-21T14:24:29-04:00SSO to CMx in the CloudCheckr Federal region is failing<p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>14:24</var> EDT</small><br><strong>Resolved</strong> - This issue's cause has been identified, and as of 1pm US Eastern time this issue has been resolved.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>14:06</var> EDT</small><br><strong>Update</strong> - This issue's cause has been identified, and as of 1pm US Eastern time this issue has been resolved.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>12:36</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>12:10</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate the issue.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>10:35</var> EDT</small><br><strong>Investigating</strong> - The issue is currently being investigated.</p>tag:trust.cloudcheckr.com,2005:Incident/98211352022-04-22T04:00:23-04:002022-04-22T04:00:23-04:00CloudCheckr US Region weekly deployment - changed from 4/25/2022 to 4/22/2022 starting at 1:00 AM Eastern Time<p><small>Apr <var data-var='date'>22</var>, <var data-var='time'>04:00</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>22</var>, <var data-var='time'>01:01</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>11:55</var> EDT</small><br><strong>Scheduled</strong> - The regular weekly product update for CloudCheckr US Region has been moved from the regular Monday schedule to Friday, April 22, 2022. This is for this week only; we will return to the regular Monday maintenance window with the following deploys.</p>tag:trust.cloudcheckr.com,2005:Incident/87849042021-12-07T21:59:17-05:002021-12-07T21:59:17-05:00AWS Errors in US-EAST-1 impacting CloudCheckr<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>21:59</var> EST</small><br><strong>Resolved</strong> - The issue causing elevated error rates in AWS US-EAST-1 has been resolved, restoring CloudCheckr operations to normal.<br /><br />Further details can be found here: https://status.aws.amazon.com/</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>21:32</var> EST</small><br><strong>Update</strong> - We are continuing to monitor for further issues.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>19:54</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>15:13</var> EST</small><br><strong>Identified</strong> - The root cause of API errors is an impairment of several network devices in the US-EAST-1 Region. AWS continues to work toward resolving the issues.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>15:11</var> EST</small><br><strong>Update</strong> - AWS continues to work on the issue and reports some progress, but does not yet have an ETA for full resolution.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>14:06</var> EST</small><br><strong>Investigating</strong> - AWS is experiencing API errors in US-EAST-1. This may result in errors or timeouts as you login or navigate through CloudCheckr.<br /><br />Current status can be found directly on the AWS site https://status.aws.amazon.com/</p>tag:trust.cloudcheckr.com,2005:Incident/75463682021-07-19T09:41:00-04:002021-07-19T09:41:00-04:00GOV Region unavailable<p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>09:41</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>09:25</var> EDT</small><br><strong>Update</strong> - Service has been restored. We rolled back to 23.6.1 while we investigate the cause of issues with the 24.0 deploy.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>09:12</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:trust.cloudcheckr.com,2005:Incident/64461662021-03-03T13:21:34-05:002021-03-03T13:21:34-05:00Intermittent degraded performance in app-eu.cloudcheckr.com<p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>13:21</var> EST</small><br><strong>Resolved</strong> - Performance has stabilized and has returned to normal.</p><p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>12:39</var> EST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:trust.cloudcheckr.com,2005:Incident/55867852020-11-17T10:23:45-05:002020-11-17T10:23:46-05:00CMx Errors in Gov Region<p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>10:23</var> EST</small><br><strong>Resolved</strong> - This issue has been resolved, and all environments are fully operational. Our engineers will continue to monitor for any further issues.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>10:00</var> EST</small><br><strong>Investigating</strong> - There is an issue preventing user logins into the CloudCheckr Gov CMx environment (app-gov.cloudcheckr.com). Our team is investigating the issue.</p>tag:trust.cloudcheckr.com,2005:Incident/53077502020-10-12T11:43:39-04:002020-10-12T11:44:48-04:00CMx - US - 10-12-2020 Incident<p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>11:43</var> EDT</small><br><strong>Resolved</strong> - All CloudCheckr environments are fully operational. Our engineers will continue to monitor for any further issues.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>11:35</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>11:24</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating an issue with the CMx US region. All other regions are operational. app.cloudcheckr.com is currently operational. Our engineers are working on an incident with the CMx environment</p>tag:trust.cloudcheckr.com,2005:Incident/48828632020-08-18T13:19:59-04:002020-08-18T13:19:59-04:00We are currently investing reports of access permission errors in the CloudCheckr environment.<p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>13:19</var> EDT</small><br><strong>Resolved</strong> - The access permission issue has been resolved as of 1:13 PM ET</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>12:59</var> EDT</small><br><strong>Identified</strong> - The cause of the issue has been identified and a resolution being pursued. There is not yet an ETA for resolution.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>12:47</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:trust.cloudcheckr.com,2005:Incident/44116372020-06-25T13:33:24-04:002020-06-25T13:33:24-04:00Job Processing Delays<p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>13:33</var> EDT</small><br><strong>Resolved</strong> - All systems and jobs are back to normal as of Wed. 6/24 1 AM following software release. The continued monitoring of the environment has seen no issues since the release. CloudCheckr systems and jobs are processing and fully operational.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>14:37</var> EDT</small><br><strong>Monitoring</strong> - Several configuration changes have been put in place and we continue to monitor the situation.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>08:09</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>18:25</var> EDT</small><br><strong>Investigating</strong> - We are experiencing unusual database load that is slowing down job processing. This may extend the intervals between data updates in the CloudCheckr application.</p>tag:trust.cloudcheckr.com,2005:Incident/43992742020-06-18T07:00:00-04:002020-06-18T09:39:21-04:00User Access - CloudCheckr Federal<p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>07:00</var> EDT</small><br><strong>Resolved</strong> - Customers reported an issue preventing them from logging into the CloudCheckr Federal environment Our DevOps team isolated the issue and took corrective action. As of 7: 53 AM Eastern user access was restored to environment,</p>tag:trust.cloudcheckr.com,2005:Incident/43895832020-06-17T09:21:59-04:002020-06-17T09:33:36-04:006-17-2020 - CloudCheckr HSE<p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>09:21</var> EDT</small><br><strong>Resolved</strong> - We have isolated the incident a determined that a front end web instance was in a bad state. The instance did not "fail the health check" which caused it to appear healthy. We have taken corrective action and confirmed as of 9:05 AM Eastern users can now successfully login and access the environment.</p><p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>09:10</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>08:58</var> EDT</small><br><strong>Update</strong> - Customers are experiencing issues logging into the CloudCheckr Federal environment We’re aware of the issue and are working on it urgently. Please know our teams are working hard to take corrective action</p><p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>08:47</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating reports of an incident in the CloudCheckr environment</p>tag:trust.cloudcheckr.com,2005:Incident/34509982020-01-21T07:52:02-05:002020-01-21T07:52:02-05:0018.1 software release 21 JAN 2020<p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>07:52</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>15:43</var> EST</small><br><strong>Scheduled</strong> - The purpose of this notice is to inform you minor down time is expected with the 18.1 software release at 1:00am Eastern time (GMT-05:00), 21 JAN 2020. Typically a CloudCheckr release is designed to cause no downtime or interruption of access to the CloudCheckr application. The upcoming 18.1 release will cause an anticipated interruption to last between 1 to 1.5 hours. We apologize for the inconvenience. Following the release the CloudCheckr application will be upgraded to 18.1 and function as normal.</p>tag:trust.cloudcheckr.com,2005:Incident/34408762020-01-17T13:38:01-05:002020-01-17T13:43:05-05:00Intermittent performance issues<p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>13:38</var> EST</small><br><strong>Resolved</strong> - This issue has been resolved. Performance will continue to be monitored.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>12:18</var> EST</small><br><strong>Investigating</strong> - We are currently investigating reports of degraded performance in the US region. You may be experiencing slow login and/or page load times; our engineers are working to identify and resolve the issue.</p>tag:trust.cloudcheckr.com,2005:Incident/33415702019-12-20T13:21:50-05:002019-12-20T13:21:50-05:00Microsoft Azure CSP API Failure<p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>13:21</var> EST</small><br><strong>Resolved</strong> - This issue was resolved by Microsoft yesterday (December 19, 2019). We have monitored over the last 24 hours with no reported issues. This issue is resolved.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>15:14</var> EST</small><br><strong>Monitoring</strong> - Microsoft has reported that their billing issue has been resolved. CloudCheckr will continue to monitor over the next 24 hours.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>13:54</var> EST</small><br><strong>Identified</strong> - Azure Impact Only (AWS is not impacted)<br /><br />You may have noticed a disruption in your CloudCheckr Azure Billing Collector. This is a result of an issue on the Microsoft side.<br /><br />Microsoft Azure’s Utilization API requests cannot be completed at this time due to an error within Microsoft Azure. The Microsoft team is actively working to fix the issue. <br /><br />For additional information, please visit Microsoft's Partner Center Status Page: https://partner.microsoft.com/en-US/support/csp/service-health-status<br /><br />We will continue to provide updates as we are aware.</p>tag:trust.cloudcheckr.com,2005:Incident/32329682019-11-21T08:45:38-05:002019-11-21T08:45:38-05:0011-19-2019 Degradation of email delivery<p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>08:45</var> EST</small><br><strong>Resolved</strong> - Email delivery has been restored in all CloudCheckr Regions</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>15:51</var> EST</small><br><strong>Update</strong> - Email delivery has been restored in the CloudCheckr ANZ - Australia & New Zealand Region.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>15:27</var> EST</small><br><strong>Update</strong> - Email delivery has been restored in the CloudCheckr European Region.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>09:14</var> EST</small><br><strong>Identified</strong> - CloudCheckr has identified this issue and plans to reactivate email deliveries later today, following our latest release.</p><p><small>Nov <var data-var='date'>19</var>, <var data-var='time'>12:31</var> EST</small><br><strong>Investigating</strong> - CloudCheckr is currently experiencing an issue with our email generation of data. <br /><br />While our team works to correct the degradation, CloudCheckr has temporarily disabled the email job. <br /><br />During this time you will not receive emails from the CloudCheckr application. In addition, changes you make to Multi-Account Views (MAVs) will not be saved. <br /><br />We apologize for the inconvenience and will continue to provide updates.</p>tag:trust.cloudcheckr.com,2005:Incident/28723072019-08-16T14:17:51-04:002019-08-16T14:17:51-04:0008-16-2019 EU degraded performance<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>14:17</var> EDT</small><br><strong>Resolved</strong> - This issue is resolved. All instances are healthy and fully operational.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>11:41</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>10:56</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating reports of degraded performance in the EU region. Our engineers are actively identifying, addressing and correcting the situation.</p>tag:trust.cloudcheckr.com,2005:Incident/25741462019-06-22T02:00:27-04:002019-06-23T22:58:43-04:006-19-2019 Known Issue Alert<p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>02:00</var> EDT</small><br><strong>Resolved</strong> - A software patch was released to resolve the performance issue at 2pm 6/22/2019 Eastern time. As of that time, the system is operating normally. Please contact support if you experience any issues.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>12:20</var> EDT</small><br><strong>Update</strong> - UPDATE: Our engineers have identified and developed a fix to address the root cause of the application slowness. This fix is currently going through our testing process. This fix will be deployed once testing is completed, at which point we expect the system to normalize. An update will be provided once the fix has been released. We apologize for the inconvenience.<br /><br />Some customers may experience intermittent slow performance in two areas: <br />- Processing updated billing and inventory data<br />- When running certain reports within the web application.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>12:59</var> EDT</small><br><strong>Identified</strong> - Some customers may experience intermittent slow performance in two areas: (a) processing updated billing and inventory data; and (b) when running certain reports within the web application.<br /><br />The cause of this has been identified as unexpected system load. Our engineers are working to resolve this performance issue as a top priority, and we expect the system to operate normally soon. We apologize for the inconvenience.</p>tag:trust.cloudcheckr.com,2005:Incident/23249612019-04-12T13:13:07-04:002019-04-12T13:13:07-04:00EU region outage<p><small>Apr <var data-var='date'>12</var>, <var data-var='time'>13:13</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>12</var>, <var data-var='time'>12:48</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring for unusual activity. Currently, the system is accessible and functioning as expected.</p><p><small>Apr <var data-var='date'>12</var>, <var data-var='time'>12:42</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating an issue with the CloudCheckr EU environment. It is currently our highest priority and we hope to have a resolution shortly.</p>tag:trust.cloudcheckr.com,2005:Incident/23077602019-04-04T11:34:55-04:002019-04-04T11:35:10-04:004-4-2019 GOV Cloud Outage<p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>11:34</var> EDT</small><br><strong>Resolved</strong> - This incident was resolved at 11:05am. <br />An RCA, once completed, will be available upon requestion to CloudCheckr Support.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>10:30</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating an issue affecting GOV Cloud customers preventing some customers from connecting to the CloudCheckr environment.</p>tag:trust.cloudcheckr.com,2005:Incident/23077372019-04-04T10:23:08-04:002019-04-04T10:23:08-04:004-3-2019 GOV cloud outage<p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>10:23</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>10:22</var> EDT</small><br><strong>Investigating</strong> - On 4-3-2019 CloudCheckr internally discovered an issue with a single front end Web instance at 11:54 am. The CloudCheckr engineering team investigated and resolved the situation by 12:27 pm.</p>tag:trust.cloudcheckr.com,2005:Incident/18818732018-08-28T07:15:11-04:002018-11-19T11:09:43-05:008-27 EU incident<p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>07:15</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p>tag:trust.cloudcheckr.com,2005:Incident/18716872018-08-20T06:00:43-04:002018-11-19T11:09:43-05:008-20 EU region outage<p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>06:00</var> EDT</small><br><strong>Resolved</strong> - A subset of EU users were presented with a server error message when attempting to login or navigate the web UI.</p>