All Demandbase platforms are fully functional now. We will continue working on post-mortem and learning from this incident.
Posted Dec 22, 2021 - 10:16 PST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Dec 22, 2021 - 09:13 PST
Update
We have fully recovered from this incident in our Sales Intelligence Cloud and now running sanity checks
Posted Dec 22, 2021 - 09:12 PST
Update
We are continuing to investigate this issue.
Posted Dec 22, 2021 - 08:16 PST
Investigating
We are closely working with our AWS partner team. Below are the updates -
We are investigating increased EC2 launch failures and networking connectivity issues for some instances in a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. Other Availability Zones within the US-EAST-1 Region are not affected by this issue.
05:18 AM PST: [NDA update] We continue to make progress in restoring power to the affected data center within the affected Availability Zone (USE1-AZ4) in the US-EAST-1 Region. We have now restored power to the majority of instances and networking devices within the affected data center and are starting to see some early signs of recovery. Customers experiencing connectivity or instance availability issues within the affected Availability Zone, should start to see some recovery as power is restored to the affected data center. RunInstances API error rates are returning to normal levels and we are working to recover affected EC2 instances and EBS volumes. While we would expect continued improvement over the coming hour, we would still recommend failing away from the Availability Zone if you are able to do so to mitigate this issue.
05:39 AM PST [NDA update] We have now restored power to all instances and network devices within the affected data center and are seeing recovery for the majority of EC2 instances and EBS volumes within the affected Availability Zone. Network connectivity within the affected Availability Zone has also returned to normal levels. While all services are starting to see meaningful recovery, services which were hosting endpoints within the affected data center - such as single-AZ RDS databases, ElastiCache, etc. - would have seen impact during the event, but are starting to see recovery now. Given the level of recovery, if you have not yet failed away from the affected Availability Zone, you should be starting to see recovery at this stage.
5:55 AM PST [NDA Update] We have now restored power to all instances and network devices within the affected data center and are seeing recovery for the majority of EC2 instances and EBS volumes within the affected Availability Zone. We continue to make progress in recovering the remaining EC2 instances and EBS volumes within the affected Availability Zone. If you are able to relaunch affected EC2 instances within the affected Availability Zone, that may help to speed up recovery. We have a small number of affected EBS volumes that are still experiencing degraded IO performance that we are working to recover. Given that some hardware lost power, there is some risk of impaired hardware, which may delay recovery for a very small number of EC2 instances, which we will message to affected customers if that happens.