A Cascade of Failures: A Breakdown of the Massive AWS Outage
Somewhere tonight in northern Virginia, a group of AWS administrators are probably enjoying a beverage after a very long day of troubleshooting.
Amazon Web Services suffered a cascade of failures Monday across its US-EAST-1 Region, causing multiple outages across a dizzying array of cloud services, including AWS Lambda, Amazon API Gateway, Amazon Appflow, Amazon Aurora DSQL Service and others.
As it is all too often the case, the culprit was DNS misconfiguration. Go figure.
Of AWS’ 15 regions worldwide, US-EAST-1 is probably the largest, with clusters of data centers spread across Loudoun, Prince William, and Fairfax counties. And judging from today’s outage, many of today’s largest businesses have at least a footprint in the region.
AWS is now almost fully recovered, according to the company, with the backlog of customers’ services being completed within the next few hours. Snapchat, Reddit, Venmo and other cloud services reliant on AWS are also showing recovery.
How US-EAST-1 Went Down
The problem first manifested itself aorund 3 a.m. EDT, when multiple services reported increased error rates of DNS resolution of the DynamoDB API endpoints. That problem was reported within three hours, and by 6 a.m., the staff was confident that, after a ramp-up period, services would soon be at full speed.
“We can confirm global services and features that rely on US-EAST-1 have also recovered. We continue to work towards full resolution and will provide updates as we have more information to share,” they wrote optimistically in the log at 6:03 a.m.
Almost all the services recovered, that is. Requests to launch new EC2 instances (or services that launch EC2 instances such as ECS) still got met with error rates in the US-EAST-1 region. Initially, the suspected culprit was stale caches, which needed to be flushed.
The admin team remained confident they could easily fix the EC2 problem, though two hours later, errors were still occuring when launching EC2 instances. They advised not launching instances with this region designated as the availability zone.
Worse yet, the Lambda service, shakey from the start, was starting to have significant recovery issues as well. And as the morning wore on, a pestilence of downed AWS services plagued the admin team.
More Issues With EC2
“We can confirm significant API errors and connectivity issues across multiple services in the US-EAST-1 Region,” they wrote at 10:14 a.m. They traced the problem to the EC2 internal network, which hampered with DynamoDB, SQS, Amazon Connect and other services.
The problem turned out to be the monitoring system for the load balancers that was stressing out the Lambda service.
The last message, posted at 6:48 p.m. EDT, noted that EC2 launches have been restored, though there is a two-hour backlog of work for services that require EC2 launches, such as Redshift, as well as a backlog of analytics and reporting data.
Widespread Impact on Major Online Businesses
Although only a single region was effected, it would prove to have a profound impact across many of the biggest cloud services on the internet. The Downdetector site, which reports on the availability of cloud services, saw a huge influx of outages of AWS services throughout the day, most all of them from US-EAST-1 Region.
Source: Downdetector
This in turn caused issues for the many companies relying on AWS. Downdetector reported AWS-relate3d issues today at Snapchat, Apple Music, Reddit, Venmo, Doordash, Hulu and Amazon itself. The degree to which they were impacted is presumably measured by how heavily they relied on this particular region.
