Addition of new servers led to massive outage last week: AWS
by IANS
Updated Nov 29, 2020
AWS, which is the Cloud arm of e-commerce behemoth Amazon, said they were able to confirm a root cause which was not driven by memory pressure as earlier reported.
"Rather, the new capacity had caused all of the servers in the fleet to exceed the maximum number of threads allowed by an operating system configuration," the company said in a technical blog post.
"As this limit was being exceeded, cache construction was failing to complete and front-end servers were ending up with useless shard-maps that left them unable to route requests to back-end clusters," it added.
The capacity addition was being made to the front-end fleet of AWS.
Each server in the front-end fleet maintains a cache of information, including membership details and shard ownership for the back-end clusters, called a shard-map.
On Thursday morning, the multi-hour outage affected a large portion of the Internet.
Several apps and services posted on Twitter that they were experiencing problems with the AWS services.
Some of those were Acorns,Adobe Spark,Autodesk,Coinbase,Glassdoor,Flickr,iRobot and The Washington Post, among others.
Amazon said it mitigated the impact to the subsystem within its Kinesis Data Streams APIs and other dependable services -- responsible for the processing of incoming requests.
"We continued to slowly add traffic to the front-end fleet with the Kinesis error rate steadily dropping from noon onward. Kinesis fully returned to normal".
Amazon Kinesis enables real-time processing of streaming data. In addition to its direct use by customers, Kinesis is used by several other AWS services. These services also saw impact during the event.
Outside of the service issues, AWS experienced some delays in communicating service status to customers during the early part of the outage.
"We want to apologise for the impact this event caused for our customers," AWS said.
Related Articles
- Optical Illusion Brain Test: If you have Sharp Eyes Find the Number 442 in 20 Secs
- London-based firm Nothing to release its Phone (2)
- Covid will continue to cause mini-waves, not become seasonal yet: Scientists
- Hackers offering crypto accounts for as low as $30 on darknet
- Reddit's new feature to allow users to share its content on other platforms
- Surgical masks can help kids fight respiratory infections: Study
- LinkedIn's new AI feature to write messages to hiring team
- Microsoft introduces Xbox Game Pass' new Friend Referral programme
- Disbursed over Rs 31 cr in claims to delivery partners in FY22-23: Swiggy
- India emerging as favourable destination for clinical trials: Report