Jamie Thingelstad

This was the worst production outage I’ve experienced in Amazon Web Services, lasting over 21 hours. AWS has also published the full incident report.

Latest Update (1:18 AM PST): We have restored all traffic to Kinesis Data Streams via all endpoints and it is now operating normally. We have also resolved the error rates invoking CloudWatch APIs. We continue to work towards full recovery for IoT SiteWise and details of the service status is below. All other services are operating normally. We have identified the root cause of the Kinesis Data Streams event, and have completed immediate actions to prevent recurrence.

Previous Update (12:43 PM PST): We have restored all traffic to Kinesis Data Streams via all endpoints and it is now operating normally. We have also resolved the error rates invoking CloudWatch APIs. We continue to work towards full recovery for IoT SiteWise and Elastic Container Service; details of these services’ status is below. All other services are operating normally. We have identified the root cause of the Kinesis Data Streams event, and have completed immediate actions to prevent recurrence.

Previous Update (11:43 PM PST): We have restored all traffic to Kinesis Data Streams via all endpoints, and have resolved the error rates invoking CloudWatch APIs. We are continuing to closely monitor Kinesis and work toward full recovery of all services. We have identified the root cause of the Kinesis Data Streams event, and have completed immediate actions to prevent recurrence. Kinesis and CloudWatch are operating normally.

Previous Update (10:30 PM PST): We have restored all traffic to Kinesis Data Streams from Internet-facing endpoints, and we are continuing to incrementally restore all requests to Kinesis Data Streams using VPC Endpoints. We are also beginning to observe incremental recovery of CloudWatch metrics functionality for new incoming metrics, and working towards full recovery. The backlog of metrics will take additional time to populate.

Previous Update (9:06 PM PST): Over the past two hours, we have continued to bring more traffic in to Kinesis Data Streams, which is leading to gradual recovery of applications that use Kinesis directly, as well as dependent services within the US-EAST-1 Region. We are bringing traffic in more slowly than anticipated while we closely monitor each change to ensure continued stability. We expect that over the next few hours, we will complete restoring Kinesis Data Streams to normal operations.

CloudWatch metrics remain delayed in the US-EAST-1 Region. Once we have restored the throttles for Kinesis to previous levels, we will be restoring CloudWatch metrics functionality. We expect to see recovery of CloudWatch metrics at that stage for new incoming metrics, but the backlog of metrics may take additional time to populate.

Previous Update (6:23 PM PST): We’d like to provide an update on the issue affecting the Kinesis Data Streams API, and other dependent services, within the US-EAST-1 Region. We have now fully mitigated the impact to the subsystem within Kinesis that is responsible for the processing of incoming requests and are no longer seeing increased error rates or latencies. However, we are not yet taking the full traffic load and are working to relax request throttles on the service. Over the next few hours we expect to relax these throttles to previous levels. We expect customers to begin seeing recovery as these throttles are relaxed over this timeframe.

We got takeout from Petite León for the first time tonight and thought it was great. We enjoyed the Roasted Beets, Pollo al Carbon, Le Petite Cheeseburger and Frites. Recommend trying it for some delicious takeout.

Buttondown says that I have written 448,665 words in the first 166 issues of the Weekly Thing. The average non-fiction book has less than 100,000 words. Even if half of the Weekly Thing words aren’t mine, I’ve possibly written two books worth of words? 🤯

Preparing Wassail Tea. This is always the official beginning of the holiday season to me!

I leveled up my webinar broadcasting capabilities with the addition of a green screen. This setup worked incredibly well with Zoom for our all team meeting today.

Here is what it looks like in Zoom with a virtual background applied.

The theme for the team meeting was a live show where I was the host in the control room. It was a lot of fun to do.

Happy that Noodlesoft has made Hazel 5 a regular application, and no longer part of System Preferences!

We had a nice 4.4 mile hike around Pike Island at Fort Snelling State Park today. This is 15 mins from our house but we had never been here! We will definitely be back, maybe with snow shoes next time.

Family time with Monopoly: Yellowstone edition.

We volunteered at Feed My Starving Children today for the first time since the pandemic. They have modified the process to make it a safe as possible. We set a new post-Covid record of 133 boxes packed. Enough to feed 79 kids for a year!

I upgraded two iMacs and a MacBook Air to macOS Big Sur. One of the iMacs was not happy after upgrade but booting into rescue mode and reinstalling Big Sur fixed whatever was ailing it. My initial impression is positive. I like the UI changes and the updates to core applications. Performance seems a little slower but gets faster after apps have been used a couple times. Might be some data migrations happening in the background? Bringing Widgets and Control Center to macOS seems like a win. I have a lot fewer menu bar icons.

Delicious Barbera d’Asti. 🍷