It wasn’t even a week ago when I wrote about a major GitHub incident. Yesterday, they had another big incident, which lasted almost eight hours. There’s a public write-up already posted, which is surprisingly quick. While I’m personally very impatient to read these, I also know that it takes time to collect and synthesize the information you need to do a good job with them. I wish they had posted this as a preliminary write-up and then done a more detailed write-up in a couple of weeks. That being said, let’s look at the write-up!

Saturation strikes yet again

The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic.

The failure mode is yet another example of saturation, a topic I’ve written about again and again on this blog. Heck, I even gave a talk on saturation a month ago.

Here’s the full paragraph on the failure mode:

The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers.

Based on this, it sounds like the failure cascade looked like:

increase in external traffic → istio sidecars saturate (concurrency limits) → HAProxy nodes saturate (flow limits) → authentication requests fail

An increase in load on the system saturated one of the components (istio sidecar), and that propagated to another component (HAProxy), whose saturation broke the auth flow.

I wish they had included an architectural diagram here, that showed the relationship between the load balancers, the service that whose Istio sidecar pod saturated, the gateway, and the services that handle auth requests. Also, the wording gives the impression that only a single sidecar pod that saturated (an Istio sidecar pod), which would be surprising, but I’m also not confident that this is what the authors intended.

Diagnostic details: missing in action

The write-up doesn’t talk about the diagnostic work of the incident responders at all, which is a shame. I can’t tell from this write-up how difficult it was for them to figure out what was happening. There were auth failures, but it doesn’t sound like there was an increase in auth traffic per se, nor was the problem caused by recent changes to the auth system, which is where I would think to look first.

As somebody who was watching the updates to the status page as the incident was happening, I was struck by how they updates alternated between “we have identified the problem” and “we are experiencing issues:

I can imagine how frustrating it must have been for the responders to think they had found and fixed the problem, only to continue to see impact.

Retries made things worse

Retries are one of the tools in our toolbox to improve availability. And, usually, retries do improve availability! But retry logic also adds complexity to a system, and adding complexity to a system can introduce new failure modes. In this incident, retries hurt rather than helped, by increasing the load on an overloaded system.

The problem was worsened by optimistic retry logic which overloaded internal load balancers.

…

Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop.

This is a great example of unexpected behavior of a subsystem whose primary purpose was to improve reliability from my conjecture on why reliable systems fail.

The Copilot Token Service sees 10X traffic

Note that there were two independent retry behaviors mentioned in the previous section:

1. optimistic retry logic against the load balancers
2. client retry logic against the Copilot Token Service

It turns out that the client retry logic was due to a previously undiscovered bug in Visual Studio Code(!), which led to one particular service (Copilot Token Service) taking longer to recover:

Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.

…

Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop. Copilot Token Service traffic increased from a normal 7–9K RPS to 70–100K RPS.

There’s no way you’re pushing out a VS Code bugfix to mitigate an incident! You’ve got to mitigate that on the server side, which is what the responders did, which brings us to the next section.

Mitigating the incident: multiple strategies

While the write-up doesn’t discuss diagnostic work, it does mention multiple mitigations that the responders undertook during the incident:

• shifting traffic from the Central US region to the Northern Virginia region
• paused HAProxy on the four saturated nodes
• changed gateway retry logic (via PR)
• blocked inbound Copilot Token Service token requests at the load balancer (returned 403s)
• gradually ramping up blocked traffic

As responders, we are always limited in our ability to intervene based on the tools that we have at our immediate disposal. It’s incredibly useful to be able to do things like selectively block traffic, or dynamically change or even disable a reliability-related subsystem. Think about how difficult it would be to block specific types of requests during an incident in your organization, and to ramp that traffic back up slowly after the system recovers. Note how the responders had to use a pull request to change the behavior of the gateway retry logic. I wonder if the failure mode made this more difficult to carry out, but the write-up doesn’t say.

“Never again” means never preparing for a novel incident

The writeup ends, as most writeups do, with some action items intended to prevent recurrence.

To prevent recurrence, our follow-up actions include:

• Correcting autoscaling policies to account for service-mesh sidecar concurrency and capacity.
• Auditing Istio request, concurrency, and scaling limits across affected services.
• Reviewing retry limits and backoff behavior across gateways and clients.
• Addressing the VS Code retry behavior that amplified Copilot token traffic.
• Improving load-balancer capacity monitoring and regional failover safeguards.

My eternal lament is that people spend too much of their focus on preventing the last incident from recurring. It’s not that I’m opposed to preventative work. It’s that I also want us to spend time on getting better at dealing with novel incidents. Engineering cycles are a finite resource, and every cycle spent on prevention is a cycle not spent on improving our ability to respond effectively to new incidents. And I promise you, you are going to face novel incidents in the future.

After all, I don’t think GitHub customers who experienced this outage take much solace in knowing that it was a different failure mode from the previous incident.

The folks at Microsoft Azure recently wrote up a post incident review for a networking issue in their West U.S region. From the included timeline, it looks like the impact was on the order of five hours. It’s a pretty short write-up, but let’s take a look at the contributors.

On 23 July 2026, a break-fix repair was initiated on an optical device to address a network reliability risk.

The first contributor mentioned in the write-up was work that was done to repair a device in their networking stack. Here I can’t help but think of the first bullet in my conjecture on why reliable systems fail. They made a change to the system in order to fix an ongoing problem, and due to a set of circumstances, things got worse rather than better.

A defect in our blast radius analysis system incorrectly expanded the scope of the repair event to include all optical devices egressing a specific datacenter. 

The second contributor mentioned was a (presumably) latent defect in their system. Note the irony of the failure mode here: I suspect this blast radius analysis system usually contributes to reliability, but in this case it hurt reliability by increasing the blast radius.

The safety validation step, which is designed to confirm that at least one of the two redundant datacenter paths remains available, ran but incorrectly concluded the operation was safe.

The third contributor mentioned was a safety check (good!) that passed even though the action was unsafe (bad!).

The checks validated each device individually rather than evaluating the aggregate effect of isolating all devices at once, a scenario that was not accounted for because the system was never designed to process a full datacenter’s worth of devices in a single request.

The reason it failed was due to an interaction with the second contributor: the blast radius being all of the optical devices egressing the datacenter. The designers never envisioned that the check would have to handle the sort of scenario that occurred as a result of the blast radius analysis system defect.

As a result, routes were withdrawn from multiple devices simultaneously, disrupting connectivity between the datacenter and the WAN – therefore impacting traffic entering or leaving the West US region.

It sounds like this change effectively disconnected the West US datacenter from the internet.

Once the route withdrawals took effect at 14:44 UTC, physical links and routing adjacencies continued to appear healthy, which initially masked the correlation between the break-fix activity and the connectivity disruption

Here we have our fourth contributor: the operators were receiving misleading signals from the system. The links and routes looked healthy, even though connectivity was broken.

The impact presented as a WAN routing anomaly, as third-party networks could not reach Azure in the region, rather than as a datacenter connectivity failure.

Our fifth contributor is another flavor of misleading signals. The symptoms presented as a routing issue between Azure and third-parties.

Although all physical work in the region was stopped, our engineers could not correlate to this recent change because the preparation activities in advance of the break-fix did not succeed, so the physical layer and traffic appeared healthy.

This is the sixth contributor mentioned in the writeup. The writing is a little oblique here, but I think what they are saying is that the repair event did not show up in their event log because the repair event didn’t actually complete. It sounds like the preparation activities were the ones that triggered the incident. But, because the repair event didn’t actually happen, the operators looking for events that correlate in time with the onset of the incident didn’t see the triggering event because it didn’t show up in the log of events. That’s my best guess, anyways.

Our automated recovery and rollback system detected the device failures, and attempted multiple retries to restore the affected devices. However, because that system depended on the same datacenter connectivity that had been disrupted, its automated rollback attempts were unsuccessful.

This is the seventh and final contributor mentioned. Azure has an automated recovery and rollback system (good!), but the failure mode in this case prevented automated rollback from succeeding (bad!).

As always, I’d love to know more about how the operators identified what the failure mode actually was, and how they traced it back to the optical device repair work.

The other day, Reginald Braithwaite posted the following toot. For posterity, I’ve also included my own response to it:

Braithwaite’s post is dripping with sarcasm, but make no mistake, incident reports written entirely by LLMs is coming. And I am not looking forward to this future.

Before I dive in here, I want to note that there is a lot of toil you need to do in order to gather the data you need to write a good incident report, and LLMs can help significantly reduce that toil. I’ve got no issues there. But there’s a world of difference between using LLMs to help you assemble the ingredients involved in writing an incident report, and using an LLM to actually write the report itself.

Braithwaite’s post is horrifying to me precisely because of the seduction of the LLM as a tool for generating an incident report. After all, you can just ask it to write the report, and it’ll do it. And that’s exactly what scares me.

There’s a famous quote by the cartoonist Dick Guindon: “Writing is Nature’s way of showing you how sloppy your thinking is“. You might think you understand a concept, but it’s only when you put metaphorical pen to paper, when you actually try to explain the concept in written words to a potential reader, that you realize how fuzzy your understanding actually is. Writing in your own words forces you to confront how much you actually understand what it is that you’re writing about. Or, as Leslie Lamport put it, “If you’re thinking without writing, you only think you’re thinking.”

Having an LLM generate the text of an incident write-up bypasses this thinking step. Now there’s no human in the loop of the writing process that has to confront whether the explanation is actually consistent with the evidence that they’ve gathered. Instead, what you get is a plausible explanation of what happened to someone who is not intimately familiar with the details. They might read, nod along, and think, “yes, that makes sense.” But the LLM may have invented couplings between systems that aren’t there, and may miss critical interactions that were actually part of the incident, and because nobody did the hard work of actually synthesizing the data to do the write-up, nobody will notice. Because if you’re trying to reduce the writing effort, how much effort are you really going to put into checking the LLMs work.

In my view, LLM-generated incident write-ups are more dangerous than using LLM for coding or for AI SRE style tasks. For coding tasks, there’s always a testing step to check that the code exhibits the desired behavior, even if nobody looks at the code itself for meaningful details. For AI SRE tasks, either the LLM output helps you resolve the incident, or it doesn’t. In both cases, Nature is the ultimate arbiter of the LLM output.

But incident write-ups aren’t like that. The consequences of a poor report aren’t immediately apparent the way incorrect code or an incorrect operational diagnosis are in the moment. Instead, we get incident reports that have the superficially correct form, but are actually incorrect, with no obvious test for correctness.

And, because incident reports are time-consuming to write, the temptation to use AI tools to generate them will be overwhelming. But these LLMs will not go around talking to people that were involved in the incident. These reports will be simulacra; they will have the right form, but they will not provide readers with genuine insights into the nature of the system. The amount of learning will be significantly curtailed.

And, yes, people will probably use AI to summarize them as well.

It’s not a future I’m looking forward to.