On July 8, 2026, the Australian telecom company Telstra experienced a significant outage related to an NTP issue. Telstra hired a consulting firm to do an incident review, and a couple of weeks ago, that firm released a public report: Findings Report on the July 8th Telstra Outage.

I’m using this blog post to write up my understanding of the failure mode. I found the report was often missing important details that I needed to make sense of what happened, so I tried to fill the gaps by looking at additional sources, and made my best guesses where I could. It was also more difficult for me because I don’t have domain expertise in any of the technologies involved here: I have some basic familiarity with NTP, and I have no familiarity at all with the various mobile telecom technologies involved.

Note: all block quotes in this post are from the public report unless explicitly indicated otherwise.

2010-2020 Telstra NTP system architecture

Telstra runs three NTP servers in their mobile network, geographically distributed at telecom exchanges across Australia.

These servers provide time synchronization to their clients, which could potentially be other NTP servers.

Note that some NTP clients are configured to point to multiple servers for redundancy. The diagram above shows one such client. Clients connected to multiple NTP servers will play an important role later in the incident.

According to the report, starting in 2010, Telstra ran two NTP servers at the Sydney and Melbourne exchanges. At each site, one server ran at stratum 2, and the other ran at stratum 3. The Perth exchange ran a single NTP server, at stratum 3. For their stratum 1 sources, the stratum 2 servers were clients of NTP servers hosted at Australia’s National Measurement Institute. It is the Melbourne site that will play the most significant role in this incident.

A note about arrows: NTP diagrams typically show arrows going from lower stratus to higher stratus (e.g., “stratum 1 → stratum 2″). However, I’ve drawn the arrows the other way because I’m used to the “client → server” convention of arrows directions.

I’m assuming that the NTP clients communicated only with the stratum 3 server, but there isn’t enough detail in the report to verify this was the case. I really wish they had drawn diagrams in the report!

For the purposes of this incident, the important detail here is that each stratum 3 server has a redundant stratum 2 source: all of the stratum 3 servers can use either the Sydney or Melbourne stratum 2 servers.

An upgrade leads to an old redundancy being lost, and two new redundancies added: all three changes will contribute to the incident

Six years before impact

In 2020, Telstra upgrades the timing hardware on their mobile network. With this upgrade, their NTP servers now run on Microchip SSU 2000 synchronization supply units, with the supporting vendor being a local company: Scientific Devices. This upgrade results in three significant changes to the architecture.

No more local NTP sources

Due to a limitation in the NTP timing chassis, a stratum 2 NTP server could not provide time to a
stratum 3 NTP server in the same chassis. – Findings Report on the July 8th Telstra Outage

With this upgrade, they are no longer able to have a stratum 3 NTP server running on the SSU 2000 use another NTP server on the same unit as its source. As a consequence, the Melbourne and Sydney stratum 3 NTP servers lose a level of redundancy: they now each have only one stratum 2 source.

Unfortunately, the writeup doesn’t explain why the limitation existed. I tried to figure this out by reading what SSU-2000 docs I could find. My best guess was that this was a security restriction, as described below (emphasis mine). For context: the PackeTime module is the thing that runs NTP on the SSU 2000, and NTP runs on the service port. “Independent mode” means that the two Ethernet ports of the module run independently (i.e., bonded), which I presume is how they had things set up so they could run two independent NTP servers.

In the Independent mode, the two ports are each assigned unique IP addresses and can each serve time to independent networks. From a security perspective, the important thing to note about the Independent mode is that the PackeTime Module does not support routing protocols between the two service ports or between the service ports and the maintenance port. This prevents a malicious attack on Port A (network A) to be used to send a malicious attack via Port B (network B) or vice-versa, or between Port A or B networks and the maintenance port’s network. – SSU-2000 Synchronization Supply Unit and the SDU-2000 Synchronization Distribution Unit: User Guide, Revision G – June 2008, Chapter 8, Output Module Reference Data, PackeTime Module, p274

GPS card for failover

However, while the upgrade cost them one form of redundancy, it granted them another form: each SSU 2000 unit had a GPS card in it. GPS gives you access to a high-accuracy clock, since it receives time information from multiple satellites which each have atomic clocks in them. The presence of a GPS card gave the NTP servers running inside of the SSU 2000 access to a stratum 0 source. The GPS card was therefore available as a failover time source if the stratum 3 NTP servers lost access to their (now sole) stratum 2 source.

But there is a problem lurking within this failover solution: a GPS rollover bug in the card.

Internally, GPS keeps track of the number of weeks since January 5, 1980, and it uses 10 bits to represent this count. This means that the count rolls over every 1024 weeks (roughly every 19.7 years). Some GPS devices were not originally designed to handle the rollover. The GPS card in the SSU 2000 is one of those devices. (One thing I learned here is that while the GPS rollover itself happened on April 6, 2019, devices that are vulnerable to rollover don’t necessarily roll over on the same day. For the SSU 2000, the rollover was September 18, 2022).

This means that the GPS card’s date will jump backwards in time after it rolls over. The rollover date is September 18, 2022. But just hitting that date isn’t enough: as long as the card is still running, it will behave properly. It’s only when it is powered off and on after that date that the rollover will happen.

Running the symmetric variant of NTP

There is also one additional architectural change that happens in the upgrade.

In the client/server variant, a persistent client sends packet mode 4 packets to a server, which returns packet mode 3 packets. Servers provide synchronization to one or more clients, but do not accept
synchronization from them…

In the symmetric variant, a peer operates as both a server and client. – RFC 5905

The mobile core timing architecture was changed from client/server to peering, meaning that the
NTP server in the NTP timing chassis could acquire any NTP source that it could find and select to
use any of these sources that was providing the most stable and accurate time. This introduced
uncertainty in the time sources being used in mobile core timing hierarchy. – Findings Report on the July 8th Telstra Outage

Here’s where I struggle the most with the writeup, because I’m not familiar enough with the underlying technology. RFC 5905, which describes NTP, does not use the term “peering” anywhere, but it does talk about multiple variants of the protocol, including the client/server variant and the symmetric variant. I assume peering in the report is referring to the symmetric variant.

Here’s an example of behavior which I believe can only happen with the symmetric variant. Consider the following network of five NTP servers, labeled A-E.

Now, assume that server A goes down. This means that node B no longer has a source. When running in client/server mode, C is a client of B, which means that B cannot be a client of C. However, when running symmetric mode, B can become a client of C, like this:

This behavior will eventually contribute to increasing the severity of the incident by propagating bad time data further through the network. But we’re still six years away from that.

SSU 2000 GPS rollover day comes and goes with no issues

Four years before impact

On September 18, 2022, the rollover day for the SSU 2000, nothing happens. Because the GPS cards have been running uninterrupted, they do not roll over: that only happens after being reset. In addition, the cards are not even in use, since they are only used for failover, and failover has not occurred. The problem will only manifest when the GPU card is restarted, which will not happen for years.

Melbourne NTP server cannot reach Sydney, engineers fail over to GPS card

Nine months before impact

Several months before the outage, there’s a minor incident where the stratum 3 NTP server in the Melbourne exchange has been experiencing errors attempting to connect to its stratum 2 source in the Sydney exchange:

In October 2025, personnel were working on the Melbourne NTP server due to an incident report that the NTP stratum 3 clock was losing connectivity multiple times per day to its stratum 2 source clock at Sydney, resulting in NTP alarms.

The report notes that the server used peering to dynamically identify another NTP server, which happened to be a stratum 4 source, leading the Melbourne NTP server to become a stratum 5 source. However, they don’t know anything about this server:

The incident report also noted that the Melbourne NTP had become a stratum 5 source meaning that it was receiving time from a stratum 4 source in the mobile core timing architecture. Remember, Melbourne had only one stratum 2 source and when that source was lost, it used peering to find another source which happened to be a stratum 4 client of Sydney or Perth.

According to the report authors, this meant that timing loops were already present in the system (emphasis mine)

It is critical to have at least two independent reference sources. A timing loop is when sources that
appear independent are in fact validating one another – directly, or by tracing back to a common
reference. To avoid timing loops, you must clearly define the server hierarchy and document all
relationships between servers.

…

looping was already occurring in mobile core timing when an intended NTP stratum 3 server was able to become a client of one (or more) of the clients and that a circular loop was established when the stratum 2 source was lost. 

My understanding of the topology at the time of the October 2025 incident is captured in the following diagram, where the red arrow indicates a failed connection, and the blue arrows are network edges that were not in the previous diagram. Here I’m assuming the client in question was connected to the Perth stratum 3 server, but it may just as well have been the Sydney stratum 3 server.

While this evidence indicates that looping was already happening in the network in October 2025, the looping itself had not yet contributed to an incident: that will come later.

The network engineers mitigate the incident by failing over the Melbourne NTP server to use the GPS card as its source.

To resolve this issue the GPS card was activated and connected to the stratum 3 NTP server in Melbourne as a means of replacing the lost Sydney stratum 2 source. the incident appear to have chosen this approach as the best means to address the incident. The GPS card is known as a reference source (aka stratum 0) and so this connection made Melbourne a stratum 1 NTP source, effectively the same level as the NMI stratum 1 sources in the mobile core timing architecture. This impacted the clients of Melbourne, making them stratum 2. In peering mode, these stratum 2 clients can be time servers at the same priority as Sydney and Melbourne stratum 2 who were the intended stratum 2 sources.

A reminder from the SSU 2000 vendor about the firmware update

Seven months before impact

In January 2026, the vendor supporting the SSU 2000, Scientific Devices, sends a reminder to Telstra that there is a firmware update for the SSU 2000 related to the GPS card.

Here I’ll quote the CEO’s Senate testimony (emphasis mine):

It’s still subject to our investigation, but there was definitely a reminder in January of this year that
this update needed to be done. We believe it does go back several years that it was there. To provide some context, our investigation so far has shown that it’s not a case of the teams just ignoring that update. They did consider it. The way this was designed and the way we were using this particular server and this card, this update was affecting a feature that we were not using at that time.
…
We know that our teams did get the alert of a software update. As we said, they assessed that through our investigation, we can see, and made a choice. Again, this is not an excuse; it’s putting it in context. There are many software updates and changes getting made across the network. Any software update creates a risk, and we put it into testing. In this case, they thought, given the design that was in place, it was not relevant because we were not using this equipment in the way that would trigger this risk. – (Telstra CEO) Vicki Brady, Senate Environment and Communications References Committee

It sounds like the engineer who made the call was not aware that the GPS card was a failover source. So was the use of the GPS card an in-the-moment workaround using available resources, or was it an explicit part of the architecture that the engineer making the call about firmware updates was not aware of? Or was it something else? There simply isn’t enough information to tell. The public write-up implies that it was part of the architecture:

The architecture indicated that the GPS card was available to be used as an alternative failover source for time of day.

On the other hand, we have this Senate testimony from the Group Owner, End-to-End Service Performance and Resilience of Telstra

Based on the intentional design of that SSU 2000, the GPS module is not used as part of that design. Gerard Tracey, Senate Environment and Communications References Committee

I don’t know how to reconcile those two statements, beyond “different people in the system had different understandings of the architecture”. If you think you can eliminate these sorts of mismatches in understanding in your organization, good luck to you!

One last thing I want to point out here is the CEO’s comment that “any software update creates a risk”. So, not doing updates is dangerous, but doing updates is dangerous as well! It’s easy to judge the engineer in hindsight for this decision, but I have personally seen many incidents that were triggered by performing software updates.

Impact starts: A repair triggers a poisoned clock

On July 8, 2026, Telstra network engineers took the SSU 2000 offline in Melbourne in order to fix an issue with a redundant power supply:

When the maintenance team came to do the work on this server, they were required to power it down and take it offline because they needed to replace what we call the chassis, the case that it sits in, because its backup power supply was not working. So they put in a brand-new chassis. – (Telstra CEO) Vicki Brady, Senate Environment and Communications References Committee

The report doesn’t say how long the unit was offline, nor does it discuss what happened to the Sydney stratum 3 NTP server that had Melbourne as its sole source. One thing that’s clear is that there was no system impact while the unit was down.

It was when the unit came up again that the trouble started. At 2:50 AM, the unit is powered up and comes back online. When the GPS card comes back online, the rollback issue manifests, sending its clock back 19.7 years, from July 2026 to November 2006.

The poisoned clock time spreads across the network

Since Melbourne was operating at stratum 1 and peering was enabled and Melbourne stratum 2 had been turned off by the same chassis replacement, the incorrect date propagated throughout the mobile core timing network.

Because the Melbourne NTP servers had gone down for maintenance, Sydney lost its source for its stratum 3 server. While presumably it had a GPS card to fail over to, they were configured to only fail over manually. So, unlike Melbourne, Sydney does not switch to using GPS card as a source.

However, because the NTP servers are configured to support peering, the Sydney NTP server is able to use one of its clients a source. When Melbourne comes back up, an NTP client common to the two sites becomes a stratum 2, using Melbourne as the source. The Sydney NTP server uses this client as its time source, and now the poisoned time has propagated to the Sydney site, with its clients impacted as well.

(Note: The report does not say whether the Perth NTP deployment was impacted by this, so I’m going to assume it wasn’t).

Impact: mobile users can’t call or use data

70 minutes after impact start

Starting at around 4am July 8th, 2026, when Mobility Management Entity (MME) instances began to reset, end users of Telstra’s network experienced varying levels of impact in their ability to communicate on the network.

At this point of the write-up, I’m really out of my depth, because the details of the failure are telecom-specific. I don’t even know if an MME instance is a software thing or a hardware thing. Based on this case study on Ericsson’s website, I assume that Ericsson is the vendor, and according to this Ericsson page, it sounds like it could be either one, as it says: Our latest software release can be deployed as a physical network function (PNF), as well as a virtual network function (VNF).

What I do know is that it provides a control plane function that communicates between users’ phones and the mobility network. When these instances reset, it broke mobile functionality, including emergency service calls (Triple Zero in Australia).

These impacts precluded some users from being able to make/receive voice calls, including Emergency Calls (per the Emergency Call Service Determination:2019, also referred to as Triple Zero calls), send/receive messages, or access mobile data services. Those without data services were unable to do things such as access the web, stream content and most mobile applications were unusable as well. 

During the worst of the incident, 45% of mobile users were impacted:

At its peak, this initial issue impacted approximately 45 per cent of all calls and data sessions on our mobile network. – (Telstra CEO) Vicki Brady, Senate Environment and Communications References Committee

Impact: valid IP addresses get blocked

Until the correct network date was restored, information for approximately 30,000 IP address prefixes were incrementally corrupted (written with the wrong date) by the Diameter Signalling Controller (DSC) in such a way that access attempts using those addresses were blocked from using the network regardless of available capacity.

My guess based on this is that a Diameter Signalling Controller (once again, I’m guessing they’re using an Ericsson product) is that the DSC keeps a table of session information about active IP addresses on the network. And because entries got written to the table with the rollback date, the system interpreted these as invalid and did not allow these addresses to access the network.

Note that this is a poisoned data scenario: network engineers had to fix these records before the users could regain access.

Telstra, with help from their equipment vendor, applied a workaround during the Outage and then at a later time performed repairs to remove the bad IP address prefix entries.

Omnipresent risks

A couple of weeks ago, I wrote a post titled omnipresent availability risks in cloud software. Here I’ll point out some of those risks that I see manifesting in this incident.

Problem area: saturation

This is a pretty clear example of an incident that involved saturation. The original MME instance resets don’t take out the entire mobile network, but they have the effect of reducing capacity enough that the remaining capacity is not sufficient to meet demand. From the report:

Initially users were impacted by a loss of network capacity (due to failure of MME instances) which caused some users to be disconnected from the network. The remaining capacity was insufficient to enable those users to all reconnect deterministically nor was it capable of servicing the nominal network traffic load. So essentially the network was overloaded.

Problem area: networking

Admittedly, this one is a gimme: every large-scale telecom-related outage is going to be networking related, since what telecoms provide is network access! That was no exception here.

Problem area: security

It’s unclear to me in this case whether security systems played a role at all in this incident. I hypothesized that the loss of redundancy was due to a security restriction in the SSU 2000 unit, but I don’t know for certain that’s the case. It’s also possible that the Diameter Signalling Controller blocking the IP addresses was related to a security function, but there’s not enough information.

Essential non-standard changes: mitigating an operational issue

This is a clear case where the act of mitigating an operational issue was itself a contributor to a new incident. The trigger for this incident was repairing the power supply of the SSU 2000 unit in the Melbourne exchange. The repair itself went fine: it was powering up the unit that exposed the latent bug in the GPS card.

Essential increase in essential complexity: reliability subsystem

The GPS card, which was the source of the poisoned time, was a reliability subsystem! It exists to provide failover functionality in the case where an NTP lost access to its source. And unexpected behavior of that reliability subsystem, when engaged, bit them hard.

Essential increase in essential complexity: migration

I think Telstra’s 2020 upgrade of their timing hardware qualifies as a migration. This resulted in architectural changes that created a system vulnerability, although this vulnerability would not manifest for six years.

I do appreciate how the CEO was sanguine about how the nature of these sorts of changes means that incidents are inevitable.

What I would say is that networks and environments like a mobile network are absolutely complex, and it’s evolving—any sort of complex technology environment—and certainly networks are not infallible. But our job is to make sure we are taking the actions and those steps that will mitigate as best we can. I would love to be able to sit here and say will be zero outages. The reality of a complex network environment with fast evolving technology is that no telco around the world could guarantee that. – (Telstra CEO) Vicki Brady, Senate Environment and Communications References Committee

Some parting observations

There is no NTP haiku

In my experience, NTP isn’t a technology that frequently shows up incidents, at least not compared to other networking protocols such as DNS or BGP. The authors of the public report takes Telstra to task for a lack of sufficient NTP expertise within the company, and I can imagine playing a more significant role in the telecom domain than in other cloud software domains. But I can’t help but wonder if the fact that NTP isn’t generally problematic contributes to organizations having less in-house NTP expertise.

Appalling incuriosity from the investigators on how the work gets done

the need for stronger competency and more curiosity to investigate when something looks wrong

As is always the case in these sorts of public reports, there’s a lot of ink spilled on the subject what engineers should have done in the run-up to the incident but didn’t. The bullet “Increase attention to the operation and management process discipline in all areas” pretty much says it all.

Whenever I see this sort of content in a report, my response is, “boy, the investigators really blew it.” And that’s because you can’t understand how the incident happens unless you understand how the work really gets done within the organization. The authors show an acute lack of curiosity about the circumstances of the individuals who were involved in making the decisions.

I’ll note that these authors managed to mess up the section numbers in their report: it goes 1,2,1,2 instead of 1,2,3,4. This leads them to reference a non-existent section 4.1. I would ask the authors: Do you think we should be taking advice on how to run a reliable telecom network from people who can’t even number sections correctly in a ten page document? Or do you think we should consider the circumstances that could lead to this sort of error?

Where are the other system vulnerabilities?

We observed instances where other areas and elements of the network and organizations were performing at better levels than NTP. 

Finally, it’s easy to point out problems in a system that were exposed by a recent incident. The really impressive thing, to me, is to identify the problems that were not exposed here. I have high confidence that there are other architectural vulnerabilities in the system, just waiting to be turned into incidents because of future changes. And yet, somehow, investigators never manage to find new vulnerabilities that could lead to the next outage. Funny thing, that.

References

• Findings Report on the July 8th Telstra Outage By Technology Audit Partners, August 24, 2026.
• As it happened: Telstra execs grilled at Senate hearing into outage, David Swan and Elias Visontay, Sydney Morning Herald, July 17, 2026.
• Senate Environment and Communications References Committee, Triple Zero service outage, Committee Hansard, July 17, 2026
• Triple Zero service outage Submission 7 – Supplementary Submission: Environment and Communications References Committee inquiry into the: Triple Zero service outage, July 16, 2026
• NTP servers: S200 and S300 Series Internal Week Number Rollover event on September 18, 2022, community.ntppool.org, Steve Sommars, July 27, 2022
• The 18th September 2022 Rollover Event, Chronos Technology, Louise Davies, June 9, 2022
• RFC 5905: Network Time Protocol Version 4: Protocol and Algorithms Specification, D. Mills, J. Martin, J. Burbank, W. Kasch, June 2010.
• SSU-2000 Synchronization Supply Unit and the SDU-2000 Synchronization Distribution Unit: User Guide, Revision G – June 2008

The other day, I wrote about a GitHub Actions incident. But GitHub Actions isn’t the only CI-as-a-service out there. Buildkite, which is another CI company, recently released a public write-up of an incident they experienced last week. It’s a great one: their incident report contains a lot more detail than GitHub’s. Here I’ll discuss my own observations.

The failure mode

The write-up describes what I found to be a fascinating failure mode. Here I’m going to do my best to explain it in my own words.

A bit about Kubernetes

First, some Kubernetes terminology to get us all on the same page. When you’re using Kubernetes to run your services and jobs, we say that you are running on a Kubernetes cluster. A cluster is made up of multiple nodes. If you’re running on EKS (Amazon’s Kubernetes offering), I believe those nodes are going to be EC2 instances.

Each node exists to run pods, which are the compute workloads that you run on Kubernetes. If you’ve been a service owner for a service that runs on Kubernetes, then you’re almost certainly familiar with the notion of pods, because that’s what’s exposed to you. But, unless you’ve worked in infrastructure, you’re likely not familiar with nodes and clusters, because those implementation details are deliberately hidden from the service owners.

Here’s a diagram that shows a four-node Kubernetes cluster. Each black rectangle depicts a pod that is running within that node. A pod is a set of containers that all get scheduled together (typically an application container and related sidecars), but that particular detail isn’t relevant here.

I’ve also labeled the blank part of one of the nodes as headroom. That’s not explicitly a Kubernetes concept. Rather, it’s a general term that here means “the amount of additional compute jobs that could still be run on this node.”

A few brief words on autoscaling

Let’s take the service owner’s perspective again. You own a service that runs on Kubernetes. That service is deployed as a group of pods, what Kubernetes calls a replicaset. Note that each pod is identical, so the only reason to have multiple pods is because you need additional compute resources. The amount of compute resources that your service needs can vary over time. For example, if you’re driven by requests, then the volume of requests can vary. Or, if your service consumes from a job queue, the number of pending jobs can vary at any one time.

Kubernetes deals with workloads whose resource needs vary with time by supporting autoscaling. The system I’m most familiar with is the HorizontalPodAutoscaler, which will increase or decrease the number of size of a replicaset depending on the load.

---

A brief sidebar about scaling terminology. There’s a distinction between horizontal scaling and vertical scaling. If you’re adding more servers, you’re horizontally scaling, whereas if you’re increasing the size of your servers, you’re vertically scaling. Technically, it’s more accurate to use the term scale out to refer to horizontal scaling, and scale up to refer to vertical scaling. and, indeed, the Buildkite write-up uses scale out. But I’m used to saying scale up for horizontal scaling as well, because that’s the kind of scaling I deal with more often. In this post, I use scale up and scale out interchangeably.

---

Here’s an example of a graph of a deployment that is undergoing autoscaling.

The red line indicates the desired replica count. That’s the number of pods that the autoscaler has determined should be running. At the beginning of this graph, the desired number of pods is 6. Later on, the autoscaler adjusts the desired replica count to 8.

The green indicates the number of pods that are in a running state. Note at the beginning of the graph that there are 6 pods in running state, and at the end there are 8 pods. Here’s how the Kubernetes docs describe the running state:

The Pod has been bound to a node, and all of the containers have been created. At least one container is still running, or is in the process of starting or restarting. – Kubernetes pod lifecycle docs

The gray is the interesting bit. That shows the pods that are in the pending state. Here’s how the Kubernetes docs describe the pending state:

The Pod has been accepted by the Kubernetes cluster, but one or more of the containers has not been set up and made ready to run. This includes time a Pod spends waiting to be scheduled as well as the time spent downloading container images over the network. – Kubernetes pod lifecycle docs

After the autoscaler increases the desired replica count to 8, another Kubernetes controller sees the difference between current replica count (6) and desired (8), and spins up two new pods. But those pods don’t immediately start running. Each new pod has to be scheduled onto a node in the cluster that has headroom. Until that happens, the pod is in the pending state.

A deployment consumes all of the headroom, workers try to scale up

Let’s turn back to the write-up:

At 22:44 UTC, an application deploy created a surge in application Pod volume… This surge consumed the available headroom on already-deployed Nodes.

When you deploy a new version of your software onto a Kubernetes cluster, the cluster spins up new pods with containers that are running the new version. In this case, these pods new consumed all of the available resources on all of the nodes of the cluster.

In Buildkite’s case, in addition to an application deployment eating up all the headroom, there was a replicaset of background workers that was being scaled up by the pod autoscaler at the same time. From the write-up:

At 22:44 UTC, an application deploy created a surge in application Pod volume… This surge consumed the available headroom on already-deployed Nodes, which limited applications’ capacity to autoscale promptly. Some background workers (including the aforementioned notification workers) were also attempting to scale out at this time. 

So, we have one application (background workers) trying to increase its replica count, at the same time as another application has eaten up all of the available capacity.

You can scale up your k8s cluster too, but you’ll need to wait

At 22:44 UTC, an application deploy created a surge in application Pod volume, which caused our EKS cluster to scale out. (emphasis added)

Just like a service owner can configure autoscaling for their service, a Kubernetes administrator can configure autoscaling for their cluster: when it starts to run out of headroom, the cluster autoscaler can automatically request new EC2 instances from AWS. Instead of using the horizontal pod autoscaler (which is for scaling pods), you use a cluster autoscaler, like Karpenter, which happens to have been written by AWS. But the general concept of autoscaling is the same here.

However, as we saw in the diagram above, autoscaling isn’t instantaneous: it takes time to provision new EC2 instances and add them to a Kubernetes cluster.

The headroom shortage and subsequent cluster autoscaling delayed provisioning of the compute requested by those services. Since there had been no change in the metrics that triggered the services to scale up, those services requested even more Pods.

Normally the impact of such runaway autoscaling would be limited by the services’ configured maximums. However, as mentioned previously the maximums for these services had been set higher than usual.

Here’s my best guess of the behavior they saw, based on the text.

The above diagram is a sort of schematic representation of my understanding of what happened. Some services running on the cluster scaled up, meaning the desired number of replicas increased for those services (that’s the red line going upwards). However, because the cluster was out of capacity, and the new nodes had not come online yet, the new pods were still in pending state waiting for the new capacity.

From the write-up, it sounds like the desired replica count continued to increase during this period. When configuring an autoscaling policy, you always configure a maximum number of replicas constant. This max replicas constant for the worker pool in question had previously been set to a high value when the pool was first created.

To ensure sufficient capacity for both pools, we initially configured each with the same high maxReplica count as the original shared pool, with the intention of reviewing and adjusting the limits for both pools downwards at a later date…

Normally the impact of such runaway autoscaling would be limited by the services’ configured maximums. However, as mentioned previously the maximums for these services had been set higher than usual.

Eventually, the new cluster nodes came online, and the new pods came online. This led to a large number of new pods coming online in the cluster.

(I can’t tell from the text whether the worker pool actually scaled all of the way up to max replicas or not, though).

Après autoscaling, le déluge

Buildkite services running in one of our production Kubernetes clusters depend on an internal DNS service CoreDNS to locate databases, queues, and other application components.

Once the new EKS nodes come online, the worker pool rapidly scales up. That’s not a problem for the worker pool itself. But it turns out that this rapid introduction of new compute resources into the cluster is a problem for a different service: CoreDNS.

The application deployment and runaway autoscaling combined to trigger an unusually high rate of change to applications, network endpoints, and cluster nodes.

From the write-up, it sounds like CoreDNS is the service discovery mechanism used by Buildkite. A service discovery system needs to keep track of the various resources in your system. The more churn you have in these dynamic resources, the greater the load is going to be on your service discovery system.

In this case, the step change in new cluster nodes and new application pods coming online put too much pressure on their CoreDNS service, and it fell over.

Once CoreDNS went unhealthy, the overall system was in real trouble:

CoreDNS unavailability caused failures across APIs, job dispatch, and notifications for all customers.

Like in the GitHub incident, the overloaded service wasn’t able to recover on its own. To remediate, the responders reduced the load by stopping new deploys, and then gave it more resources.

Retries and delayed work increased the load during recovery…

How we responded

We paused further application deployments, deployed more CoreDNS service replicas, raised the memory available to each CoreDNS replica, and expanded the node pool available to run them. The new set of CoreDNS Pods came into service by 23:12 UTC. After that, DNS errors fell rapidly. Customer-facing services processed the accumulated backlog and recovered fully by 23:16 UTC.

Actions taken for safety end up increasing risk

We often make operational decisions where we believe we are reducing risk. As mentioned earlier, it’s notable that the max replicas threshold was intentionally set high to reduce the risk of running out of capacity.

To ensure sufficient capacity for both pools, we initially configured each with the same high maxReplica count as the original shared pool, with the intention of reviewing and adjusting the limits for both pools downwards at a later date.

Pick a max replicas that’s too low, and your service can become overloaded: I’ve seen it many times. But picking a max replicas that’s too high created a risk that (I can only assume) nobody even conceived of in the moment. That’s the nature of the complex systems that we work with.

Control loops will drive you loopy

In one sense, this is a story of too much autoscaling, as it was a sudden, dramatic increase in scaling that led to CoreDNS being overwhlemed. But it’s also a story of not enough autoscaling: CoreDNS was not configured to autoscale at all.

The cluster’s CoreDNS service was running at a fixed size and did not automatically scale with the size or rate of change of the cluster.

An autoscaler is an example of a control loop in your system. In this particular incident, two control loops (the horizontal pod autoscaler and the cluster autoscaler) interacted in an unexpected way that led to an increase in CoreDNS, which was missing a control loop to ensure it had sufficient resources. In general, it’s difficult to reason about the behavior of interacting control loops. It’s these sorts of complex interactions that can make incidents hard to deal with.

If you’ve ever taken a course in control systems, you’ll know that stability is one of the primary concerns with building a control system, and that delays in your system can impact its stability. This was an interesting case of that where there were delays introduced because of the time required for the cluster to scale up. And, indeed, that led to that worker pool getting overscaled.

Rapid scale up as an increased load scenario

Because your application behaves differently on startup than when it’s fully warmed, if you’re in a situation when many pods are starting up, that’s a different mode of operation than your system normally deals with. Here we saw CoreDNS be the system that got overwhelmed. But any system that gets more interaction during startup than steady-state is at potential risk of being knocked over in a scenario like this.

Most load tests I’ve seen involved sending a significant amount of additional traffic to a system that’s in steady state. I’ve never seen anybody run a load test where they just dramatically scaled up a single service and watched what happened. But I bet you’d find some interesting failure modes that way that you wouldn’t find with traditional load testing.

Saturation, migration, networking, misbehaving reliability subsystems

The Buildkite incident checks off multiple boxes in my list of omnipresent availability risks in cloud software. Here, working involved in a migration (from ECS to EKS), and systems intended to improve reliability (autoscalers) resulted in saturation (CoreDNS), which impacted a networking system (once again, CoreDNS).

So many contributors

There’s a lot more detail in the write-up than I’ve covered in this post. I tried to collect the various contributors that they explicitly mention in the write-up. Here are the ones I spotted:

1. migrating from ECS to EKS
2. The EKS cluster was increasing in size during the course of the migration
3. An application deploy, which spiked the number of application pods
4. all remaining headroom in the EKS cluster being consumed by the application deploy
5. EKS cluster had to autoscale in order to make the new capacity available, this took time
6. background workers were attempting to scale out at the same time
7. metrics used for scaling background workers remaining unchanged while cluster is waiting for new capacity to come online
8. time-sensitive notification jobs had been moved from general-purpose pool to low-latency worker pool
9. high maxReplica count set on the low-latency worker pool
10. CoreDNS used for service discovery
11. large number of pods coming online led to high rate of change to apps, network endpoints, cluster nodes
12. CoreDNS runs at fixed size (does not autoscale)
13. defect in monitoring query which masked increases in CoreDNS query response times
14. Recent performance gains in application hid client-side latency increases
15. CoreDNS pods exceeded allowed memory limits, were restarted by kubernetes
16. continued demand for DNS kept load high on CoreDNS
17. DNS retries kept load high on CoreDNS

Finally, I commend Buildkite for providing so many technical details about the incident. It increases my esteem for their engineering organization.

It wasn’t even a week ago when I wrote about a major GitHub incident. Yesterday, they had another big incident, which lasted almost eight hours. There’s a public write-up already posted, which is surprisingly quick. While I’m personally very impatient to read these, I also know that it takes time to collect and synthesize the information you need to do a good job with them. I wish they had posted this as a preliminary write-up and then done a more detailed write-up in a couple of weeks. That being said, let’s look at the write-up!

Saturation strikes yet again

The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic.

The failure mode is yet another example of saturation, a topic I’ve written about again and again on this blog. Heck, I even gave a talk on saturation a month ago.

Here’s the full paragraph on the failure mode:

The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers.

Based on this, it sounds like the failure cascade looked like:

increase in external traffic → istio sidecars saturate (concurrency limits) → HAProxy nodes saturate (flow limits) → authentication requests fail

An increase in load on the system saturated one of the components (istio sidecar), and that propagated to another component (HAProxy), whose saturation broke the auth flow.

I wish they had included an architectural diagram here, that showed the relationship between the load balancers, the service that whose Istio sidecar pod saturated, the gateway, and the services that handle auth requests. Also, the wording gives the impression that only a single sidecar pod that saturated (an Istio sidecar pod), which would be surprising, but I’m also not confident that this is what the authors intended.

Diagnostic details: missing in action

The write-up doesn’t talk about the diagnostic work of the incident responders at all, which is a shame. I can’t tell from this write-up how difficult it was for them to figure out what was happening. There were auth failures, but it doesn’t sound like there was an increase in auth traffic per se, nor was the problem caused by recent changes to the auth system, which is where I would think to look first.

As somebody who was watching the updates to the status page as the incident was happening, I was struck by how they updates alternated between “we have identified the problem” and “we are experiencing issues:

I can imagine how frustrating it must have been for the responders to think they had found and fixed the problem, only to continue to see impact.

Retries made things worse

Retries are one of the tools in our toolbox to improve availability. And, usually, retries do improve availability! But retry logic also adds complexity to a system, and adding complexity to a system can introduce new failure modes. In this incident, retries hurt rather than helped, by increasing the load on an overloaded system.

The problem was worsened by optimistic retry logic which overloaded internal load balancers.

…

Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop.

This is a great example of unexpected behavior of a subsystem whose primary purpose was to improve reliability from my conjecture on why reliable systems fail.

The Copilot Token Service sees 10X traffic

Note that there were two independent retry behaviors mentioned in the previous section:

1. optimistic retry logic against the load balancers
2. client retry logic against the Copilot Token Service

It turns out that the client retry logic was due to a previously undiscovered bug in Visual Studio Code(!), which led to one particular service (Copilot Token Service) taking longer to recover:

Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.

…

Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop. Copilot Token Service traffic increased from a normal 7–9K RPS to 70–100K RPS.

There’s no way you’re pushing out a VS Code bugfix to mitigate an incident! You’ve got to mitigate that on the server side, which is what the responders did, which brings us to the next section.

Mitigating the incident: multiple strategies

While the write-up doesn’t discuss diagnostic work, it does mention multiple mitigations that the responders undertook during the incident:

• shifting traffic from the Central US region to the Northern Virginia region
• paused HAProxy on the four saturated nodes
• changed gateway retry logic (via PR)
• blocked inbound Copilot Token Service token requests at the load balancer (returned 403s)
• gradually ramping up blocked traffic

As responders, we are always limited in our ability to intervene based on the tools that we have at our immediate disposal. It’s incredibly useful to be able to do things like selectively block traffic, or dynamically change or even disable a reliability-related subsystem. Think about how difficult it would be to block specific types of requests during an incident in your organization, and to ramp that traffic back up slowly after the system recovers. Note how the responders had to use a pull request to change the behavior of the gateway retry logic. I wonder if the failure mode made this more difficult to carry out, but the write-up doesn’t say.

“Never again” means never preparing for a novel incident

The writeup ends, as most writeups do, with some action items intended to prevent recurrence.

To prevent recurrence, our follow-up actions include:

• Correcting autoscaling policies to account for service-mesh sidecar concurrency and capacity.
• Auditing Istio request, concurrency, and scaling limits across affected services.
• Reviewing retry limits and backoff behavior across gateways and clients.
• Addressing the VS Code retry behavior that amplified Copilot token traffic.
• Improving load-balancer capacity monitoring and regional failover safeguards.

My eternal lament is that people spend too much of their focus on preventing the last incident from recurring. It’s not that I’m opposed to preventative work. It’s that I also want us to spend time on getting better at dealing with novel incidents. Engineering cycles are a finite resource, and every cycle spent on prevention is a cycle not spent on improving our ability to respond effectively to new incidents. And I promise you, you are going to face novel incidents in the future.

After all, I don’t think GitHub customers who experienced this outage take much solace in knowing that it was a different failure mode from the previous incident.

The folks at Microsoft Azure recently wrote up a post incident review for a networking issue in their West U.S region. From the included timeline, it looks like the impact was on the order of five hours. It’s a pretty short write-up, but let’s take a look at the contributors.

On 23 July 2026, a break-fix repair was initiated on an optical device to address a network reliability risk.

The first contributor mentioned in the write-up was work that was done to repair a device in their networking stack. Here I can’t help but think of the first bullet in my conjecture on why reliable systems fail. They made a change to the system in order to fix an ongoing problem, and due to a set of circumstances, things got worse rather than better.

A defect in our blast radius analysis system incorrectly expanded the scope of the repair event to include all optical devices egressing a specific datacenter. 

The second contributor mentioned was a (presumably) latent defect in their system. Note the irony of the failure mode here: I suspect this blast radius analysis system usually contributes to reliability, but in this case it hurt reliability by increasing the blast radius.

The safety validation step, which is designed to confirm that at least one of the two redundant datacenter paths remains available, ran but incorrectly concluded the operation was safe.

The third contributor mentioned was a safety check (good!) that passed even though the action was unsafe (bad!).

The checks validated each device individually rather than evaluating the aggregate effect of isolating all devices at once, a scenario that was not accounted for because the system was never designed to process a full datacenter’s worth of devices in a single request.

The reason it failed was due to an interaction with the second contributor: the blast radius being all of the optical devices egressing the datacenter. The designers never envisioned that the check would have to handle the sort of scenario that occurred as a result of the blast radius analysis system defect.

As a result, routes were withdrawn from multiple devices simultaneously, disrupting connectivity between the datacenter and the WAN – therefore impacting traffic entering or leaving the West US region.

It sounds like this change effectively disconnected the West US datacenter from the internet.

Once the route withdrawals took effect at 14:44 UTC, physical links and routing adjacencies continued to appear healthy, which initially masked the correlation between the break-fix activity and the connectivity disruption

Here we have our fourth contributor: the operators were receiving misleading signals from the system. The links and routes looked healthy, even though connectivity was broken.

The impact presented as a WAN routing anomaly, as third-party networks could not reach Azure in the region, rather than as a datacenter connectivity failure.

Our fifth contributor is another flavor of misleading signals. The symptoms presented as a routing issue between Azure and third-parties.

Although all physical work in the region was stopped, our engineers could not correlate to this recent change because the preparation activities in advance of the break-fix did not succeed, so the physical layer and traffic appeared healthy.

This is the sixth contributor mentioned in the writeup. The writing is a little oblique here, but I think what they are saying is that the repair event did not show up in their event log because the repair event didn’t actually complete. It sounds like the preparation activities were the ones that triggered the incident. But, because the repair event didn’t actually happen, the operators looking for events that correlate in time with the onset of the incident didn’t see the triggering event because it didn’t show up in the log of events. That’s my best guess, anyways.

Our automated recovery and rollback system detected the device failures, and attempted multiple retries to restore the affected devices. However, because that system depended on the same datacenter connectivity that had been disrupted, its automated rollback attempts were unsuccessful.

This is the seventh and final contributor mentioned. Azure has an automated recovery and rollback system (good!), but the failure mode in this case prevented automated rollback from succeeding (bad!).

As always, I’d love to know more about how the operators identified what the failure mode actually was, and how they traced it back to the optical device repair work.