This was a big week for IT incidents making it into the news. I wanted to take a moment and capture links on what’s been reported on each of these so far.
Microsoft 365 (Thu Nov 12)
FAA ground stop (Fri Nov 13)
Workday (Fri Nov 13)
Anthropic (Fri Nov 13)
United (Mon Nov 16)
JPMorgan Chase (Mon Nov 16)
National Security Agency (Tue Nov 17)
Apple services (Tue Nov 17)
Marriott (Tue Nov 17)
Note that none of these incidents involved cybersecurity attacks, according to the reporting. While there is reporting that Wall Street executives and the National Security Agency are telling congress that the outages are related to fixing vulnerabilities, the NSA incident is the only one where there is explicit reporting that it involved vulnerability patching work.
Microsoft 365 (Thu Nov 12)
Charles Lamanna of Microsoft posted an abridged version of Microsoft’s preliminary post-incident report on Twitter. There’s a ton of detail there, and I’m not even going to try to summarize. I’ll write a separate blog post about this one after the final report lands.
More details on the root cause can be found in the Microsoft 365 Admin center, and an abridged version can be found below.
We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations.
The FAA Telecommunications Infrastructure (FTI) network became saturated (1,000 connection attempts every 4-5 seconds). The current hypothesis is that this was triggered by the deployment of a new router at a site in El Dorado, Arkansas.
Services in Workday’s disaster recovery site unexpectedly wrote to production databases, leading to data consistency problems.
Unfortunately, I couldn’t find a publicly available report from Workday. Here are some reports from other parties that were reporting on what Workday told them:
I couldn’t find any details about the nature of the failure mode for the incident(s) that Anthropic experienced on Friday. Here are the official reports:
The United outage is described in the press as a systemwide operational outage that affected pilot applications, including crew communications, messaging systems, and the ability to provide final aircraft weight information for departing flights. Beyond that, I couldn’t find any details about the nature of the failure mode.
While there is nothing official from United, there are United statements reported in the media, as well as statements from the union that represents United pilots:
The JPMorgan Chase incident delayed Zelle transactions. The tweet below mentions a software bug in on-premise hardware systems. Beyond that, I couldn’t find any details.
JPMorganChase has experienced a temporary issue related to a software bug in on-premise hardware systems – there is no evidence this is cyber-related. It is taking longer than usual to service some customers.
Where we were party to a transaction involving other banks, those…
Performance on the NSA’s classified internal communication network (NSA Net) degraded during this incident. According to the reporting, it was triggered by a software patch intended to fix PHP vulnerabilities that were discovered last month.
Several Apple services experienced an outage, including iCloud and Apple Pay. I couldn’t find anything directly from Apple. According to a statement from Apple reported by the press, it was triggered by an internal update.
Marriott experienced sign-in issues on their website (marriott.com) and their app (Marriott Bonvoy). Beyond that, I couldn’t find any additional details. Nothing official from Marriott, here’s reporting that includes a statement from them.
I saw quite a bit of chatter on Twitter about the topic of exporting private keys from hardware security modules (HSMs). I’m not going to comment on that because it’s outside my domain of expertise. What I did want to comment on were some of the details mentioned in the write-up that get at the constraints that the engineers at eMudhra faced, and how those constraints influenced decisions that the engineers made in the moment.
Technical background
Before we can get into it, I need to say a little about public key certificates. This was largely for my own benefit, because trying to explain the technology forces me to learn more about it.
Certificates and browsers
When your web browser makes a secure connection to a website, it does a check to verify that you are connected to the real website, and not some fake website that was set up by a nefarious actor who is trying to fool you. As part of the protocol, the website will return a server certificate that attests “this is the right website”. Your web browser will check that the certificate was properly signed by a trusted authority.
Let’s use everybody’s favorite example website: example.com. Your browser can show you information about the server certificate by clicking on the little icon next in the browser bar > Connection is secure > Certificate details. Here’s what you see in Google Chrome for example.com.
Certificate authorities
For the purposes of this post, the important bit is the “Issued By” section: That tells us which entity issued the certificate. An issuer of certificates is called a Certificate Authority (CA). For example.com, the CA’s common name is Cloudflare TLS Issuing ECC CA 3.
The browser needs the public key of the CA in order to check that the certificate’s signature is valid. It also needs to know whether it can trust the CA that issued the certificate. After all, anybody can set up a CA and issue whatever server certificates they want. The browser has to ensure that the certificate was issued by a trusted CA.
Browsers use the same mechanism for checking whether to trust a CA as they do for checking whether to trust a server: they check a certificate! Instead of a server certificate it’s a CA certificate, but it’s also issued by a CA. In Chrome’s certificate viewer, under “details”, you can see the chain of CAs involved:
The UI will only show three lines, but there’s actually one more CA if you scroll up: SSL.com TLS ECC Root CA 2022
Browsers need a way to trust some CAs without verifying that their CA certificates were signed by another CA, or else you have an infinite regress problem. And so there are a set of CAs that browsers automatically trust. You can view the Chrome Root Store, the set of CAs that are trusted in Chrome, by pointing your browser to chrome://certificate-manager/crscerts. It looks like this:
The root CAs my browser currently trusts
The website normally sends over the server certificate and intermediate certificates needed to build the chain up to one that is known to already be trusted by browsers. The browser then checks the chain to verify that the server certificate was signed by a trusted CA.
For example.com, here’s a diagram that depicts the various CAs that are involved in the chain.
Intermediate CAs
CAs that are already trusted by browsers are referred to as root CAs, and CAs below the root are called intermediate CAs or subordinate CAs. I’m calling out this terminology here because the CA involved in this incident was an intermediate CA. The important thing to remember is that an intermediate CA must have a CA certificate that’s signed by another CA. (Because root CAs are already trusted by browsers, their CA certificate doesn’t need to be signed by another trusted CA, and so they can be self-signed).
The ingredients of a CA
In order to operate an intermediate certificate authority, you need two things:
a public-private keypair
CA certificate
The private key is used to sign certificates. The public key is used by the browser to verify that the signature is valid. The CA certificate demonstrates that another CA trusts you.
A bit of detail that will be important later is that, in order to get that signed CA certificate, you need to already have generated your public-private keypair. If you haven’t generated a keypair yet, you can’t get that certificate.
Hardware security modules (HSMs)
If somebody gets a hold of your CA’s private key, then they can use it to forge certificates. The industry uses hardware security modules (HSMs), which are dedicated hardware devices you connect to a server to securely manage sensitive private keys.
If your server has an HSM installed, it can generate the keypair and keep the private key within the HSM, which means your server never needs to load the private key into memory in order to perform cryptographic functions like signing certificates. HSMs play an important role in this incident because the lack of HSMs ended up becoming a constraint, as we’ll see.
The constraints
With those background details out of the way, let’s dive into the constraints that eMudhra engineers faced as mentioned in the writeup.
CAs must be physically based in Europe and the U.S.
While eMudhra is based in India, their customers had data sovereignty requirements that there had to be CAs physically located in datacenters in the U.S. and Europe. This meant that they needed HSMs at these U.S. and European datacenters, since the private key for the CAs had to be managed by HSM.
The naming convention they used for the CAs at the three sites are:
Series A – based in India
Series B – based in Netherlands
Series C – based in United States
According to the writeup, eMudhra ordered the HSMs for the U.S. site in April of 2025. (It doesn’t say when they ordered the HSMs for Europe). This timing will become important later on.
Supporting older clients
(This section is speculative because the rationale isn’t explicitly stated in the incident report).
eMudhra was setting up an intermediate CA. Recall from earlier that an intermediate CA needs to have a CA certificate signed by a different CA. The eMudhra intermediate CA’s certificate is signed by eMudhra’s own root CA, for which they use the brand name emSign.
Here’s a certificate associated with the American (C series) data center: EM DV TLS CA ‑ G3C. You can see the issuer was emSign Root TLS CA – G3.
If you are a CA and the client doesn’t trust your CA certificate, then you have a problem: the client won’t trust the certificates that you issue. However, you can get your CA certificate cross-signed by a CA that the client does trust, and then you’re good.
There are some older clients out there, ones that don’t have eMudhra’s in their trust stores, and eMudhra dealt with that by having their CA certificate cross-signed by an older CA. In this case, that CA was managed by a separate company: Sectigo (As an aside, here’s Sectigo’s statement on this incident).
Sectigo’s CAs go by the name “AAA Certificate Services”, and you can see the cross-signed certificate here. Below is a diagram of the various CA certificates involved here.
(Note that cross-signing technically creates separate certificates, which I’m not explicitly depicting)
To sum up, the need to support older clients (which didn’t trust their root CA) created a constraint that eMudhra had to get their CA certificate cross-signed by another CA; in this case, that CA was operated by Sectigo.
Client certificates and Chrome Root Store Policy Change Deadline
On the web, browsers use server certificates to verify the identity of websites, but it doesn’t work the other way: websites generally don’t use client certificates to verify user identity, even though browsers support it. Instead, they use things like username and password.
However, there are domains where public-key certificates are used by both servers and clients for verifying each other’s identities, and the use of both server and client certificates is called mutual TLS. One place I have personally seen this used was internally at Netflix. And some enterprises apparently do use client certificates in browsers. eMudhra apparently wanted their new intermediate CA to be able to issue both server certificates and client certificates. The incident writeup doesn’t say why eMudhra wanted this capability.
But there was an issue with this new intermediate CA being able to issue client certificates as well as server certificates. The intermediate CA would be chained to their root CA (emSign ECC Root CA – G3 in the diagram above), which was in the Chrome Root Store. Google had long required that CAs below a root CA in the Trust Store had to be included in the Common CA database (CCADB). But they were about to add a new constraint: intermediate CAs with a root CA in the Chrome Root Store would only be permitted to issue server certificates, not client certificates.
Back in February 2023, on a Chrome Root Program Policy page called Moving Forward, Together, in a section called Focusing on simplicity, they discussed how they value CAs that focus exclusively on web use cases (emphasis mine)
Though multipurpose root CAs have offered flexibility in addressing subscriber use cases over the last few decades, they are now faced with satisfying the demands of multiple sets of increasingly rigorous expectations and requirements that do not always align. We believe in the value of purpose-built, dedicated infrastructures and the increased opportunity for innovation that comes along with them. We also think there’s value in reducing the complexity of the Web PKI to increase the security and stability of the ecosystem. By promoting dedicated-use hierarchies, we can better align the related policies and processes that CAs must conform to with the expectations of subscribers, relying parties, and root program operators.
They noted that they may add a restriction that CAs in the program only support server authentication use cases [emphasis in the original]
In a future policy update or CA/Browser Forum Ballot Proposal, we may introduce:
a requirement that would restrict extendedKeyUsage values in new subordinate CA and TLS server authentication subscriber certificates to only “id-kp-serverAuth.” Modifying the definition of a dedicated TLS PKI hierarchy to prohibit use of other extendedKeyUsage values (i.e., id-kp-clientAuth) in new CA certificates capable of issuing TLS server certificates more closely aligns CA practices with web browser use cases (i.e., securing connections to websites). Further, it eliminates the risk and unintended consequences of other use cases (e.g., IoT) from negatively affecting website authentication.
Google eventually committed to this policy, announcing that they would be enforcing this as of June 15, 2025. Here’s the relevant bit from Section 3.2.1 of the Chrome Root Program Policy doc [emphasis in the original]:
The Chrome Root Program will only accept [Common CA Database] Root Inclusion Requests from Applicant PKI hierarchies that are dedicated to TLS server authentication certificate issuance.
To qualify as a dedicated TLS server authentication PKI hierarchy under this policy:
All corresponding unexpired and unrevoked subordinate CA certificates operated beneath an applicant root CA MUST:
when disclosed to the CCADB…
prior to June 15, 2025, include the extendedKeyUsage extension and (1) only assert an extendedKeyUsage purpose of id-kp-serverAuth OR (2) only assert extendedKeyUsage purposes of id-kp-serverAuth and id-kp-clientAuth.
on or after June 15, 2025, include the extendedKeyUsage extension and only assert an extendedKeyUsage purpose of id-kp-serverAuth.
…
The writeup mentions this date, so I assume this is what they were referring to. The thing is, though, their root CA was already in the trust store, and for intermediate CAs with roots already in the key store, according to section 3.2.2 of that document, the deadline was a year later: June 15, 2026.
At any rate, I assume eMudhra believed that if they wanted their intermediate CA to be able to issue client certificates, they had to disclose it to the CCADB before June 15, 2025. That meant they needed it cross-signed by Sectigo before then. Which meant that they needed to generate the keypairs, which required the HSMs.
Supply chain delays
The HSMs had been ordered in April of 2025. But there was a problem: they were taking longer to arrive than expected. The incident report states that “delivery of the HSMs ordered for the US facility was delayed by global supply‑chain constraints.” and doesn’t go into more detail than that, nor does it mention anything about the European facility.
What is the clear, though, is that this added production pressure. If the HSMs didn’t make it to the site quickly enough for them to generate the keys and get the certificates cross-signed by Certigo then eMudhra could miss Google’s June deadline for disclosing to the CCADB. Alas, the business implications of missing the deadline aren’t spelled out in the report, but it was clearly enough to act as a significant constraint.
Requiring WebTrust certified environments
The report says “It was also eMudhra’s intention to only operate HSMs within WebTrust certified environments…” Unfortunately, they don’t go into detail about why that’s their intention.
The Server Cert working group of the CA/Browser Forum requires compliance audits for CAs (see section 8), using on approved auditing scheme, which explicitly mentions WebTrust as one of these approved schemes. But it doesn’t explicitly say that an audit is required before operating an HSM at the site. However, what’s clear is that eMudhra had an internal policy about this.
Presumably, the HSMs had to be in place first before the WebTrust certification audit could happen, and that would introduce additional delays. Even if the HSM made it before the June deadline, they couldn’t use the HSM to generate the keypair because that would violate their policy.
Moving keys across HSMs from different vendors
The HSMs that the eMudhra engineers used to generate the key pairs were made by a vendor named Utimaco, but the new HSMs they had purchased for U.S. and European sites were made by a different vendor: Crypto4A. According to the write-up, the fact that the HSMs were manufactured by different vendors constrained their ability to transfer keys from one to the other. I assume based on this that vendors provide better support for transferring keys between HSMs of the same vendor. But in this case, the eMudhra engineers had to write a custom Python script to support the transfer, and the resulting script they wrote had a bug in it that led to the unintentional truncation of the private key to a single byte.
The outcome: a workaround leads to an incident
Based on the constraints they were facing, the eMudhra engineers decided to generate the keypairs for the European and U.S. sites at the (WebTrust certified) India site, and then export the keys out of the Indian HSM into the U.S. and European ones when they were able to. This was a workaround and … things went wrong during a rehearsal of the transfer process, and as a result they ended up with a cryptographically weak private key. You can read about the specifics of the failure mode elsewhere, it’s not my focus here.
Instead, my focus is how constraints shape behavior. Some of the constraints were business-oriented constraints (wanting to support mutual TLS, support for older clients), some were technical (transferring keys across HSMs manufactured by different vendors) and some were external (supply chain induced delays), but others were explicitly designed to improve security:
Data sovereignty requirements
Chrome Root Store Policy change to increase simplicity
Hardware security module requirements
WebTrust certification requirements
These are all well-intentioned, reasonable requirements, that nevertheless conspired against the eMudhra engineers in this scenario. The goals of each of these is to increase security, and yet, in this particular scenario, they each contributed to a significant security incident.
It’s also a textbook example of the role of production pressure. Look at how these factors interacted to create a hard deadline, and how that deadline influenced the work. It’s tempting to fault the engineers for doing the wrong thing here. But if you really want to understand how incidents happen, you have to look at how constraints influence decisions and workarounds.
Google recently published an incident report for a GCP zonal outage that happened back on October 19. This involved a power outage at a datacenter, which sounds like it’s less directly relevant for those of us who use clouds rather than run them. But the details of the incident reveal how it’s a great example of the nature of complex systems failure.
There’s a surprising amount of detail packed into the relatively short write-up. I’ll recapitulate the bits I find most interesting here. As always, though, I recommend you read the original first.
Power distribution in europe-west-2b
Here’s my attempt at a high-level representation of how power is delivered to the GCP eureope-west-2b datacenter involved in the incident.
This is very hand-wavy because:
I don’t have any domain expertise in datacenter power distribution
The only info I have on the datacenter power topology is what’s in the original write-up
I’m not showing any detail about how the switchboard feeds multiple circuits (for the two reasons stated above, and because it isn’t germane to the failure here)
The part that I have least confidence in is the block that I’ve labeled “UPS subsystem (battery backup)”. I’m using it to
indicates that there’s a battery-backup UPS that feeds the data center load if power is interrupted, and
there are multiple power sources that can potentially feed the load, which somehow interconnect
My goal with this diagram is to give the reader a visual sense of the following facts:
The datacenter had two independent power feeds that come from the primary utility provider, which are labelled “feed A” and “feed B”. They refer to this as a 2N architecture.
Each of these two feeds terminates into a separate power subsystem called a switchboard
If one of the feeds fails, the switchboards can be bridged (indicated by the open switch between them), which enables the datacenter to be powered by a single feed
There are two (redundant) programmable logic controllers (PLCs) which can automatically activate emergency diesel generators. The PLCs will activate the generators if they sense that both feeds are no longer delivering power. Each PLC reads from two voltage sensors, one on each feed.
There is an uninterruptible power supply (UPS) system – a set of battery banks that provide a few minutes of power to give the emergency generators time to come online in the event that the two feeds both fail.
The datacenter is also served by a standby utility provider. Manual intervention is required to switch to using the standby power
If you aren’t familiar with PLCs, they are basically simple embedded processors that are used in industrial control systems.
Now, let’s get into what happened here.
A load test reveals an issue with feed B, a breaker is opened to fix it
Six days before impact
On October 14, 2026, Google engineers run their quarterly generator load test. The test reveals an issue that requires corrective maintenance (the report doesn’t say what this issue is).
The engineers open the circuit breaker on feed B in order to perform the maintenance, and configure the switchboards in the mode where a single feed can power the datacenter:
Everything is still working fine at this point. However, one level of redundancy has been temporarily eliminated so the maintenance can happen.
An unrelated cable fault takes out feed A
Seven minutes before impact
On October 19, 2206, at 6:53 BST, something goes wrong with an underground cable on feed A, which interrupts the flow of power on that feed.
Sensors show good voltage, the generators stay off
With the fault on feed A and the breaker open on feed B, the datacenter is not receiving power from either of the two feeds. With the power interruption, the UPS system kicks in, delivering several minutes of power in anticipation of the emergency generators coming online.
The expected behavior in this scenario is for the PLCs to detect that power has been interrupted on both feeds and to start those generators. The PLCs are configured to trigger the generators if they detect that the voltages on both feeds are too low. However, while voltage sensor A detects a drop in voltage, voltage sensor B does not.
The issue is not with the sensor: it correctly indicates that the line on feed B is still energized. Rather, the issue is in the configuration of the PLC itself. It is configured to trigger the generators on undervoltage of the two lines. But, as this case demonstrates, expected voltage on the feed does not guarantee that power is being delivered. (After the incident, Google engineers will change the configuration so that the PLC also takes into account the state of the breaker).
Even though there are two PLCs, they are configured to use identical logic, and hence both of them conclude that feed B is still delivering power.
As a result, the diesel generators never activate.
The UPS is exhausted
Impact starts
At 07:00 BST, the batteries in the UPS start to become depleted, and parts of the data center lose power. As a result, the services that were previously running in this zone no longer respond to requests. There is now an incident.
The response begins: the stubbornness of a PLC
Four minutes after impact starts
The failures trigger alerts, and the Google engineers spring into action.
Because the PLC supports receiving commands via remote connections, engineers at the datacenter connect to the PLC and attempt to transfer the load to the diesel generator. But because the PLC sees expected voltage levels on sensor B, it does not allow this transfer to occur: the engineers can’t use it to switch power to the emergency generators.
Workloads are automatically rerouted to other zones
Nine minutes after impact
With services in the europe-west2-b zone no longer being reachable, GCP’s systems automatically fail over the impacted traffic and workloads to other zones in the europe-west2 region. As a result, these other zones get slammed with new load, and they show elevated 500 and 503 API error rates as the compute resources in these zones become saturated.
The responders also take action to mitigate the impact by making adjustments to regional service routing and preventing the other zones in europe-west2 from making calls into eureope-west2-b.
The standby utility provider comes online
Fifty-four minutes after impact starts
With the failure to bring the emergency generators online, the engineers turn to the standby utility provider as a source of power. This strategy succeeds, and at 7:54 BST, power is restored.
Multiple contributing factors
One lesson I always try to drive home when discussing incidents is the fact that complex systems failures require multiple conditions to be present at the same time. Here are the conditions mentioned in the writeup:
The underground cable fault that happened on feed A
Configuration of the PLC that led to it not triggering when the breaker was open
The problem on feed B that warranted corrective maintenance
Running the quarterly load test that discovered the problem (!)
The breaker opened on feed B in order to perform the maintenance
The PLC not supporting shifting load if it detected one of the feeds being active
In particular, note how the datacenter had four separate sources of power:
Feed A
Feed B
Emergency diesel generators
Standby utility provider
Here they experienced three(!) co-incident failures:
The underground cable fault
The problem detected by the quarterly load test
The misconfigured PLC
Fighting against the automation
In addition to the failures above, there was the issue of not being able to manually trigger a failover to the emergency generators. I cannot imagine the frustration that Google engineers must have felt at the PLC not permitting them to redirect the load to the emergency generators(!).
Yet again, saturation rears its ugly head again
Here we see two forms of saturation at play during the incdient:
The datacenter running out of power (an essential resource)
The adjacent zones getting saturated as traffic was rerouted to them
Once a system reaches a certain level of reliability, most major incidents will involve:
A manual intervention that was intended to mitigate a minor incident, or
Unexpected behavior of a subsystem whose primary purpose was to improve reliability
This is a clear example of the first scenario, where the manual intervention was opening the breaker on feed B in order to perform maintenance. (Although, technically, it doesn’t sound like the manual intervention was intended to mitigate a minor incident, but I’m still going to count it). In this case, that eliminated a source of redundancy that left the datacenter vulnerable to a confluence of other events.
I think you can make an argument for the second one as well, when the PLC could not be used to shift traffic to the emergency generators, because the sensor indicated expected voltage on feed B. Fortunately, the engineers had another source of power available to them.
I suspect that, if they had not had access to the standby utility, they would have figured out a way to shift the load to the emergency diesel generators, but it would have taken them much longer.
As you’ve no doubt heard, the India-based public-key infrastructure company eMudhra recently announced that eight of their global TLS intermedia certificate authorities were cryptographically weak: it was trivial to reconstruct the private key from the public key, which meant that attackers could easily forge digitally signed certificates. If you’re a terminally online person like I am, you’ve almost certainly seen the numerous jokes referencing the number four, which was a reference to one of these weak private keys:
I’m anxiously awaiting the report with the full story about what happened. eMudhra has contracted with KPMG to investigate the incident, and hopefully they’ll share some public form of the report once the investigation is complete. In the meantime, Lex Neva pointed me at this bugzilla ticket with some preliminary details about the incident. Here’s part of a comment from Naveen Kumar ML, which I am assuming from context is an eMudhra employee. I’ve made some small formatting adjustments below.
We have done some preliminary assessment, there appears to be a flaw in the KGC [Key Generation Ceremony] script that was used on that specific day to create these ECC [Elliptic Curve Cryptography] issuers.
As for these ECC keys, our review of the KGC, assessment of the code and view of HSM [Hardware Security Module] keys so far indicate that:
a) a set of original ECC keys were created onboard as expected. b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr [certificate signing request] (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo).
…
Some additional context – these issuers created were meant for export to our US and European data centers to solve data sovereignty issues for some of our customers(but never ended up operationally in production except for one).
We are still assessing the code to see except [sic] steps where this process went wrong.
At Key Ceremony 1, the PKI Operations script generated each B and C key, then wrapped, unwrapped and re-imported it as a planned rehearsal. From that day there were two objects with the same label for each CA, and the participants created them. What we did not understand then was that the key objects contained different data, and the CA certificates had been issued over the re-imported objects and not the originally generated ones.
This stuff is outside my personal area of expertise, but I’m going to do my best to describe in my words what happened at eMudhra back on April 30, 2025.
That day, eMudhra was creating a new certificate issuer. One of the steps involved in creating a new issuer is generating a public-private key pair, and the process for doing that is called a key generation ceremony. Nothing went wrong with the generation of this initial key pair: the private key was secure.
It’s not explicitly noted above, but my assumption is that the key pair was generated on Indian servers. Some of eMudhra’s customers have data sovereignty requirements: certain data has to be housed in a particular geographic region. eMudhra specifically calls out the need to house issuers in US and European data centers, so presumably these are American and European customers. As a consequence, eMudhra has to securely transmit the private key to servers based in U.S. and Europe.
One thing I learned from reading this is the process of encrypting a private key so it can be transmitted is called wrapping, and decrypting it is called unwrapping. During their key generation ceremony, they tested wrapping and unwrapping. And, somehow, (this appears to not yet be fully understand), a new (insecure) key was created rather than decrypting the original (secure) key.
What really jumped out at me here was that the insecure keys were introduced by a process change that was necessary due to data sovereignty customer requirements. Data sovereignty is itself a security concern; these customers imposed the requirement in order to increase security. And, yet, that requirement ultimately contributed to the creation of a security vulnerability. Even more ironically, this was described as a test of the wrap/unwrap process. Testing contributed to the introduction of a security vulnerability!
This is not to say “data sovereignty is bad” or “testing is bad”, even though both activities contributed here. Rather, it is yet another reminder that all of our actions involve risk and trade-offs, even if we are not explicitly aware of these. In particular, adding additional requirements, even security or reliability requirements, invariably means adding additional complexity to the system, which introduces new risks, whether we are aware of them in the moment or not. Never forget: all practitioner actions are gambles.
The research field known as cognitive systems engineering (CSE) emerged in the early 1980s as a subfield of safety science. That time period saw a dramatic rise in the use of computer-based automation for controlling safety-critical systems. One of the big ideas of CSE, I would even call it the big idea of the field, is that introducing new automation changes the nature of the work that humans do. It’s not that automation means that humans now have less to do: instead, it means that humans face a different kind of work, some of it being genuinely novel.
It’s in the spirit of CSE that I read with interest the recent Anthropic white paper from Nathan Wilmers, Regina Catipon, Maxim Massenkoff, and Peter McCrory titled How employment is changing at Anthropic. It’s a study of how work is changing inside of Anthropic with the embrace of LLM agents. Here are some thoughts I had reading this.
Attention is all used up
Keeping the agents busy is a challenge: these agents spend more time waiting for human input than they do getting work done. Notably, there’s no discussion in the paper about whether agent utilization is genuinely important to the business or not. But what’s clear from the paper is that humans are the bottleneck. More specifically, they point out that human attention is the bottleneck.
Figure 6 in the paper has a fun graph about how much time it would take a human to simply read all of the output generated by their agents, given the average number of tokens outputted per session, assuming 0.75 words per token, and a reading speed of 250 words per minute. For marketing and comms, it would take 12 hours for a person to read the output generated by a week of agent usage, which is 30% of a 40-hour work week (assuming 40 hour work weeks is naive, I know, but let’s go with it). For legal, policy & trust, it’s 26 hours, which is 65% of a 40-hour work week. For finance and accounting, it’s 43 hours of reading, or 108% of a work week. And for engineering and research, it would take 424 hours(!). I’m not going to compute the work week percentage for this one, but the discriminating reader will note that there are only 168 hours in an entire week. What this means is that there is literally not enough time in a week for engineers to read all of the Claude output, even if they work 24×7.
They’re like, I need my juice box opened and it’s an emergency that my juice box is opened. And that is voiced with the same level of concern as the one who has sliced their finger and is bleeding.
The quote above is taken from an engineer pointing out that not all agent requests for human attention are created equal, but that the agents themselves do a poor job at coordinating with the humans to effectively communicate the level of urgency involved in a request. Here I want to reference a famous CSE paper, now over twenty years old, called Ten challenges for making automation a “team player” in joint human-agent activity, written by Gary Klein, David Woods, Jeffrey Bradshaw, Robert Hoffman, and Paul Feltovich. Two of the challenges mentioned in the paper are:
An agent must have a model of the behavior and beliefs of the other people involved in the joint activity
An agent must be able to observe the behavior of the humans and use that information to identify their current states and intentions
These are about how agents need to have a model of the humans they are interacting with, in order to effectively coordinate with them In CSE terms, these agents are not good team players. Because the agents don’t have a good model of the engineer’s current goals and state, they aren’t able to make decisions about when it’s appropriate to vie for the engineer’s attention.
From engineers to managers (but still engineers)
One solution to this deluge is rising automation. But this requires work of a different kind.
The paper describes how the work at Anthropic is shifting to meta-work, more commonly referred to as management. More and more of their engineers’ time is spent on coordinating work in dealing with the agents. The paper has some fun examples of this, including attempts to automate it: an agent that triages requests from other agents, doing performance reviews of agents, writing specs for multi-agent runs, looping one agent to review the output of another agent.
The paper even tries to estimate the amount of time spent on meta-work, as shown in figure 7, which has the fantastic title “Becoming managers of Claudes”. The group with the highest meta-work overhead is infrastructure, at 29%.
Judgment and autonomy: the decline of the junior hire
“the thing that raised eyebrows was the uplift for performance engineers in particular and it basically overnight changed what we were looking for.”
One of the main themes in the paper is a hiring trend: they are hiring fewer junior engineers. However, this drop in junior hiring isn’t related to the fact that seniors have deeper domain expertise than juniors. In fact, they specifically call out that domain expertise is becoming less of an issue for hiring managers. Instead, they note that hiring managers are looking for candidates who exhibit good judgment and are capable of working autonomously. Junior engineers are typically responsible for executing on well-defined tasks, and it sounds like they see have internalized the idea that LLM agents can replace juniors at doing this sort of work. But they aren’t yet able to use LLM agents to replace the judgment of senior engineers, and they don’t yet trust their LLM agents to work at the level of autonomy that a senior engineer works at.
This is both unsurprising and depressing. Senior software engineers don’t spring fully formed from Zeus’s forehead; if the industry stops hiring junior engineers, eventually there won’t be senior engineers around anymore. I’d had a sense that it was getting harder for junior devs to find positions, and this is a data point in support of that theory.
I also wonder if Anthropic will tailor their interviews to focus on assessing the quality of judgment of their senior candidates, as well as their ability to work autonomously. I bet if you asked them today, they’d say “we already do assess that”. But I also bet they don’t just have an explicit “judgment” interview and an explicit “autonomy” interview, that they try to gather this info from the existing interview tropes, like coding and system design. Would be very interesting if they transitioned entirely to focusing on experience for these senior hiring interview rounds, although there’s no evidence in the paper about this happening.
Grab bag of trends
A break from managing the agents typically looks like jumping into a meeting.
Time spent in face-to-face meetings is trending upwards. The authors hypothesize that engineers spend more time on strategic work, and resolving questions of strategy requires more conversation among the humans.
The percentage of non-engineers is dropping at Anthropic. They note that the ratio of engineers to product managers is increasing over time: currently the ratio is seventeen engineers to one product manager. This surprised me: I would have assumed that product managers would be endlessly busy coming up with new product ideas, and so the demand for product managers would be going up relative to engineers rather than going down. Then again, I don’t really know what product managers do at Anthropic.
There is organizational pressure to have work discussions happen in places where the agents can observe. This means having them in public Slack channels, as opposed to private DMs or in unrecorded meetings. They want to ensure that the work is visible so that it is legible to the agents.
Outro
It was fun to see Anthropic make an attempt to observe itself like this. I hope more companies look at how AI is changing the work that their employees do, and what new challenges their employees face. I also hope people discover CSE research and how relevant it is to the age of LLM-based automation.
On Mastodon, Ole Peder Brandtzæg pointed me to (yet) another GitHub incident that happened back on Sept 23. The write-up is very short, so I’ll just excerpt the whole thing here (emphasis mine):
Starting at 07:57 UTC on September 23, GitHub experienced elevated 500 and 404 responses across several application pages. This caused failures when installing GitHub Apps, creating organizations, and making some organization membership changes. Customers also experienced delayed label updates and stale search results in Projects.
The infrastructure failure was isolated to our Azure Central US region. The API errors were mitigated by 10:58 UTC on September 23. Projects’ processing continued to recover while an accumulated backlog was drained, and full service was restored at 04:55 UTC on September 24.
The incident was caused by a failed planned maintenance operation on a primary database. Automated recovery initiated an emergency database failover, after which several replicas in the affected region were unable to resume replication correctly. This reduced available database capacity and caused the API errors and downstream Projects processing delays.
We have mitigated the immediate failure mode. We are also improving maintenance safety checks, database failover handling, post-failover replica validation, and downstream processing resilience to reduce the likelihood and impact of similar incidents.
There really isn’t much to go on here, but let’s make the best of it.
Unlike previous GitHub incidents, this one doesn’t sound like a “hit a tipping point due to increased load” sort of failure mode. Instead, the trigger here was planned operations work. They don’t discuss the rationale for the planned maintenance, and so it may very well be that the work was being done to increase capacity, in which case it does rhyme with the other “increased load on GitHub” incidents. Or perhaps the planned maintenance was intended to deal with some other issue. There just isn’t enough information here.
The incident was caused by a failed planned maintenance operation on a primary database.
All we really know from this write-up is that something went wrong with the maintenance operation, because it describes the operation as “failed”. From the write-up, it sounds like they were running a database cluster, where they had a primary and replicas.
A primary-replica database cluster: all database nodes can service reads, and the primary services writes
The nice thing about this sort of setup is that, if the primary fails, then a replica can automatically take over as primary, which significantly reduces downtime, which is how this database cluster was configured.
Automated recovery initiated an emergency database failover, after which several replicas in the affected region were unable to resume replication correctly.
But, again, all we get from the write-up is something went wrong with this process. It sounds like a new replica was successfully promoted to primary, but the other replicas were somehow unable to connect to the new primary.
Another advantage of a primary-replica style database cluster is that you can divide up the read request traffic across the entire cluster, which gives you more capacity. Conversely, if you lose the replicas, then the primary has to service all of the requests. This increases the load on the primary.
When the replicas fail to replicate, the primary becomes responsible for servicing all of the read traffic
This reduced available database capacity and caused the API errors and downstream Projects processing delays.
Based on the write-up, it sounds like the primary was not able to handle all of the traffic that resulted from losing the replicas. Note that even though this incident was not triggered by an increase in load, it still ended up being a saturation-related failure mode because of the loss of capacity due to the effective loss of the replica nodes.
Lex Neva (of SRE Weekly fame) pointed me to this recent GCP incident report, which describes networking issues they experienced in their us-central1 region. This incident is yet another reminder of why I don’t work in networking: I think that networking engineers are required to work within systems that are inherently unsafe by their very nature.
Now, I’m on the record for stating that human error is not a useful construct. However, I recognize that I’m in the minority here, and that most people believe that human error is a real and useful thing to talk about, so I’ll go with it here. I believe that for a system to be safe, it needs to be error-tolerant. An error-tolerant system is a system that does not fail hard even when a human makes an error. Achieving error-tolerance means having things like cross-checks, and the ability to recover quickly if an error does lead to a genuine problem. If you think about the way we typically roll out code to production, there are typically multiple checks involved (e.g., code review, unit tests, CI checks, tests done in a staging environment, canarying). In addition, we use techniques like progressive rollouts, blue-green deploys, and feature flags to reduce the harm if bad code makes it out to production.
I’m not a network engineer myself (I’ve never managed more than a single rack of servers, which involves a grand total of two switches), so I can’t speak from direct experience about the nature of the work. But every time I read an incident write-up that involves networking, it screams to me “this is not an error-tolerant system that network engineers have to work within”. Whenever you have to muck around with networking gear, very bad things can happen. From the GCP report, here’s the entire (sigh) “Root Cause” section (emphasis added):
The event was triggered by the inadvertent physical disconnection of network fiber-optic cables during routine hardware maintenance. A technician was performing a scheduled capacity upgrade on data center routers that support a fraction of capacity in the us-central1-b zone and a small fraction of capacity in the us-central1-f zone.
During the hands-on maintenance, optical fibers connecting the data center routers to the network fabric were unplugged and the optical transceivers were replaced with a transceiver that supports higher-density connectivity. This transceiver was not compatible with the transceivers still in place in the data center fabric, causing a loss of connectivity for each fiber.
The network is designed with redundant routers in separate data center rooms within the same building so that failure of one router does not interrupt service. The maintenance was intended to upgrade one router at a time over several days, with traffic diversions at each step and verification between steps that the network has returned to a fully-connected state.
However, a procedural error in the manually orchestrated upgrade process caused the complete list of transceiver replacements across all routers to be issued to the technician without instructions to sequence the work one router at a time. The maintenance workflow also did not include the expected human and software verification steps for detecting unintended disruption.
As a result, the technician sequentially unplugged all fiber paths across the affected devices within 13 minutes. The speed and nature of the error prevented warnings of incorrect action from reaching the engineer before connectivity was lost. Additionally, a standing procedure to halt if light is detected on any fiber-optic cable after unplug was not followed.
This resulted in compute capacity in the impacted zone being isolated from the network. Customers were unable to reach their virtual machines, and those virtual machines could not establish connections outside their zone.
The report attributes the incident to two errors:
correctly following an incorrect procedure
incorrectly following a correct procedure
Contrast the paucity of this analysis with the talk that Sean Klein gave at QCon SF 2025 about an Azure global networking outage that occurred back in January 2023. That also involved a networking engineer carrying out a change where the instructions contain a procedural error. However, Klein goes very deep into the history of how that came to be.
We’re also provided with no context about the nature of the work, so I have no idea how it came to be how the procedural error was introduced, or that the engineer didn’t follow the standing procedure. (Maybe they were distracted by some other aspect of the work that is part of the procedure?) All we have is the hindsight judgment that they didn’t do the right thing.
This line also jumped out at me:
The maintenance workflow also did not include the expected human and software verification steps for detecting unintended disruption.
Our procedures are never going to be perfect. And that’s exactly why I wince every time I see “human didn’t follow the procedures” as a reason for an incident. Documented procedures are a tool, and the human consuming them has to use their judgment about how to apply the tool. If your system assumes humans perfectly following perfect procedures then, well, you’re in for some surprises.
Here are the follow-up action items (emphasis mine):
Reinforce training in us-central1 region and globally around safe maintenance techniques, including the mandatory requirement to verify whether light is on every unplugged fiber.
Finish the migration of network upgrade workflows to a fully automatically sequenced orchestration system. Most common regional network workflows were completed in 2025, and zonal workflows are in progress.
Complete the deployment of work stop alerting for actions resulting in the unintended disconnection of live fiber cables in Google data centers.
Complete the deployment of the automated system that will move regional traffic away from faulty zones, reducing the time from approximately 19 minutes in this instance to around 5 minutes.
Now, I’m confident that Google did a more extensive investigation than what they wrote up. But even so, attributing the outage to two human errors, and listing training as an action item (blame and train!) are big red flags to me that they don’t understand how work actually gets done in their own system.
I also note with interest how the the other actions involve deployment of new automation, which will likely introduce novel failure modes even as they eliminate the sorts of failure modes that occurred in this incident. I hope the Google SREs are ready for them.
In the meantime, I recommend they check out the book Behind Human Error.
When it comes to public incident write-ups, GitHub continues to be the gift that keeps on giving. They keep suffering collateral damage from the AI boom, as more developers using AI means more load on their system. Their developer focus also means that they provide some technical details about these incidents.
Today I’m writing about the incident that happened to them on September 13. The report has the vague title Incident with several GitHub Services, which was the title when the incident was declared. It’s a shame they don’t re-title their incidents later on.
The report is short enough that I’ll simply excerpt the three paragraphs that discuss the failure mode.
The cause was an internal data-cleanup job that began writing to a shared database cluster at 07:33 UTC. That cluster stores permission data read on nearly every authenticated request. The safeguard that was pacing the background job watched only one health signal — how far the database replicas were lagging — and that signal stayed low the whole time. It did not account for the load building on the primary itself, so the job kept writing while the primary quietly ran toward its limit.
When the primary ran out of available connections, requests that needed it could not complete. First, there was no quick timeout on these database calls, so request handlers waited on the stalled database instead of failing fast, and the shared request-handling capacity degraded into site-wide errors. Second, a retry loop around token creation kept re-sending the writes that were already failing, which held the database saturated rather than letting it recover.
Monitoring declared the incident at 08:50 UTC, but due to the broad impact and amplification from token creation, it took time to identify the source of the load. First responders mitigated by shedding internal load and pausing the job, and all services recovered by 10:44 UTC.
Here’s my attempt to depict this text with a diagram, and to rephrase the above in my own words.
Within GitHub, there is a database cluster in the typical SQL database style cluster configuration, with a primary and read replicas, where the primary is responsible for handling all of the writes. This cluster takes online traffic: meaning that activity from external users will generate queries against this cluster.
GitHub ran an offline job (i.e., not driven by user traffic) that made queries against the database to clean up unneeded data. The rate at which this cleanup job queried the database was governed by a safeguard mechanism that monitored the health of the database: the safeguard system would reduce the rate of queries if the database appeared to be going unhealthy. The safeguard monitored the health of the database using the replica lag (if they were using MySQL, presumably they were tracking Seconds_Behind_Source). An increase in this replica lag can be a symptom of high load on the replicas.
In this scenario, the replicas themselves were healthy, and the replica lag stayed below the threshold that the safeguard was monitoring. However, it was the primary node of the database cluster that was at risk of saturation. Specifically, the primary reached its maximum number of client connections (on MySQL, this is controlled by the max_connections server system variable). This means that any attempt to query the database over a new connection would fail with an error (on MySQL, this would be ER_CON_COUNT_ERROR, with an error message of Too many connections.
When some web servers attempted to make calls to this database, the requests did not succeed immediately, but neither did they fail immediately. Instead, these requests were stuck waiting. Unfortunately, the timeouts configured on these request handlers were long. This meant that these blocked requests accumulated in the impacted web servers. Each in-flight request consumes some amount of resources, and enough of these blocked requests accumulated in the web servers that they themselves became saturated, which led to sitewide impact.
To make matters work, there was a token creation process that kept trying to query the primary, failing, and then retrying. This placed a continual load on the primary that made it more difficult for the system to recover.
Aside: some unanswered technical questions
Here some questions I had that I couldn’t figure out from the text.
How did exhausted connections lead to timeouts on the web servers? Was it the case that the web servers were using connection pooling, not all of the connections in the pool were active, and the clients were stuck waiting for a new active connection that never came?
How did that background job lead to the primary exceeding the maximum number of connections? Did this job consume a surprisingly number of connections? How close was the database to the limit before the job started?
—
You can never truly know where the safety boundary is
When you have an offline job running against an online database, there’s always a risk that it can negatively impact the performance of the database, and thereby affect customer traffic. The nice thing about offline jobs is that they are, in principle, fully controllable by the organization. With user traffic, on the other hand, you are at the mercy of your users.
And, so, GitHub had an automated system (the safeguard) monitor the health of the database while the cleanup job was running, to ensure that the job was not applying too much load to the database. If the load was getting high, then the safeguard would slow down the rate at which the cleanup job was querying the database. The problem was that the metric being monitored by the safeguard did not give a complete picture of the health of the cluster. As far, as the safeguard was concerned, the database was still healthy, even though it was running out of connections. This is a great example of a gray failure: when your internal health monitoring systems register, incorrectly, that the system is healthy. In other word, the metrics are good, but the users are unhappy.
Putting things in resilience engineering terms, the system misjudged the location of the safety boundary. There was no signal that the system was getting close to being in an unsafe state until the boundary was crossed. Note that we never actually know where the safety boundary is until we actually cross it. But crossing the safety boundary is very, very bad. And so we always have to make an estimate about where the safety boundary is, and then we avoid getting too close. But there are so many limits in the system, that the chances of us not monitoring all of them is, tragically, quite high.
Offline jobs and online databases: sometimes you gotta run ’em
In an ideal world, we would want all of the query traffic for our online databases to be, well, online traffic. That’s why, for example, we don’t run our analytics queries against our online databases. However, there are scenarios where you have no choice: you have to run a job against an online database, because you want to effect some sort of change to the online data. This is a great example: here the job has to be run on the online database because the goal of the job is to make a change to that data.
The double-edged sword of cleanup
When it comes to the topic of “cleanup”, both for data and code, once you’ve seen enough incidents, you’ll notice a pattern: cleanup is a frequent risk. Sometimes incidents happen because a cleanup job was attempted, and the thing being cleaned up was still in use. And sometimes incidents happen because some entity that wasn’t even used anymore interacted in an unexpected way with a different part of the system. In other words: an absence of cleanup contributed to the incident. So, not cleaning up unused stuff is a risk: it can lead to incidents. But the act of cleanup is itself a change to the system that carries risk. It’s all risk tradeoffs.
The asymmetry of misconfigured timeouts
Timeouts are an essential tool in building reliable distributed systems. But there’s an asymmetry in the timeouts-are-too-short problem versus the timeouts-are-too-long problem. If your timeouts are too short, you’ll notice this during the normal operation of your system, by seeing excessive timeouts happening. And, so, when that happens, you adjust the timeouts to be longer.
However, timeouts that are too long are harder to detect. If you’re lucky, you might notice some sluggishness in your user interface and diagnose the problem as being related to timeouts being too long. But, in many cases, you won’t have any signal at all that timeouts are too long until they contribute to an incident. As we saw here, long timeouts create a saturation risk by increasing the number of in-flight requests in a service, thereby depleting its resources.
You probably have timeouts that are configured to be too long in your system, and nobody has noticed them yet.
Retries help, until they hurt
Retries are yet another essential tool in building reliable distributed systems. Because transient failures are common, retries can effectively mask these failures from your users: they’ll never know about that one bad pod in your deployment. But retries increase load, and in a saturation failure mode, you can get a pathological behavior where retries exacerbate the incident. That’s what happened in this case, where a particular service (token creation) kept retrying over and over again, leading to unending load on the database.
You need retries! But, like misconfigured timeouts, it can be hard to catch misconfigured retries until the incident happens. That’s exactly why our industry has a specific term for failure modes where retries made things worse: retry storms.
When you’re overloaded, you need to reduce the load
Let’s look at the last quoted paragraph again, since it has some details about the incident response (emphasis mine).
Monitoring declared the incident at 08:50 UTC, but due to the broad impact and amplification from token creation, it took time to identify the source of the load. First responders mitigated by shedding internal load and pausing the job, and all services recovered by 10:44 UTC.
Those of us who do incident response know that our first priority is always getting the system back to healthy. If you can quickly identify the source of the problem, that’s great! Go ahead and roll back that problematic deploy. But often we can’t tell what’s causing the problem. In a saturation failure mode, we can often identify which service has gotten overloaded (and, very often, it’s a database), but it isn’t obvious what the source of the problem actually is. In scenarios like this, you want the responders to be able to:
quickly identify the sources of load (which queries, and who is making them?)
manually shed load (block specific queries/sources)
To have that operational tooling at your fingertips during an incident, you need to have built it in advance. During the incident, it’s too late, you’re stuck with whatever tooling is available.
Your systems will eventually become overloaded, this I promise you. Your online databases at particular risk. The time to prepare for this is now.
The solution, as always, is to increase complexity
Here’s the last paragraph of the write-up:
To prevent recurrence, we are rate-limiting background jobs against shared, customer-serving databases by default, and adding automatic pausing and paging on primary-server load rather than replication lag alone. We are also surfacing running background work directly alongside database health signals so responders can see and pause it without leaving those dashboards, bounding retries in the token-issuing path, and adding request-level timeouts so one unhealthy database cannot consume shared web server capacity. In addition, we are breaking apart this database cluster to remove the single point of failure. We will be moving various service-specific data, including the authorization data, out of this shared cluster in the next two weeks.
These all look good interventions to me. But note how these all increase complexity! They all involve adding new stuff to the system, as post-incident interventions always do. I am on record as stating that increasing complexity in order to improve reliability is inevitable. But we should be sanguine about this. By increasing complexity, we eliminate known failure modes at the cost of introducing novel failure modes. And introduce them we do.
A hallway in a tech company office. A and B are standing in the hallway. B is holding a multiple page document.
A: Wow, that incident from a couple of weeks ago was a real doozy. Do you know how it happened?
B brandishes the document.
B: As it happens, I just finished reading through the incident write-up. They did a pretty thorough investigation.
A: You printed it out? I didn’t know we even had printers in the office anymore.
B: Yeah, I find it easier to read on paper than a computer screen. Anyways, there’s a lot of great detail here, about the history behind the changes that contributed to the incident, about how confusing things were for the responders during the incident response, the tools that the responders used to diagnose what happened, the levers they pulled to actually mitigate the customer impact… I learned a ton from it. Would you like my copy?
A looks skeptically at the document.
A: Wow, that’s a long report.
B: Yeah, it’s about thirty pages. But that’s what it took the authors to explain everything. Like I said, lots of great detail.
A: I don’t know about you, but I’m pretty busy these days.
B: To be honest, I am too. The upcoming launch is keeping my whole team really busy. Not to mention all of these migrations that are going on, that we need to spend time on.
A: Tell me about it! I just don’t have the time to spend reading a doc like that. Can you just tell me the high points, so I can get the most out of it without reading the whole thing?
B: Well, the thing is, I’m pretty sure you’d learn different things from the write-up than I would. I’m sure there’s stuff in here that you already know, that I didn’t. And also, vice versa.
A: Hmmmm…
B: And even if I try to recount what I learned… well, it’s not the same as reading it. I’m not good at articulating what I learned. Just giving you a list of bullet points is not going to be as effective at just going through and reading the whole thing. And, from experience reading lists of “learnings” don’t stick in my head the way the same way as the stuff I learn reading a well-written incident write-up like this one.
A: I see what you’re saying, I really do. I’d like to be able to sit down and spend the time really reading through this whole document. But, given all of the current obligations that business has placed on me, I don’t have that time to invest. I want to learn from this incident, and I know I would learn more if I spent the time on reading the whole thing, but there’s a limit to how much time I can commit to that. So I need to have the important details summarized.
B: Just to make sure I understand: you’re saying that, even though you admit you’d learn more if you read the entire write-up, because of the various work pressures that you are facing, you’re making an explicit trade-off to reduce the amount of time you spend on this task, in order to have the time to spend on other tasks you have to work on?
On July 8, 2026, the Australian telecom company Telstra experienced a significant outage related to an NTP issue. Telstra hired a consulting firm to do an incident review, and a couple of weeks ago, that firm released a public report: Findings Report on the July 8th Telstra Outage.
I’m using this blog post to write up my understanding of the failure mode. I found the report was often missing important details that I needed to make sense of what happened, so I tried to fill the gaps by looking at additional sources, and made my best guesses where I could. It was also more difficult for me because I don’t have domain expertise in any of the technologies involved here: I have some basic familiarity with NTP, and I have no familiarity at all with the various mobile telecom technologies involved.
Note: all block quotes in this post are from the public report unless explicitly indicated otherwise.
2010-2020 Telstra NTP system architecture
Telstra runs three NTP servers in their mobile network, geographically distributed at telecom exchanges across Australia.
Telstra NTP servers run at exchanges in Sydney, Melbourne, and Perth
These servers provide time synchronization to their clients, which could potentially be other NTP servers.
A diagram that shows NTP clients, including a client that talks to two of the servers
Note that some NTP clients are configured to point to multiple servers for redundancy. The diagram above shows one such client. Clients connected to multiple NTP servers will play an important role later in the incident.
According to the report, starting in 2010, Telstra ran two NTP servers at the Sydney and Melbourne exchanges. At each site, one server ran at stratum 2, and the other ran at stratum 3. The Perth exchange ran a single NTP server, at stratum 3. For their stratum 1 sources, the stratum 2 servers were clients of NTP servers hosted at Australia’s National Measurement Institute. It is the Melbourne site that will play the most significant role in this incident.
Telstra system diagram, circa 2010
A note about arrows: NTP diagrams typically show arrows going from lower stratus to higher stratus (e.g., “stratum 1 →stratum 2″). However, I’ve drawn the arrows the other way because I’m used to the “client → server” convention of arrows directions.
I’m assuming that the NTP clients communicated only with the stratum 3 server, but there isn’t enough detail in the report to verify this was the case. I really wish they had drawn diagrams in the report!
For the purposes of this incident, the important detail here is that each stratum 3 server has a redundant stratum 2 source: all of the stratum 3 servers can use either the Sydney or Melbourne stratum 2 servers.
An upgrade leads to an old redundancy being lost, and two new redundancies added: all three changes will contribute to the incident
Six years before impact
In 2020, Telstra upgrades the timing hardware on their mobile network. With this upgrade, their NTP servers now run on Microchip SSU 2000 synchronization supply units, with the supporting vendor being a local company: Scientific Devices. This upgrade results in three significant changes to the architecture.
No more local NTP sources
Due to a limitation in the NTP timing chassis, a stratum 2 NTP server could not provide time to a stratum 3 NTP server in the same chassis. – Findings Report on the July 8th Telstra Outage
With this upgrade, they are no longer able to have a stratum 3 NTP server running on the SSU 2000 use another NTP server on the same unit as its source. As a consequence, the Melbourne and Sydney stratum 3 NTP servers lose a level of redundancy: they now each have only one stratum 2 source.
The new architecture eliminates the redundant local connections in Sydney and Melbourne
Unfortunately, the writeup doesn’t explain why the limitation existed. I tried to figure this out by reading what SSU-2000 docs I could find. My best guess was that this was a security restriction, as described below (emphasis mine). For context: the PackeTime module is the thing that runs NTP on the SSU 2000, and NTP runs on the service port. “Independent mode” means that the two Ethernet ports of the module run independently (i.e., bonded), which I presume is how they had things set up so they could run two independent NTP servers.
In the Independent mode, the two ports are each assigned unique IP addresses and can each serve time to independent networks. From a security perspective, the important thing to note about the Independent mode is that the PackeTime Module does not support routing protocols between the two service ports or between the service ports and the maintenance port. This prevents a malicious attack on Port A (network A) to be used to send a malicious attack via Port B (network B) or vice-versa, or between Port A or B networks and the maintenance port’s network. – SSU-2000 Synchronization Supply Unit and the SDU-2000 Synchronization Distribution Unit: User Guide, Revision G – June 2008, Chapter 8, Output Module Reference Data, PackeTime Module, p274
GPS card for failover
However, while the upgrade cost them one form of redundancy, it granted them another form: each SSU 2000 unit had a GPS card in it. GPS gives you access to a high-accuracy clock, since it receives time information from multiple satellites which each have atomic clocks in them. The presence of a GPS card gave the NTP servers running inside of the SSU 2000 access to a stratum 0 source. The GPS card was therefore available as a failover time source if the stratum 3 NTP servers lost access to their (now sole) stratum 2 source.
But there is a problem lurking within this failover solution: a GPS rollover bug in the card.
An announcement I found online about the GPS rollover issue for SSU 2000
Internally, GPS keeps track of the number of weeks since January 5, 1980, and it uses 10 bits to represent this count. This means that the count rolls over every 1024 weeks (roughly every 19.7 years). Some GPS devices were not originally designed to handle the rollover. The GPS card in the SSU 2000 is one of those devices. (One thing I learned here is that while the GPS rollover itself happened on April 6, 2019, devices that are vulnerable to rollover don’t necessarily roll over on the same day. For the SSU 2000, the rollover was September 18, 2022).
This means that the GPS card’s date will jump backwards in time after it rolls over. The rollover date is September 18, 2022. But just hitting that date isn’t enough: as long as the card is still running, it will behave properly. It’s only when it is powered off and on after that date that the rollover will happen.
Running the symmetric variant of NTP
There is also one additional architectural change that happens in the upgrade.
In the client/server variant, a persistent client sends packet mode 4 packets to a server, which returns packet mode 3 packets. Servers provide synchronization to one or more clients, but do not accept synchronization from them…
In the symmetric variant, a peer operates as both a server and client. – RFC 5905
The mobile core timing architecture was changed from client/server to peering, meaning that the NTP server in the NTP timing chassis could acquire any NTP source that it could find and select to use any of these sources that was providing the most stable and accurate time. This introduced uncertainty in the time sources being used in mobile core timing hierarchy. – Findings Report on the July 8th Telstra Outage
Here’s where I struggle the most with the writeup, because I’m not familiar enough with the underlying technology. RFC 5905, which describes NTP, does not use the term “peering” anywhere, but it does talk about multiple variants of the protocol, including the client/server variant and the symmetric variant. I assume peering in the report is referring to the symmetric variant.
Here’s an example of behavior which I believe can only happen with the symmetric variant. Consider the following network of five NTP servers, labeled A-E.
Example of a configuration of NTP servers
Now, assume that server A goes down. This means that node B no longer has a source. When running in client/server mode, C is a client of B, which means that B cannot be a client of C. However, when running symmetric mode, B can become a client of C, like this:
B uses C as its source after A goes down
This behavior will eventually contribute to increasing the severity of the incident by propagating bad time data further through the network. But we’re still six years away from that.
SSU 2000 GPS rollover day comes and goes with no issues
Four years before impact
On September 18, 2022, the rollover day for the SSU 2000, nothing happens. Because the GPS cards have been running uninterrupted, they do not roll over: that only happens after being reset. In addition, the cards are not even in use, since they are only used for failover, and failover has not occurred. The problem will only manifest when the GPU card is restarted, which will not happen for years.
Melbourne NTP server cannot reach Sydney, engineers fail over to GPS card
Nine months before impact
Several months before the outage, there’s a minor incident where the stratum 3 NTP server in the Melbourne exchange has been experiencing errors attempting to connect to its stratum 2 source in the Sydney exchange:
In October 2025, personnel were working on the Melbourne NTP server due to an incident report that the NTP stratum 3 clock was losing connectivity multiple times per day to its stratum 2 source clock at Sydney, resulting in NTP alarms.
The report notes that the server used peering to dynamically identify another NTP server, which happened to be a stratum 4 source, leading the Melbourne NTP server to become a stratum 5 source. However, they don’t know anything about this server:
The incident report also noted that the Melbourne NTP had become a stratum 5 source meaning that it was receiving time from a stratum 4 source in the mobile core timing architecture. Remember, Melbourne had only one stratum 2 source and when that source was lost, it used peering to find another source which happened to be a stratum 4 client of Sydney or Perth.
According to the report authors, this meant that timing loops were already present in the system (emphasis mine)
It is critical to have at least two independent reference sources. A timing loop is when sources that appear independent are in fact validating one another – directly, or by tracing back to a common reference. To avoid timing loops, you must clearly define the server hierarchy and document all relationships between servers.
…
looping was already occurring in mobile core timing when an intended NTP stratum 3 server was able to become a client of one (or more) of the clients and that a circular loop was established when the stratum 2 source was lost.
My understanding of the topology at the time of the October 2025 incident is captured in the following diagram, where the red arrow indicates a failed connection, and the blue arrows are network edges that were not in the previous diagram. Here I’m assuming the client in question was connected to the Perth stratum 3 server, but it may just as well have been the Sydney stratum 3 server.
A client of the Melbourne NTP server becomes its source after connection to Sydney fails
While this evidence indicates that looping was already happening in the network in October 2025, the looping itself had not yet contributed to an incident: that will come later.
The network engineers mitigate the incident by failing over the Melbourne NTP server to use the GPS card as its source.
To resolve this issue the GPS card was activated and connected to the stratum 3 NTP server in Melbourne as a means of replacing the lost Sydney stratum 2 source. the incident appear to have chosen this approach as the best means to address the incident. The GPS card is known as a reference source (aka stratum 0) and so this connection made Melbourne a stratum 1 NTP source, effectively the same level as the NMI stratum 1 sources in the mobile core timing architecture. This impacted the clients of Melbourne, making them stratum 2. In peering mode, these stratum 2 clients can be time servers at the same priority as Sydney and Melbourne stratum 2 who were the intended stratum 2 sources.
Melbourne NTP server becomes stratum 1 after GPS failover, its clients become stratum 2
A reminder from the SSU 2000 vendor about the firmware update
Seven months before impact
In January 2026, the vendor supporting the SSU 2000, Scientific Devices, sends a reminder to Telstra that there is a firmware update for the SSU 2000 related to the GPS card.
Here I’ll quote the CEO’s Senate testimony (emphasis mine):
It’s still subject to our investigation, but there was definitely a reminder in January of this year that this update needed to be done. We believe it does go back several years that it was there. To provide some context, our investigation so far has shown that it’s not a case of the teams just ignoring that update. They did consider it. The way this was designed and the way we were using this particular server and this card, this update was affecting a feature that we were not using at that time. … We know that our teams did get the alert of a software update. As we said, they assessed that through our investigation, we can see, and made a choice. Again, this is not an excuse; it’s putting it in context. There are many software updates and changes getting made across the network. Any software update creates a risk, and we put it into testing. In this case, they thought, given the design that was in place, it was not relevant because we were not using this equipment in the way that would trigger this risk. – (Telstra CEO) Vicki Brady, Senate Environment and Communications References Committee
It sounds like the engineer who made the call was not aware that the GPS card was a failover source. So was the use of the GPS card an in-the-moment workaround using available resources, or was it an explicit part of the architecture that the engineer making the call about firmware updates was not aware of? Or was it something else? There simply isn’t enough information to tell. The public write-up implies that it was part of the architecture:
The architecture indicated that the GPS card was available to be used as an alternative failover source for time of day.
On the other hand, we have this Senate testimony from the Group Owner, End-to-End Service Performance and Resilience of Telstra
I don’t know how to reconcile those two statements, beyond “different people in the system had different understandings of the architecture”. If you think you can eliminate these sorts of mismatches in understanding in your organization, good luck to you!
One last thing I want to point out here is the CEO’s comment that “any software update creates a risk”. So, not doing updates is dangerous, but doing updates is dangerous as well! It’s easy to judge the engineer in hindsight for this decision, but I have personally seen many incidents that were triggered by performing software updates.
Impact starts: A repair triggers a poisoned clock
On July 8, 2026, Telstra network engineers took the SSU 2000 offline in Melbourne in order to fix an issue with a redundant power supply:
When the maintenance team came to do the work on this server, they were required to power it down and take it offline because they needed to replace what we call the chassis, the case that it sits in, because its backup power supply was not working. So they put in a brand-new chassis. – (Telstra CEO) Vicki Brady, Senate Environment and Communications References Committee
Melbourne NTP servers go offline as SSU 2000 is powered down for maintenance
The report doesn’t say how long the unit was offline, nor does it discuss what happened to the Sydney stratum 3 NTP server that had Melbourne as its sole source. One thing that’s clear is that there was no system impact while the unit was down.
It was when the unit came up again that the trouble started. At 2:50 AM, the unit is powered up and comes back online. When the GPS card comes back online, the rollback issue manifests, sending its clock back 19.7 years, from July 2026 to November 2006.
The poisoned clock time spreads across the network
Since Melbourne was operating at stratum 1 and peering was enabled and Melbourne stratum 2 had been turned off by the same chassis replacement, the incorrect date propagated throughout the mobile core timing network.
Because the Melbourne NTP servers had gone down for maintenance, Sydney lost its source for its stratum 3 server. While presumably it had a GPS card to fail over to, they were configured to only fail over manually. So, unlike Melbourne, Sydney does not switch to using GPS card as a source.
However, because the NTP servers are configured to support peering, the Sydney NTP server is able to use one of its clients a source. When Melbourne comes back up, an NTP client common to the two sites becomes a stratum 2, using Melbourne as the source. The Sydney NTP server uses this client as its time source, and now the poisoned time has propagated to the Sydney site, with its clients impacted as well.
Orange shows the propagation of the incorrect time from Melbourne NTP server to Sydney NTP server
(Note: The report does not say whether the Perth NTP deployment was impacted by this, so I’m going to assume it wasn’t).
Impact: mobile users can’t call or use data
70 minutes after impact start
Starting at around 4am July 8th, 2026, when Mobility Management Entity (MME) instances began to reset, end users of Telstra’s network experienced varying levels of impact in their ability to communicate on the network.
At this point of the write-up, I’m really out of my depth, because the details of the failure are telecom-specific. I don’t even know if an MME instance is a software thing or a hardware thing. Based on this case study on Ericsson’s website, I assume that Ericsson is the vendor, and according to this Ericsson page, it sounds like it could be either one, as it says: Our latest software release can be deployed as a physical network function (PNF), as well as a virtual network function (VNF).
What I do know is that it provides a control plane function that communicates between users’ phones and the mobility network. When these instances reset, it broke mobile functionality, including emergency service calls (Triple Zero in Australia).
These impacts precluded some users from being able to make/receive voice calls, including Emergency Calls (per the Emergency Call Service Determination:2019, also referred to as Triple Zero calls), send/receive messages, or access mobile data services. Those without data services were unable to do things such as access the web, stream content and most mobile applications were unusable as well.
During the worst of the incident, 45% of mobile users were impacted:
Until the correct network date was restored, information for approximately 30,000 IP address prefixes were incrementally corrupted (written with the wrong date) by the Diameter Signalling Controller (DSC) in such a way that access attempts using those addresses were blocked from using the network regardless of available capacity.
My guess based on this is that a Diameter Signalling Controller (once again, I’m guessing they’re using an Ericsson product) is that the DSC keeps a table of session information about active IP addresses on the network. And because entries got written to the table with the rollback date, the system interpreted these as invalid and did not allow these addresses to access the network.
Note that this is a poisoned data scenario: network engineers had to fix these records before the users could regain access.
Telstra, with help from their equipment vendor, applied a workaround during the Outage and then at a later time performed repairs to remove the bad IP address prefix entries.
This is a pretty clear example of an incident that involved saturation. The original MME instance resets don’t take out the entire mobile network, but they have the effect of reducing capacity enough that the remaining capacity is not sufficient to meet demand. From the report:
Initially users were impacted by a loss of network capacity (due to failure of MME instances) which caused some users to be disconnected from the network. The remaining capacity was insufficient to enable those users to all reconnect deterministically nor was it capable of servicing the nominal network traffic load. So essentially the network was overloaded.
Problem area: networking
Admittedly, this one is a gimme: every large-scale telecom-related outage is going to be networking related, since what telecoms provide is network access! That was no exception here.
Problem area: security
It’s unclear to me in this case whether security systems played a role at all in this incident. I hypothesized that the loss of redundancy was due to a security restriction in the SSU 2000 unit, but I don’t know for certain that’s the case. It’s also possible that the Diameter Signalling Controller blocking the IP addresses was related to a security function, but there’s not enough information.
Essential non-standard changes: mitigating an operational issue
This is a clear case where the act of mitigating an operational issue was itself a contributor to a new incident. The trigger for this incident was repairing the power supply of the SSU 2000 unit in the Melbourne exchange. The repair itself went fine: it was powering up the unit that exposed the latent bug in the GPS card.
Essential increase in essential complexity: reliability subsystem
The GPS card, which was the source of the poisoned time, was a reliability subsystem! It exists to provide failover functionality in the case where an NTP lost access to its source. And unexpected behavior of that reliability subsystem, when engaged, bit them hard.
Essential increase in essential complexity: migration
I think Telstra’s 2020 upgrade of their timing hardware qualifies as a migration. This resulted in architectural changes that created a system vulnerability, although this vulnerability would not manifest for six years.
I do appreciate how the CEO was sanguine about how the nature of these sorts of changes means that incidents are inevitable.
What I would say is that networks and environments like a mobile network are absolutely complex, and it’s evolving—any sort of complex technology environment—and certainly networks are not infallible. But our job is to make sure we are taking the actions and those steps that will mitigate as best we can. I would love to be able to sit here and say will be zero outages. The reality of a complex network environment with fast evolving technology is that no telco around the world could guarantee that. – (Telstra CEO) Vicki Brady, Senate Environment and Communications References Committee
Some parting observations
There is no NTP haiku
In my experience, NTP isn’t a technology that frequently shows up incidents, at least not compared to other networking protocols such as DNS or BGP. The authors of the public report takes Telstra to task for a lack of sufficient NTP expertise within the company, and I can imagine playing a more significant role in the telecom domain than in other cloud software domains. But I can’t help but wonder if the fact that NTP isn’t generally problematic contributes to organizations having less in-house NTP expertise.
Appalling incuriosity from the investigators on how the work gets done
the need for stronger competency and more curiosity to investigate when something looks wrong
As is always the case in these sorts of public reports, there’s a lot of ink spilled on the subject what engineers should have done in the run-up to the incident but didn’t. The bullet “Increase attention to the operation and management process discipline in all areas” pretty much says it all.
Whenever I see this sort of content in a report, my response is, “boy, the investigators really blew it.” And that’s because you can’t understand how the incident happens unless you understand how the work really gets done within the organization. The authors show an acute lack of curiosity about the circumstances of the individuals who were involved in making the decisions.
I’ll note that these authors managed to mess up the section numbers in their report: it goes 1,2,1,2 instead of 1,2,3,4. This leads them to reference a non-existent section 4.1. I would ask the authors: Do you think we should be taking advice on how to run a reliable telecom network from people who can’t even number sections correctly in a ten page document? Or do you think we should consider the circumstances that could lead to this sort of error?
Where are the other system vulnerabilities?
We observed instances where other areas and elements of the network and organizations were performing at better levels than NTP.
Finally, it’s easy to point out problems in a system that were exposed by a recent incident. The really impressive thing, to me, is to identify the problems that were not exposed here. I have high confidence that there are other architectural vulnerabilities in the system, just waiting to be turned into incidents because of future changes. And yet, somehow, investigators never manage to find new vulnerabilities that could lead to the next outage. Funny thing, that.