How a Malicious Xz Utils Update Nearly Caused a Catastrophic Cyberattack on Linux Systems Worldwide

Due to malicious updates that were made to xz Utils, the world was close to being infected by a recently uncovered backdoor incident. The impact had the potential to cause similar devastation to the SolarWinds attack in 2020 that allowed Russian hackers to penetrate the core of many United States government agencies.

The attack was “frightfully close” to success, and was described by software and cryptography engineer Filippo Valsorda as perhaps the “best-executed supply chain attack” that had been observed to date, Ars Technica reports.

Much of the internet’s crowdsourced code is vulnerable to infiltration by bad actors and nation-states. Open source software is at the “heart of the internet,” it is largely maintained by a handful of volunteers and that makes it a major security risk for corporations and governments alike, The Economist reported. Open-source software is commonly deployed across digital infrastructure because of its low cost. That infrastructure, which is embedded across the digital world, is under attack by various enemy nation-states.

The Recent xz Utils Open Source Scare

On March 29, 2024, Andres Freund, a software engineer at Microsoft, found a “backdoor hidden in a piece of software that is part of the Linux operating system.” This backdoor came from the source code for xz Utils, which was tampered with, and allowed unauthorized access to systems using the affected versions. The source code that was compromised was of the xz Utils open-source data compression utility in Linux systems. The New York Times wrote that the engineer prevented a “potentially historic cyberattack.”

Since xz Utils is open source software, anyone can see the code as it is public and what changes have been made then.

A developer named Jia Tan began making helpful code contributions to the project and slowly earning trust. Then over time, the bad actor smuggled in malware. Russia’s foreign intelligence service, SVR, suspected to be behind the attacks, is the same intelligence service behind the SolarWinds attack.

The Open Source Security Foundation (OSSF) warned of the xz Utils attack as likely not being an isolated incident. Bad actors were caught using similar social engineering tactics to try and take over other projects such as the OpenJS Foundation for JavaScript projects.

The Vulnerability of Open Source Software

Speaking to Frontsight media, Ryan Ware, an open-source software expert, explained the sheer magnitude of the risks at play:

“Our digital infrastructure is very vulnerable,” according to Ware. “To date, there are 177,914 CVEs that have been published. Let’s say for argument's sake that there are 1 billion lines of code in open source (it’s far more, but we’ll leave that argument for another day). Let’s also say for the sake of argument that half of those CVEs are for open source (90,000 for a nice round number). That means for open source code we’ve only found one vulnerability for every 11,111 lines of code,” he said.

“Companies would kill to have code so clean that there was only one vulnerability found for every 11,000 lines of code,”  “Additionally, while there’s a bit of a dip in the amount of open source software being written right now, we’re still at all-time highs in the amount of code being written,” Ware explained. He described a scenario where, for every vulnerability found, there are an additional 5-10 more vulnerabilities to be found before the code is sound.

Lesson from xz Utils

The magnitude of the XZ Utils near miss serves as a stark reminder of the fragility of crowdsourced software and the urgent need for failsafe installations, as Ware further explains:

“We absolutely know that nation-states look to subvert the security of software. All you need to do is look at the list of APTs that are out there.” Ware said, further explaining that “historically, these threat actors have been focused on using zero-day vulnerabilities to achieve their goals.” However, he also noted that “operationally, these threat actors do not just find a zero-day and then immediately go exploit it. They hoard zero-day vulnerabilities (discovered through their ownresearch or purchased) and then utilize them when they need one that satisfies their operational goals.”

In the meantime, Ware pointed out the long-tail duration of time between the existence of a vulnerability and the time it takes for software developers to patch their systems. Threat actors can hoard vulnerabilities for a long time as this delay in repairs continues.

An Unclear Line of Vision For Software Manipulation

What there hasn’t been much visibility over time is what nation-state actors have been doing to manipulate software. “The whole incident with xz has given a bit of a window into some of this including showing what kinds of resources these nation-states can bring to bear,” said Ware. However, the xz incident isn’t the only evidence that something of this social engineering sophistication has been tried. “Honestly, my worries are what has been done by nation-states in both the realms of open source software as well as commercial software that we don’t know about right now,” Ryan highlighted.

When asked if the SolarWinds incident has faded from recent memory, in Ware’s view, it depends on who you ask:

“I don’t think the lessons of SolarWinds have faded from government officials. Much of the work around software supply chain security in OpenSSF (such as SLSA and GUAC) is being done because CISA wants to see solutions in this area,” he said.

“I think it’s definitely dropped out of the public consciousness, but at the same time I don’t know how much it permeated the public consciousness," Ware added, highlighting a perspective with implications for the average hobbyist software developer, often a member of the “public consciousness.”