A Tale of Two Memes: How we came within 0.5 seconds of being seriously screwed

Memes Tell Tales

Memes are a great way of explaining complex topics. Allow me to demonstrate:

This meme describes “the entirety of all computer-based technology you use today – explicitly, and implicitly”. What do I mean? I mean all the computing software you use directly – as in, you’re a developer, and off you go, doing some development. You’ve got your operating system, hopefully doing what it should do. You’ve got your hardware, hopefully doing what it should do. You’ve got your drivers, hopefully doing what they should do. You’ve got your software tools, hopefully doing what they should do.

If any part of that chain isn’t doing what it’s supposed to be doing – how do you know?

I’ll give you a moment to think about that.

If any part of that stack was doing something, covertly, and working against you – how would you know?

Let’s get scarier.

Computer-based technology you use implicitly. The MRI scanner you had last week; it’s a computer. It has an operating system. How do you know it is working correctly? How do the developers know it’s working correctly?

It does the thing it’s meant to do.

Okay – fair answer. And hopefully, there are a rigorous set of tests that have been developed – perhaps a suite of unit tests – to ensure that the output from the MRI is ‘correct’. By ‘correct’, I mean it measures the output from the MRI against the input, and uses some pre-determined set of criteria to say “True”, or “False”, “Pass” or “Fail”.

But someone needs to be able to maintain that MRI. Someone needs to be able to update its firmware, its underlying software, its operating system. Hopefully, we can’t just remote straight in to the MRI. But maybe we can get in, somewhere in that process, and work our way across?

The point I am trying to make is that software we rely on, either explicitly, or implicitly, uses a lot of other people’s code. If we look at the list of 3rd party libraries use in Chromium (The basis of Chrome), we can perhaps get a better understanding. I’m reading that, as about 327 libraries. Each of those libraries is a separate project, that some random person in Nebraska may be thanklessly maintaining.

The fall of the Bazaar?

If the MRI developer is creating tests – are they creating tests to test what they have built, or to test the libraries that someone else has built, that they use? If they are testing “Yep, that’s a valid MRI of the human brain”, there is likely to be a crazy amount (refer back to the image above) of other peoples code that they are using, that they are assuming “just works™”.

In fact, the world’s first open source MRI scanner released its source just last week.

Where we are, today, assumes that hundreds, if not thousands, of libraries of other peoples code “just work ™”. Why do we assume it “just works”? Thanks to Open Source, and The Cathedral and the Baazaar.

One of the most famous essays on Open Source and Linux, is The Cathedral and the Bazaar, by Eric S. Raymond (seriously, give it a read). In it, he famously formulated a law that he calls Linus’s Law, stating that “given enough eyeballs, all bugs are shallow.”. When it comes to the hot, sexy open source project of the month – sure, that makes an awful lot of sense. But let’s go back to the meme.

An awful lot of technology that we take for granted relies upon lots of other software, that the developer takes for granted. And that random person in Nebraska, who’s been working tirelessly since 2005 on that boring cog in the system, is starting to get a bit pissed off.

Which brings us to the actual story. A tale of intrigue. A tale that ‘may’ involve spies, and nation-states doing devious things. Planting seeds. A weary, underpaid and over-worked project maintainer. And the potential for a ‘colossal’ security vulnerability. Subterfuge. Geeks.

What is XZ Utils?

A superb set of command-line, lossless data compressors (I’d link to the GitHub page, but it has been suspended. More on that later…). It has been around for about 15 years, and built on other significant work on compression. And when other projects need to deal with lossless compression and decompression, it’s a frequent go-to solution.

It just works™.

Well, it “just worked”.

It turns out, the poor developer (not actually from Nebraska) was feeling burnt out. They were maintaining the project as a hobby – and users of the project would come along and make demands. Fix this! Add that! Change this! They need help supporting this library.

The only person who offers to help, turned out to have ulterior motives…

They created a backdoor in the code. A very clever, well-hidden backdoor. It has a CVSS v3 score of 10.0.

Here is a list of Linux distributions that are directly impacted by this vulnerability.

Linux distributions vulnerable to CVE 2024-3094

It turns out that version 5.6 and 5.6.1 of XZ Utils have been carefully, strategically altered to include a backdoor. This whole operation has been years in the making, with the initial social engineering, getting in with the maintainer, building confidences, gaining permissions to the project…

The attacker was moving around the open source community, asking for their updated version be included in the latest operating system rollouts. In some cases, it was already included, per the table above.

What did the backdoor do?

It is still being picked apart, but it appears it would impact the SSH login process, as it interacted directly with the sshd process. Chances are, it would’ve allowed for the backdooring via SSH into infected systems. That’s bad. Very bad.

A really good, thorough write-up providing deeper technical explanation is available here.

The malicious versions, researchers said, intentionally interfere with authentication performed by SSH, a commonly used protocol for connecting remotely to systems. SSH provides robust encryption to ensure that only authorized parties connect to a remote system. The backdoor is designed to allow a malicious actor to break the authentication and, from there, gain unauthorized access to the entire system. The backdoor works by injecting code during a key phase of the login process.

https://arstechnica.com/security/2024/03/backdoor-found-in-widely-used-linux-utility-breaks-encrypted-ssh-connections/

As an example of how ‘integrated’ this tool is, using my Windows WSL Ubuntu instance, I did a reverse-dependency check, to find out how many of the tools currently installed on Ubuntu require xz-utils.

shem@ShemPC:~$ apt rdepends xz-utils
xz-utils
Reverse Depends:
  Depends: autopoint
  Depends: needrestart
  Depends: livecd-rootfs
  Depends: lintian
  Recommends: libdpkg-perl
  Depends: dpkg-dev
  Depends: unattended-upgrades
  Depends: ubuntu-standard
  Suggests: tar
  Depends: sysstat
  Depends: printer-driver-pxljr
  Depends: printer-driver-ptouch
  Depends: printer-driver-postscript-hp
  Depends: printer-driver-m2300w
  Depends: printer-driver-foo2zjs-common
  Depends: openprinting-ppds
  Depends: needrestart
  Recommends: mailcap
  Depends: livecd-rootfs
  Depends: lintian
  Recommends: libdpkg-perl
  Depends: hplip-data
  Depends: hplip
  Depends: foomatic-db-compressed-ppds
  Suggests: file-roller
  Depends: dpkg-dev
  Recommends: devscripts
shem@ShemPC:~$

27 tools, currently installed on my VM, require xz-utils. Luckily, as Ubuntu did not agree to push out the latest version, I’m safe – as I can check, with:

shem@ShemPC:~$ apt show xz-utils
Package: xz-utils
Version: 5.2.5-2ubuntu1
Priority: standard

This thankfully shows I have version 5.2.5 installed, not the backdoored version 5.6 or 5.61, although that doesn’t mean we are 100% safe. Whilst 5.6 is the version impacted by the specific backdoor we do know about, Jia Tan worked on the project extensively for the last 2 years(!). The specific ‘safe’ version of xz-utils people should be using is still being debated.

How was it detected?

This is the one part of this story that actually gives me some hope.

One suspicious geek.

He noticed that his SSH logins were taking about 0.5 seconds longer than they should, and became suspicious.

The xz backdoor was initially caught by a software engineer at Microsoft. He noticed 500ms lag and thought something was suspicious.

This is the Silver Back Gorilla of nerds. The internet final boss. pic.twitter.com/6IyJQ2tpMm

— vx-underground (@vxunderground) March 30, 2024

That got him digging, and investigating. It’s a moral to us all – get to know and understand the tools you use. We all get a little too comfortable, and a little too complacent.

The tl;dr – Standing on the shoulders of giants

We are, where we are today, because we stand on the shoulder of giants. There is no way any one of us can fully ‘know’ our entire stack – implicitly, or explicitly. When you get your money out from the ATM, you have to assume some level of trust in your bank, and the dodgy Windows XP-powered ATM they use.

There are limits to what we can test, and what we have to assume is tested by others.

Open Source is a fantastic solution. The alternative – the Cathedral – has many limitations, as the original essay pointed out. Even earlier prominent proponents of the Cathedral have switched, and now fully endorse the Bazaar model. The Microsoft that called Linux “a cancer that attaches itself in an intellectual property sense to everything it touches” has changed significantly – as demonstrated by the Microsoft engineer that picked up on the issue.

However, let’s be real for a moment.

It is highly unlikely that the massive swing to supporting the Open Source model has been due to some kind of utilitarian moral pursuit by FTSE 100 corporations. They saw value in outsourcing significant parts of their software development costs to overworked volunteers, willing to maintain critical libraries in their free time.

The result is a Jenga tower, supported by the good will of some random person in Nebraska, who has been thanklessly maintaining the same project since 2005.

And the (possibly nation-state-backed) ‘good Samaritan’ who stepped in during their hour of need…

We could ‘easily’ be living in a world ‘right now’ (this has happened in the last few days), where various major operating systems ‘had’ pushed out a CVSS v3 10.0 vulnerability to their users, backdooring their systems. And none of us would be any the wiser…

As we don’t yet know exactly how many other projects Jia Tan worked on… maybe we’re already living in that world…