On March 29, a Microsoft Security researcher identified a piece of malicious code that was embedded in a tiny little Linux library called XZ Utils. This piece of code turned out to be an extremely dangerous backdoor that could have put a significant number of businesses at risk.
The XZ Utils vulnerability is being tracked as CVE-2024-3094.
The open source community and the business world need to learn a valuable lesson from the XZ Utils vulnerability. In this article, I’ll discuss how this breach happened, why it’s so important, and what changes I expect to see as a result.
What Is XZ Utils?
XZ Utils is a library used for on-the-fly data compression and file transfer. It’s a significant part of a utility called Secure Shell. Secure Shell, or SSH, is the primary remote access mechanism for all Linux systems.
This particular malicious code allowed a threat actor with a predetermined encryption key to upload arbitrary code to any affected remote system and execute it without being authenticated. Whoever had the magic encryption key basically had the keys to the kingdom to any Linux system that was running this particular version of this library for XZ Utils.
The vast majority of Linux distributions use XZ Utils in one format or another, and this vulnerability could have been exploited at any time. As a result, the XZ Utils vulnerability was an extremely powerful backdoor called a remote code execution (RCE).
Why Is the XZ Utils Vulnerability So Scary?
There have already been hundreds of vulnerabilities exposed this year, but this particular one stands out from the crowd.
The XZ Utils vulnerability was not caused by a developer’s coding mistake. Malicious code was intentionally introduced by a developer working on the open source project. It was an intentional backdoor inserted into the project.
Even worse, this particular vulnerability was not caught by internal reviews on the open source project.
Essentially, open source is a way of developing and supporting code by making it publicly available. Anyone that has access to the public internet can download or contribute to open source code.
In theory, people love open source because the code is just that — wide open. It’s awfully hard to hide security flaws and bugs when anybody can look at the code, right? However, this particular exploit proved that nobody’s actually looking.
What better place to hide a significantly critical backdoor bug than in plain sight?
This attack was particularly insidious because the developer spent nearly two years in what we can only assume was a social engineering exercise. In order to gain others’ trust on the project, they contributed legitimate code, bug fixes, and other updates to XZ Utils.
They were incredibly patient in adding value to a project, only to turn around and sabotage it two years later. This person accumulated other developers’ trust until they got to a point where they could confidently insert code without it being scrutinized in any detail.
Think about this: there are nearly 4 million active open source projects. If this type of vulnerability could happen to XZ Utils, it could happen on any of those.
Open source libraries and tools can also be used absolutely anywhere. A great deal of commercial software and applications are chock full of open source components. It’s entirely possible that an attack like this has already been executed many times over, and we in the community just haven’t discovered it yet.
In fact, in a recent ZDNet article, it looks like another open source project may have been targeted for a similar attack.
How Does the XZ Utils Vulnerability Affect Businesses?
This vulnerability is an example of a software supply chain breach, because this code was a flaw from a third party — in this case, Linux. The threat actor knew Linux is used by business customers, so this was very much an intentional supply chain exploit.
Interestingly enough, one could potentially consider the XZ Utils vulnerability to even be a software supply chain exploit to Linux. The Linux distros didn’t necessarily directly write XZ Utils, and they were also getting it from an open source third party.
In this particular instance, hackers breached Linux itself by attacking a widely-used supporting library. However, it’s unlikely Linux was meant to be the end target. It’s far more probable that the threat actor intended to spread this vulnerability down the supply chain, all the way to business customers.
Thankfully, this particular exploit was caught very early in the release cycle. The library was still mostly in beta testing, so it only made it out the door to a small number of Linux distros, and those distros were not widely adopted.
The only users that were truly compromised by this vulnerability were those who were willing to take on the risks of beta testing. However, if the XZ Utils vulnerability had not been found, it would have continued to flow through the release channels. It’s very likely that this vulnerability would have eventually hit every major Linux distribution.
Given the nature of the XZ Utils vulnerability, businesses would have been rife for what we call a foothold situation. This is where threat actors work to gain access to systems and then hide from the threat hunters.
Threat actors with a foothold may not even know who their target system belongs to. Once they’re in the system, then they search for anything of value or interest.
Executing remote code through a foothold situation is often done very effectively through bots. A well-crafted bot with the predetermined key could have run rampant through the Internet. Once it found a system running SSH, it could have gained a foothold and taken anything of interest.
Essentially, if this vulnerability — or one like it — isn’t caught, anyone who uses Linux distributions could have had their systems pwned.
Related: How to Identify and Address Lurking Cyber Risks in Fintech
What are the Implications for Open Source Software?
Open source software is absolutely everywhere. Many people are using open source software and don’t even know it. They weren’t given a choice as to whether they want to use open source software. It’s just the way things are.
This attack clearly shows that people really aren’t looking for vulnerabilities in open source code. I strongly suspect that this particular incident may cause some ripples in the open source community.
Implications for Businesses
Businesses are constantly concerned with indemnification. If something bad happens to them, who is legally liable? And when it comes to open source, there really isn’t indemnification under most licensing schemes.
I’m not saying nor am I advocating that we should just get rid of open source. That’s certainly not the answer. But what this incident highlights is that there’s a pretty big gap in the open source world.
Because there is no indemnification, who’s actually holding these projects accountable for quality coding practices? How do we know the code was reviewed? How do we know there isn’t malicious code? What reassurances are there that checks and balances are happening?
Implications for the Open Source Community
Open source projects are also likely to experience transformation due to the XZ Utils vulnerability. In many cases, people don’t get paid to work on open source projects — they’re donating their sweat equity to something they love.
I think it’s very possible that people are going to be a little more skeptical of open source. And if people are more critical of open source, and if developers feel like they’re under fire, they may choose not to contribute their time and energy.
Personally, what I’d like to see is some sort of a self-published ratings system. As some initial thoughts, here are some of the processes developers could be required to run:
- Code reviews
- Static analysis
- Dynamic analysis
- Pen test
Developers can use a system built on these processes to show what measures they’ve taken to make sure that vulnerabilities don’t exist within their projects. Then, not only are developers saying they do these things, but they’re publishing the results as well.
Hopefully, this can begin to give people some assurances as to what’s being done to stop these types of exploits.
Related: 5 Big Security Problems and How to Avoid Them
How Can Businesses Respond to Vulnerabilities in Open Source Projects?
It’s unlikely that open source projects are currently doing much if anything to catch these types of vulnerabilities. In the interim, businesses need to focus on equipping software bill of material (SBOM) tools to identify specific open source components. Businesses will then need to perform their own assessments of the code.
The intent of SBOM tools are to identify all components and third-party libraries that go into any given software package.
If you have a solid SBOM tool, then you know what code you’re exposed to. That way, when common vulnerabilities and exposures are announced, you’ll know right away whether you’re affected.
Even more importantly, businesses need to take on some of the responsibility of reviewing code. Somebody on the supply chain has to run static and dynamic analysis, and it’s clear that developers aren’t doing it. Businesses are the group that is most affected by open source vulnerabilities, so they need to invest in making sure what they’re using is safe.
Manage Your Open Source Use Better With Black Kilt
Black Kilt can provide the coaching and guidance businesses need to fight against vulnerabilities of every kind. In this case, we can implement SBOM tools so they can begin to identify where they’re using open source.
We also help our clients build open source repositories and track tools, which helps them get a much better handle on what’s happening in their software environment. This is especially helpful for large businesses with complex ecosystems.
It can be hard to track every piece of your environment, but it doesn’t have to be impossible. Black Kilt is here to help, beginning with a free consultation. Give us a call to get started today.
