Overview
It’s been an absolutely manic week in the Linux security community as the news
and reaction to the recent announcement of a backdoor in the xz-utils project
was announced late last week, so we dive deep into this issue and discuss how it
impacts Ubuntu and give some insights for what this means for the open source
and Linux communities in the future.
This week in Ubuntu Security Updates
20 unique CVEs addressed
[USN-6718-2] curl vulnerability
- 1 CVEs addressed in Xenial ESM (16.04 ESM), Bionic ESM (18.04 ESM)
[USN-6719-1] util-linux vulnerability
- 1 CVEs addressed in Focal (20.04 LTS), Jammy (22.04 LTS), Mantic (23.10)
[USN-6686-5] Linux kernel (Intel IoTG) vulnerabilities
- 9 CVEs addressed in Focal (20.04 LTS), Jammy (22.04 LTS)
[USN-6715-1] unixODBC vulnerability
- 1 CVEs addressed in Xenial ESM (16.04 ESM), Bionic ESM (18.04 ESM), Focal (20.04 LTS), Jammy (22.04 LTS), Mantic (23.10)
[USN-6704-4] Linux kernel (Intel IoTG) vulnerabilities
- 5 CVEs addressed in Focal (20.04 LTS), Jammy (22.04 LTS)
[USN-6707-4] Linux kernel (Azure) vulnerabilities
- 4 CVEs addressed in Jammy (22.04 LTS)
[USN-6720-1] Cacti vulnerability
- 1 CVEs addressed in Jammy (22.04 LTS)
Goings on in Ubuntu Security Community
xz-utils backdoor and Ubuntu
- https://www.openwall.com/lists/oss-security/2024/03/29/4
- Late last week, 28th / 29th March backdoor in liblzma from xz-utils was
disclosed to the open source community via oss-security mailing list - this
was in the recent 5.6.0/5.6.1 releases from late Feb/early March this year
- initially the impact was not entirely clear - assumed initially that it may
impact only xz-utils and so only in handling of xz compressed data / files -
but even with that assumption that perhaps it could then infect anything that
got compressed/decompressed
- within a few hours though became clear that the primary target was not
xz-utils/liblzma itself but openssh - and the affect was to provide a backdoor
into openssh that would allow the attacker to get remote access to any machine
running ssh server with this backdoor liblzma installed
- To paint a picture - this was all unfolding on late Thursday / Friday of the
Easter break, so lots of folks were either EOD or on leave etc - and trying to
grapple with a threat that we knew could possibly impact the impedending 24.04
LTS release
- Good news:
- TL;DR for Ubuntu, this version was only ever in the -proposed pocket for the
currently in-development 24.04 release - not in any other Ubuntu versions -
and was removed as soon as we became aware, so unless you are running the
devel release AND you had manually opted to install this version from the
-proposed pocket, you would not be affected - very lucky
- https://discourse.ubuntu.com/t/xz-liblzma-security-update/43714
- What do we know about this? A lot - there has been significant investigation
(and speculation) since it was announced, both at the social side of things
and the technical aspects of the backdoor itself
- For the purpose of the podcast, we won’t go too deep into either but will try
and cover the salient details
- Regarding the inclusion of the backdoor itself - looks to have been a very
long and patient campaign by the attacker, who slowly gained the trust of the
upstream project over the last 2 years and likely pressured the maintainer via
sock-puppet accounts to then get themselves added as an additional maintainer
- Since then they seemed to be quite a good maintainer themselves - diligently
adding new features and bug fixes etc over the past 2 years, but then suddenly
introduced the backdoor into the most recent 2 releases
- The method of introducing the backdoor was also interesting, in that it
required 2 parts - the binary containing the backdoor and the code to get this
compiled into the liblzma library at build time
- The binary was committed into the upstream git repo disguised as an xz
archive itself used as part of the test suite
- The code to inject this into the build was NOT part of the git repo, but
instead was just in the tarball prepared by the maintainer for the official
release
- And it used many levels of obfuscation to hide this backdoor within that
fake test xz archive
- So the attacker was not just patient but also very technically skilled - and
not just multiple levels of obfuscation in the build process but the backdoor
code itself contained many elements to try and make it harder to recognise and
reverse engineer, presumably to allow it to hide in plain sight
- although as we will see, this runtime obfuscation within the backdoor binary
was what gave it away eventually
- It’s often said that one of the advantages of Open Source is the huge
community, which is summarised as Linus’ Law - with enough eyeballs all bugs
are shallow - but sadly this wasn’t proven out in this case
- Backdoor was not
found by anyone doing review of the changes upstream or by the various distros
like Debian / Fedora / OpenSUSE / Arch or even Ubuntu when incorporating this
new version into their repos - but instead was found by Andres Freund, one of
the maintainers of PostgreSQL, when they were looking to benchmark some new
changes scheduled for the next PostgreSQL release
- Luckily decided to use Debian unstable for this, and Debian had incorporated
this new version into unstable a few weeks ago, and wanted to get the
performance noise floor of the system as low as possible before doing
benchmarking of PostgreSQL - noticed large transient CPU spikes in sshd and
then eventually weird memory errors in sshd due to bugs in the initial version
of the backdoor
- After a lot of painstaking work was able to determine that liblzma was the
culprit and appeared to contain some very strange code to hook into the
authentication process of sshd when it was launched via systemd
- Was able to trace that back to the aforementioned manually prepared tarballs
of xz-utils on Github
- The CPU spikes Andres observed were due to the use of things like a trie to
lookup symbol names at runtime, rather than directly encoding them in the
exploit binary, presumably to try and make reverse engineering of the binary
harder (since you can’t just run
strings
on it and get any real sensible output)
- Some excellent writeups have been done regarding both the technical aspects of
the backdoor itself, as well as the process taken by the attacker to
incorporate this into the xz-utils project - both from a community point of
view and a technical point of view
- From a technical point of view, the impact of this backdoor was to allow an
attacker to get pre-authentication remote-code execution in sshd via a
specific private key when connecting to the server - NOBUS
- Hooked into the RSA certificate validation process in sshd, looking for a
particular matching private key from the client - and if found would then
proceed to execute arbitrary commands specified by the client without
requiring usual authentication
- By using this mechanism, nobody but the attacker can use the backdoor since
they don’t have the matching private key - so the impact of the backdoor is
somewhat limited
- BUT the fact they targeted such a widely used and deployed package across a
huge number of distros, means they essentially wanted a backdoor into any
Linux machine in the future that only they could use
- Interesting to try and speculate who the attacker could be (nation state?) and
what their intended purpose was especially given this wide reaching goal of
getting this into all the major Linux distros - but it would be just
speculation
- So rather than speculate, for the purpose of this podcast episode, interesting
to look at the timeline as it concerned Ubuntu
- The publishing history of the package is all visible in Launchpad
- Packages in Ubuntu get inherited from Debian who also publish history
- Upstream published the first backdoored version 5.6.0 via GitHub tarball on 24th Feb 2024
- Debian incorporated this into unstable on 26th Feb
- On 27th Feb, the Ubuntu Archive Auto-Sync bot copied this version into noble-proposed
- Due to the ongoing time_t transition, sat it noble-proposed for the next month
- Security team heard whispers of the possible backdoor just hours before it was
publicly disclosed, and as soon as we heard of the possible backdoor, and
realised that it only affected the version in-development noble-proposed we
quickly notified the Archive Admin team who then deleted it from
noble-proposed on 29th March, neutralising the main threat
- Most important thing to know for Ubuntu, we have taken a very conservative
approach - not only have we removed this version from noble-proposed as soon
as we became aware, we are then rebuilding every binary package that got built
since that compromised version was in noble-proposed originally - out of an
abundance of caution - we don’t have any information that says this backdoor
was doing anything other than what the various writeups have found so far, BUT
we also can’t be certain that it didn’t have other functionality either - so
being very cautious and rebuilding everything that was itself built since 27th
Feb
- As such, delayed the development of 24.04 so beta release is slipping by one week
- One of the most interesting parts is the sheer luck that this was found - not
by security researchers or maintainers but by a developer from Microsoft who
happened to be looking for the right things at the right time and decided to
be curious
- Also for Ubuntu, luck that the time_t transition in Debian/Ubuntu caused many
packages to be stuck in noble-proposed and not in the release pocket, else
many Ubuntu users and developers would have been impacted if this had migrated
to the noble release pocket
- Also interesting that the attacker appears to have had quite a good grasp on
OSS development practices and was quite persistent in trying to get this
incorporated into distros - even urging for this new version to be synced to
Ubuntu so that it would land in the upcoming noble release as recently as
Thursday last week, just hours before the public disclosure
- Not only could they do all the original social engineering work upstream,
and technical work to develop and hide the backdoor, but could then
interface with distros via their established practices to try and get them
to incorporate their new backdoored version faster than they may have
otherwise
- Huge amount of work has been done to detail both the timeline of the attack as
well as the technical details of both the code used to incorporate the
backdoored code into the final liblzma binary during the build process, as
well as the details of the backdoor itself and how it operates at runtime
- In the end highlights both the challenges and strengths of OSS - lots of OSS
projects have long dependency chains - in this case openssh when integrated
with libsystemd which in turn used liblzma - and it is unclear who the
maintainers and authors are or what procedures are in place for vetting and
transferring ownership of OSS projects - all present significant challenges
for OSS
- However, significant strength of OSS is the visibility and ability for anyone
to get involved, which is what we saw in the aftermath - despite all the
advanced obfuscation techniques employed was able to be analysed in a matter
of days by the community working together - and to analyse it in a huge amount
of depth and in such an open way that it leaves little room for questioning
the validity of the assessment - anyone can double check the work and come to
the same conclusions
- This isn’t the first software supply chain attack and likely isn’t even the
first against an OSS project but it is a wake-up call to the OSS and Linux
ecosystem
Get in contact