taigrr/Megathread.md

## Megathread.md

      
    Raw
  

              Megathread.md
            
          
    SaltStack CVE-2020-11651 and CVE-2020-11652 Attack

Pre-Update

May 5, 7:00PM PST:
If you are reading this for the first time now, you will now need to nuke and destroy any system paired (e.g. minions) to an exploitable salt master.
You should also create a new salt master.
I recommend creating a backup first for forensics, but do not plan on ever reusing these systems.
Information on how to tell if you could be affected is available below.
Even if you didn't notice any unexpected symptoms, please: nuke and restart.
I cannot stress this enough: we have identified three five unrelated attacks so far which exfiltrate everything from databases, to bash history files, to ssh keys, to entire home folders.
Please snapshot your servers and start fresh.
Determine what could have been taken--if it's on your server, assume it was taken.
In fact, we have evidence to suggest some new attackers are actually including automation scripts inside their payloads to remove the mining exploit so their own exploit can fly under the radar.
Just because your machines don't seem like they've been affected, does not mean they weren't affected.
Please don't dismiss this notice as paranoia.
I have actually seen this attack play out, and attackers are using a wide net.
Do the right thing.
For more information, take a look at "Alternative malware" under the Malware Version Breakdown section.

Update Bulletin

Everyone, I've wanted to be as open about this as possible and help as many people fix their machines ASAP!
I believe we as a team have been for the most part, very successful.
Many, many sysadmins have been able to salvage their data, if not recover completely from an otherwise benign mining operation.
However, many different exploit POCs are surfacing across the web.
New iterations of the malware are showing up, and I (and others) are somewhat certain they are following the Slack thread (and saltexploit) to disguise their malware as the original payloads, when in reality their goal is very different, and possibly far more harmful.
This means two things:

I think we have hit the point where saltexploit's openness is now more harmful than it is helpful
Promising "if you have this md5sum...you only have to do this!" can now be misleading as it's possible a new attacker can intentionally disguise his malware as the original

Please stop posting code samples in the main chat and on the Github issue.
Please stop posting links to new payloads.
My DMs are open on Slack, and I will help take care of filing abuse reports etc. with the credibility and (mild) publicity I have already established.
Atlassian, for example, is now responding to my requests immediately (and good on them!), as is BunnyCDN.
At risk of soundling like a shill, I would like to reiterate my thanks to Atlassian and BunnyCDN in particular for their speedy responses and countermeasures.
Justin Black has been very helpful in diagnosing problems and testing the different versions.
I would consider him trustworthy regarding this issue as well, and his DMs on Slack are also open.
There are others, but I won't name them unless they tell me it's okay.
However, I believe we have "strains" rather than linear versions now, with new players on the field.
The remediation advice available on saltexploit will remain such that sysadmins can use it to help keep systems alive long enough to investigate whether any data was exfiltrated, and to report new vectors of attack to me and/or Justin.
The "fix" however will change--you must wipe and reset to be sure everything is gone.
Please be very careful when using your old config files as well.
You don't want it sneaking back into a new network.
I recommend you take this as an opportunity to assess whether you are happy with your current hosting provider(s) as well.
Again, I really believe in the open nature of collaboration and I really hope all here believe that, I've been extremely responsive and free-flowing with information, but there is now actual evidence that openness is no longer the right play.
Additionally, please note that all the troubleshooting and advice here stems from the mining exploit.
I cannot guarantee other, sneakier exploits were not first.
I wish you all good luck!
End of Update

If you're here, IMMEDIATELY update your salt-master. And don't forget to restart it too! turn off your Salt Master!

You'll need to start it in recovery mode with a chroot, more than likely, and remove any files you need to start up a new instance.
If you need to keep the system connected to retrieve configuration files, firewalling it from the public internet and disconnecting minions may also be a great idea. is a must!
What we know so far


Github Issue: saltstack/salt#57057
Caused by exploit to CVE-2020-11651 and CVE-2020-11652, which when combined allow for Remote Code Execution (RCE) as root on minions and masters.

This is was a crypto-mining operation


salt minions are affected, but the masters seem to have been left alone this time (unless the master is itself a minion.) and as of version 5, masters may be as well
salt-minions is a compiled xmrig binary.
salt-store contains a RAT, nspps, which continues to evolve and become more nasty
Atlassian confirms the newer revisions of the binary are a version of h2miner

Additional information


As a RAT, salt-store is more concerning than salt-minions. More on that later
There have been at least 5 different versions of the salt-store payload, each more advanced than the last.
There are additional backdoors installed, and private keys are getting stolen right and left

Seriously, change out your keys!


Symptoms


very high CPU usage, sometimes only on 1 core, sometimes on all

Fan spin! (Physical hardware only, of course.)


Mysterious process named salt-minions is CPU intensive, stemming from /tmp/salt-minions or /usr/bin/
additional binary in /var/tmp/ or /usr/bin named salt-store or salt-storer
Firewalls disabled
Most applications crashing or taken offline

Big Red Flags

The initial malware payloads act like a sledgehammer.
They tears down anything remotely CPU-intensive so they can suck out all your compute resources for that sweet, sweet Monero.
This includes, but is not limited to:

confluence
databases (i.e. redis)
docker containers
java applications
other cryptomining software
rabbitMQ
webservers
some assorted cloud platform services

There are additional system side-effects

We can confirm it often does the following:

disables SELinux
enables hugepages
turns off all system firewalls (hosting provider firewalls won't be affected)
disables AppArmor
disables the nmi watchdog
basically, it turns your machine into a dedicated mining rig

The malware may also take over your redis database and use it to replicate the attacker's data. Please check your databases' integrity!

The full shell script used as the attack vector is was available in the Slack channel.
I don't want to risk breaching GitHub's ToS by distributing it here, and I'm sure not going to throw it up on my site and distribute it for free.
However, for versions 1-43 the malware doesn't seem to do much else, provided you aren't running an unauthenticated Redis instance.
Note: version 4 did not originally ship with a backdoor that did anything but check for updates.
However, we have reports the update checker did evolve after a couple hours to do more, such as running the version 5 worm mentioned below.
Am I At Risk?

If you're here, chances are you're already compromised.
Note that this the initial malware is was Linux-specific but BSD users should still investigate their systems.
Windows users need fear not.
there are now so many strains it is very possible BSD and Windows hosts are affected as well.
We have heard reports the initial scripts didn't infect FreeBSD, but they did still manage to delete a lot of files.
Be on the lookout for missing files.
Again, this says nothing to the newer strains out there.
As the initial vector uses a shell script, many actions were run on your hosts, but it's doubtful it did much.
In case you're not sure if your salt-master could have been compromised, take a look at these tools:
Web API

Update: Saltstack has released an official tool for checking salt-masters for vulnerabilities.
Hopefully, anyone who refrained from using either of the tools mentioned below due to trust issues will be willing to give the official one a try, as it is blessed by SaltStack.
https://www.saltstack.com/lp/cve-patch-verify/
Prior Tools
The following command will create a file named "/tmp/HACKED.txt" on a vulnerable salt master when provided its public IP address.
curl -X POST -F 'ip=your.saltmaster.ip.address' https://saltexploit.com/test
This tool is meant to be fast and painless, but you should never trust strangers on the internet, and using this API requires a certain level of trust in me as a developer.
It was shared with @jscottsf and @jblac before it was announced for peer review and auditing.
I can personally endorse this tool as the author, but be careful what tools you use.
Telling 3rd-part sites and random strangers your compromised salt-master IP is a very, very bad idea.
Offline Checker

There is an offline checker made available here, courtesy of @rossengeorgiev.
How do I fix it?!

The best tried-and-true answer is: wipe and restore all your systems.
However, that's not possible for everyone to do right now, so here is what you can do.
The first step is updating creating a new salt-master.
Install all updates for your distro, and then test to see if your new salt master is still vulnerable.
If it is, take a look at "Tips for Updating Your Master"
Next, determine which version of the RAT you have.
If you didn't remove all instances of the binary by May 3, 4:15 AM PST you likely have version 5 or a hybrid payload, as the RAT binary auto-updates.
However, please find salt-store in /var/tmp and match the md5sum against one of the four in the section titled "Version Breakdown"
Clearing Out v1

All you need to do is kill all salt-store and salt-minions processes, then reboot.
This is outdated information.
An attacker can pretend their payload is salt-store but really run a second, sneakier payload in the background.
If you didn't catch it in time, meaning before May 3, at 4:15 AM PST, assume you were updated to the latest version that was available at the time you froze your minions.
Additionally, if you didn't catch the payload before May 5th, 7PM PST, assume exfil has happened.
It's also possible exfil happened before then, but that's the earliest it has been reported so far.
You will need to determine what else besides updating the master and deleting the binaries you need to fix for your own systems.
Please see a list of changes and side effects in the sections labeled "There are additional system side-effects" and "Big Red Flags" to determine what steps you need to take.
Clearing Out v4

Many people are having trouble removing version 4 of the malware.
It's much trickier than versions 1-3, and has better persistence mechanisms.
On each minion, you'll need to look for a very weirdly named process, it's usually got 5 characters long of a random string in the middle, a la NXiQS, and kill it.
Then remove everything else (salt-minions and salt-store) again.
That process is watching the other binaries and replacing them.
Also, delete /tmp/.ICEd-unix/ if you have it, then reboot your machine.
Please report in the slack channel if these steps do not work for you.
You will need to determine what else besides updating the master and deleting the binaries you need to fix for your own systems.
Please see a list of changes and side effects in the sections labeled "There are additional system side-effects" and "Big Red Flags" to determine what steps you need to take.
Note: Many people first found they had the malware installed at this iteration.
This is the last version of the malware you can be reasonably certain did not exfiltrate your data (again, with a very notable exception detailed below).
However, as with version 1, a new attacker may install version 4 to cover their own, sneakier tracks.
If you did not completely remove the malware by May 4, at 2:57 PM PST, consider your system unrecoverable.
At this point, version 5 came out, and version 4 began to install the worm as well.
Do some work to find out what may have been taken and modified, and feel free to contact me on Slack if you think you found something new, but DO NOT try to continue using these systems.
Clearing out v5

No known steps are yet available for version 5 as it has not yet been investigated.
No remediation steps will be given.
If you suspect you have version 5 or some other variant, please shut down all potentially affected systems immediately.
The malware behaves as a worm, reaching out all other hosts it can find to infect everything in sight.
## Personally, I'm seeing this event as a nice big vaccine.
A little pinch at the beginning, but nothing really bad happened.
Yes it was bad.
It was embarrassing.
But would you rather they have taken customer data and/or your private SSH keys?
What if he had taken all minions offline, or destroyed and replaced system binaries?
So far, we have no evidence to suggest data exfiltration has occurred, with one significant exception, detailed below.
Many sysadmins were lucky. Many more were not. This quickly became very bad.
The end goal may still be to mine crypto, but the methods have changed.
Unsecured systems are being compromised right and left.
Tips for Updating Your Master

As per @Exordian, some official repos do not yet have the fix upstream for salt-master.
Even after updating your whole system, you may not be running a patched version of the salt-master.
It's important to verify you've updated to the latest release, which may be more recent than what's available latest.
Look under "Additional Help" to find some verification tools.
You may want to look into using SaltStack's repos instead of your distro's to get the patch sooner.
Fedora Users: This link may be helpful to you.
Note you must restart salt-master for changes to take effect after updating.
No software patches are required minion-side~, but you still need to rotate keys.~
No longer relevant as advice is now to wipe and recreate any and all minions which were affected.)
Additional Recommended Step

In order to run unauthenticated Salt commands, you must first expose the Salt Master's Key.
This means it's possible (and very likely) the attacker may have a large database of IP/Key pairings.
ROTATE YOUR KEYS!!! It's possible to continue to send commands against an updated salt-master if you already have salt's private key (because it actually would be "authenticated" by the stolen key.)
Take a look at this rekey repo (credit to @dwoz) and seriously consider rekeying your master and minions.
Note that you may lose access to rekeyed minions not connected to the master during the rekey event.
Please plan accordingly.
Relevant Timeline

Note: timeline from March 16-April 30 sourced from F-Secure and is abridged.
March 16, 2020: F-Secure discloses vulnerability
March 24, 2020: SaltStack confirms issue, begins patching
April 23, 2020: SaltStack announces vulnerability, cautions users
April 29, 2020: SaltStack publishes patches to official SaltStack repos
April 30, 2020: F-Secure publishes additional advisory
Saturday, May 2, 2020:

~6:18 PM PST: salt-store (version 1/v1) payload is uploaded to Selectel CDN, attack begins
6:30 PM PST: payload is uploaded to Bitbucket
~6:45 PM PST: Github issue opened by @xiaopanggege , many other devs with monitoring tools follow suit
~7:00 PM PST: @aidanstevens29 correctly identifies this as an exploit of CVE-2020-11651 and CVE-2020-11652
Initial testing and debugging occurs. Several developers in the GH issue claim complete malware removal.
Midnight: Atlassian is notified by @taigrr of the malicious usage. Highest-level support ticket opened.

Sunday, May 3, 2020:

4:15 AM PST: salt-store v2 uploaded to Bitbucket
6:07 AM PST: salt-store v3 uploaded to Bitbucket
6:11 AM PST: salt-store v4 uploaded to Bitbucket
11:12 AM PST: Initial Slack Channel created to collaborate and discuss troubleshooting (35 developers joined)
12:39 PM PST: New, public Slack Channel created, 400+ developers joined and rising (many through the bridged IRC channel)
4:43 PM PST: Atlassian removes salt-store v4 from Bitbucket; script now falls back to Selectel

Monday, May 4, 2020:

1:00 AM PST: @taigrr sounds the alarm that patched, official Docker images for Saltstack have not been built or released
2:35 AM PST: New docker images available (thanks S0undt3ch!)
~1:00 AM PST: Another sample exploit released
~11:00 AM PST: @itskenny0 posted a cleanup script to assist recovering affected minions
2:30 PM PST: First report of dropper.sh, a malicious worm designed to steal SSH keys and add another backdoor
2:57 PM PST: salt-store v5 uploaded to Bitbucket
4:50 PM PST: New Bitbucket Repo detected by @jscottsf -- now defunct
4:51 PM PST: New repo reported to Atlassian by @taigrr
4:57 PM PST: Atlassian removes salt-store v5 from Bitbucket; falls back to a VPS
5:00 PM PST: More POCs surface on GitHub (seriously, when will people learn?)
5:13 PM PST: Abuse reported to UAServers for malware distribution
5:15 PM PST: @jblac begins initial testing of v5

Tuesday, May 5, 2020:

~9:03 AM PST: Selectel alerted to affected host by Ivan
~9:20 AM PST: Alias Landergate, and @taigrr sent abuse reports to SelectelCDN, BunnyCDN
9:22 AM PST: No response from UAServers, but affected host removed
9:22 AM PST: New host reported to UAServers by @taigrr
9:32 AM PST: Team voice notified of affected host
9:53 AM PST: New host reported to UAServers by @taigrr
11:50 AM PST: Bunny CDN mitigates
11:54 AM PST: Grandcosmetic notified of affected host by @taigrr
12:19 PM PST: salt-store uploaded to Bitbucket (Credit Zachary for find)
12:48 PM PST: Serverio technologijos notified of affected host by @taigrr
12:48 PM PST: Dedic notified of affected host by @taigrr
12:51 PM PST: Atlassian notified of new repo by @taigrr
12:56 PM PST: Selectel notified of additional affected host by @taigrr
1:11 PM PST: Atlassian removes new repo; takes additional countermeasures
2:22 PM PST: neohost notified of affected host by @taigrr
2:27 PM PST: alibaba notified of affected host by @taigrr
7:32 PM PST: tzulo notified of affected host by @taigrr
7:40 PM PST: webnic.cc notified of affected host by @taigrr

Version 1 of the malware was not persistent after deleting /var/tmp/salt-store and /tmp/salt-minions
There have been 4 5 versions of the malware  so far, and the malware has auto-update capabilities.
It seems to check for updates regularly (perhaps as often as every 2 minutes).
Updating your salt-master won't prevent this, as the binary can update itself.
Malware Version Breakdown

Version 1 is not persistent. This does not mean all you have to do is delete the two files mentioned above. Note the list of side-effects mentioned below.

md5sum: 8ec3385e20d6d9a88bc95831783beaeb

Version 2: Very likely is not persistent

I suspect this payload patched the mining application to mine with multiple cores instead of just one. Unclear.
md5sum: defc9efd1b430aea797e573922fa49ae

Version 3 sloppily attempts persistence! Look at your crontabs!

md5sum: unknown

Version 4 is slightly more sophisticated. See the bottom of the post (script by kenny, and "Clearing out v4") for more information.

Also makes use of crontabs
md5sum: 2c5cbc18d1796fd64f377c43175e79a3

Version 5 is very sophisticated and includes a worm to attack machines not using salt as well as those that are. Will likely require a complete system wipe.

md5sum: 33140982ace71281c40d0dab0e9d69b8

Version 6 seems to be similar to version 5, but with some of the malware's bugs fixed.

md5sum: ccec8781e629304b387c17a72cb22ec2

The crontab job performed in versions 3 and 4 does not yet do much.
It looks like a placeholder for when the malware will do something nastier later.
But keep in mind, this script can (and likely will) be updated at any point and will run arbitrary shell scripts.
The crontab job in versions 3 and 4 now acts as a backdoor.
If you didn't clear out version 4 before May 4, 2020, at 1:00 PM PST, you may have been infected with version 5. SHUT DOWN ALL AFFECTED SYSTEMS ASAP!
Version 5 contains a worm, which attempts to replicate across every other host it can contact.
Hopefully, you have firewalls set up through your hosting provider, and not just on your host.
If you do, that will help limit the spread, "flatten the curve" so to speak.
The current behavior of the script inserts a public key as root into your system so a third party can log in remotely without salt.
It also finds every host on your network, and tries all private keys it can find against them in attempts to spread.
This just turned really ugly.
Md5sums of scripts and Payloads:


01f221de0585e96a5eca04c96d624a6d
099efe116c6140481ab364c1fb42f596
0f3c1bed6400a00eda99193a0c22917d
12475ed076e46c7578b7e2abe330d53e
2c5cbc18d1796fd64f377c43175e79a3
33140982ace71281c40d0dab0e9d69b8
352418d6fdf7911194605059697d9ec9
470c7a9811c580f3b3ad9176f07022f3
47f650f4e506ae0d17683f07e4b3ce72
4a9edeacb40d49b57a94f88cd384f882
4fd0273c950888bd6d67878b89ddf235
52d7311ec94b66a4933751f8013b8dbe
52f0f887b6f628db44ef4ffc39f2e5cc
55fa7cdaf916605f70a78e0b8a7f22aa
77169fc274b26751b324c6bf5f8f02f9
827eaf2bf470155950f570d8437db8d8
839cf1f82704bc7685308ce6e025c651
8ec3385e20d6d9a88bc95831783beaeb
96f6ed310405fc12feb7daf6cac1980c
a28ded80d7ab5c69d6ccde4602eef861
aa145dec736e85cb59ff2da83f4a734c
ac289bc68c08eb997f81293c16959046
ad26246ea1d6f44c3555aeb578eda231
b356144fe73ad516fb38ae46fb944593
b889bfa84be8bb87e333daa5cb965e7a
b90946fcf2b163ca534a6fcff74e3761
ccec8781e629304b387c17a72cb22ec2
d41d8cd98f00b204e9800998ecf8427e
d770672e4c03b2596163fa28c7ec9f99
dc5b2fa7f85aa0bbd226afe3b5fcaeea
defc9efd1b430aea797e573922fa49ae
e056e3de909f03b7c0fd8e32dbb4c968
e4788174b361508b0391e6507c64d904
e600632da9a710bba3c53c1dfdd7bac1
ee623c606c8f18591f5a1714c8f3f0c0
f208c67c47f913e08a3df71e0ea6b1f3
fa733d1857fd04d90e9b5cfdc3221d8f

Alternative malware

There are now many, many strains out in the wild.
I believe this is in part due to the various exploit POCs out there on the web.
Opinion time

Publishing POCs when you know people are still patching is really irresponsible.
Putting together the Web API on saltexploit.com only took two hours to put together.
I thought very carefully about the behavior and access methods to make sure it couldn't be used for evil.
If I put my POC code into a loop with a simple bash script, it would take a script-kiddie literally minutes to start deploying payloads across the web.
Please please please, next time people are caught unpatched with a level 10 CVE, don't go pursue clicks and or publish newsworthy code at the expense of others, it's very unethical.
At time of writing, over 350 developers have sought out help in the Slack channel (some via the bridged IRC) and any one of them would likely express similar thoughts.
Back to the details

Some payloads are installing themselves in parallel with the mining application (any version) and some actively take measures to uninstall the mining application so they themselves can fly under the radar.
Therefore, if you don't see any symptoms on your minions, take precautions anyway.
Some data we have seen exfiltrated includes, but is not limited to:

Entire home folders for all users
SSH keys
Environment variables
mining wallets
bash history files
remote desktop config files and cookies

(this means even you windows hosts could be broken into at a later date if you have credentials saved!)


bash profile scripts
all salt keys and execution histories
redis databases
shadow files
man pages for any in-house software
custom kernel modules
your /opt folder
lots of other things

What have I done to keep claiming v1 is not persistent?


Ran salt-store it in a container and viewed all modified files, no persistence mechanisms I could see there. (You can reproduce my tests with virus.zip, pinned in the Slack channel sidebar.
Checked file modification timestamps on the rootfs (ext4 fs)

I'm aware this is not a reliable indicator. You can cover your tracks with touch, but just I'm listing it here anyway.


diffed my /etc folder (I have etckeeper installed) and saw only my selinux config was changed
Checked installed systemd units
checked crontab entries
checked my kernel wasn't patched (easy on Archlinux!)
My infections (5 different unrelated salt-masters, and a ton of minions for each) were resolved over 36 hours 4 days ago with no recurrences.
Other users, including @jblac confirm they have not seen any recurrences.
Larry Cashdollar has been reviewing and tearing down the binaries, performing independent research as well. He has a blog post, which you may read here.

TO REITERATE:

Take additional precautions if you did not manage to kill and remove the salt-store(r) binary before 4:15AM PST, May 3rd.
I have not personally run the above tests on any versions past v1, as my issue was already resolved, though others I trust have.
Other developers including Justin Black and Larry Cashdollar have performed their own independent testing.
Again, note this is v1.
v5 and variants thereof are live as of writing.
Related Articles


CBR
CSO
Censys
ComputerWeekly
DataCenter Knowledge
Forbes
HelpNet
The Hacker News
US-Cert

We're here to help

You are not alone. Literally hundreds of developers are struggling with this right now.


Github Issue


Slack Channel Join Link (join the community and then visit #salt-store-miner-public)!


IRC Channel (bridged to the Slack channel).
Checkout #salt-store-miner-public on freenode!


Please use caution when patching your salt masters, but do it quickly!
Credits

saltexploit.com is hosted by Tai Groot and you can follow me on Twitter for significant updates to this page.
Please note, I am not currently affiliated with SaltStack in any way, and I am just volunteering to help out.
To paraphrase the MIT liscense:
THIS INFORMATION IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE INFORMATION OR THE USE OR OTHER DEALINGS IN THE INFORMATION.
I am trying to help, please don't hurt me for it :)
Solely because people have asked, here is a donation link.
A big thank you to Justin Black, Daniel, Avasc, Jedd, Zachary, and everyone else who provided code samples and trace information in the Slack channel for helping with early troubleshooting and testing!
Additionally, thank you to those a part of the official SaltStack team who were quick to respond to questions and requests for help.