May 5, 7:00PM PST:
If you are reading this for the first time now, you will now need to nuke and destroy any system paired (e.g. minions) to an exploitable salt master. You should also create a new salt master. I recommend creating a backup first for forensics, but do not plan on ever reusing these systems. Information on how to tell if you could be affected is available below.
Even if you didn't notice any unexpected symptoms, please: nuke and restart.
I cannot stress this enough: we have identified three five unrelated attacks so far which exfiltrate everything from databases, to bash history files, to ssh keys, to entire home folders.
Please snapshot your servers and start fresh.
Determine what could have been taken--if it's on your server, assume it was taken.
In fact, we have evidence to suggest some new attackers are actually including automation scripts inside their payloads to remove the mining exploit so their own exploit can fly under the radar. Just because your machines don't seem like they've been affected, does not mean they weren't affected.
Please don't dismiss this notice as paranoia. I have actually seen this attack play out, and attackers are using a wide net. Do the right thing.
For more information, take a look at "Alternative malware" under the Malware Version Breakdown section.
Everyone, I've wanted to be as open about this as possible and help as many people fix their machines ASAP! I believe we as a team have been for the most part, very successful. Many, many sysadmins have been able to salvage their data, if not recover completely from an otherwise benign mining operation. However, many different exploit POCs are surfacing across the web. New iterations of the malware are showing up, and I (and others) are somewhat certain they are following the Slack thread (and saltexploit) to disguise their malware as the original payloads, when in reality their goal is very different, and possibly far more harmful.
This means two things:
- I think we have hit the point where saltexploit's openness is now more harmful than it is helpful
- Promising "if you have this md5sum...you only have to do this!" can now be misleading as it's possible a new attacker can intentionally disguise his malware as the original
Please stop posting code samples in the main chat and on the Github issue. Please stop posting links to new payloads.
My DMs are open on Slack, and I will help take care of filing abuse reports etc. with the credibility and (mild) publicity I have already established. Atlassian, for example, is now responding to my requests immediately (and good on them!), as is BunnyCDN. At risk of soundling like a shill, I would like to reiterate my thanks to Atlassian and BunnyCDN in particular for their speedy responses and countermeasures.
Justin Black has been very helpful in diagnosing problems and testing the different versions. I would consider him trustworthy regarding this issue as well, and his DMs on Slack are also open. There are others, but I won't name them unless they tell me it's okay.
However, I believe we have "strains" rather than linear versions now, with new players on the field. The remediation advice available on saltexploit will remain such that sysadmins can use it to help keep systems alive long enough to investigate whether any data was exfiltrated, and to report new vectors of attack to me and/or Justin. The "fix" however will change--you must wipe and reset to be sure everything is gone. Please be very careful when using your old config files as well. You don't want it sneaking back into a new network.
I recommend you take this as an opportunity to assess whether you are happy with your current hosting provider(s) as well.
Again, I really believe in the open nature of collaboration and I really hope all here believe that, I've been extremely responsive and free-flowing with information, but there is now actual evidence that openness is no longer the right play. Additionally, please note that all the troubleshooting and advice here stems from the mining exploit. I cannot guarantee other, sneakier exploits were not first.
I wish you all good luck!
If you're here, IMMEDIATELY update your salt-master. And don't forget to restart it too! turn off your Salt Master!
You'll need to start it in recovery mode with a chroot, more than likely, and remove any files you need to start up a new instance.
If you need to keep the system connected to retrieve configuration files, firewalling it from the public internet and disconnecting minions may also be a great idea. is a must!
- Github Issue: saltstack/salt#57057
- Caused by exploit to CVE-2020-11651 and CVE-2020-11652, which when combined allow for Remote Code Execution (RCE) as root on minions and masters.
- salt minions are affected,
but the masters seem to have been left alone this time (unless the master is itself a minion.)and as of version 5, masters may be as well salt-minionsis a compiled xmrig binary.salt-storecontains a RAT, nspps, which continues to evolve and become more nasty- Atlassian confirms the newer revisions of the binary are a version of h2miner
- As a RAT,
salt-storeis more concerning thansalt-minions. More on that later - There have been at least 5 different versions of the
salt-storepayload, each more advanced than the last. - There are additional backdoors installed, and private keys are getting stolen right and left
- Seriously, change out your keys!
- very high CPU usage, sometimes only on 1 core, sometimes on all
- Fan spin! (Physical hardware only, of course.)
- Mysterious process named
salt-minionsis CPU intensive, stemming from/tmp/salt-minionsor/usr/bin/ - additional binary in
/var/tmp/or/usr/binnamedsalt-storeorsalt-storer - Firewalls disabled
- Most applications crashing or taken offline
The initial malware payloads act like a sledgehammer. They tears down anything remotely CPU-intensive so they can suck out all your compute resources for that sweet, sweet Monero. This includes, but is not limited to:
- confluence
- databases (i.e. redis)
- docker containers
- java applications
- other cryptomining software
- rabbitMQ
- webservers
- some assorted cloud platform services
We can confirm it often does the following:
- disables SELinux
- enables hugepages
- turns off all system firewalls (hosting provider firewalls won't be affected)
- disables AppArmor
- disables the nmi watchdog
- basically, it turns your machine into a dedicated mining rig
The malware may also take over your redis database and use it to replicate the attacker's data. Please check your databases' integrity!
The full shell script used as the attack vector is was available in the Slack channel.
I don't want to risk breaching GitHub's ToS by distributing it here, and I'm sure not going to throw it up on my site and distribute it for free.
However, for versions 1-43 the malware doesn't seem to do much else, provided you aren't running an unauthenticated Redis instance.
Note: version 4 did not originally ship with a backdoor that did anything but check for updates. However, we have reports the update checker did evolve after a couple hours to do more, such as running the version 5 worm mentioned below.
If you're here, chances are you're already compromised.
Note that this the initial malware is was Linux-specific but BSD users should still investigate their systems.
Windows users need fear not.
there are now so many strains it is very possible BSD and Windows hosts are affected as well.
We have heard reports the initial scripts didn't infect FreeBSD, but they did still manage to delete a lot of files.
Be on the lookout for missing files.
Again, this says nothing to the newer strains out there.
As the initial vector uses a shell script, many actions were run on your hosts, but it's doubtful it did much.
In case you're not sure if your salt-master could have been compromised, take a look at these tools:
Update: Saltstack has released an official tool for checking salt-masters for vulnerabilities. Hopefully, anyone who refrained from using either of the tools mentioned below due to trust issues will be willing to give the official one a try, as it is blessed by SaltStack.
https://www.saltstack.com/lp/cve-patch-verify/
Prior Tools
The following command will create a file named "/tmp/HACKED.txt" on a vulnerable salt master when provided its public IP address.
curl -X POST -F 'ip=your.saltmaster.ip.address' https://saltexploit.com/test
This tool is meant to be fast and painless, but you should never trust strangers on the internet, and using this API requires a certain level of trust in me as a developer. It was shared with @jscottsf and @jblac before it was announced for peer review and auditing. I can personally endorse this tool as the author, but be careful what tools you use. Telling 3rd-part sites and random strangers your compromised salt-master IP is a very, very bad idea.
There is an offline checker made available here, courtesy of @rossengeorgiev.
The best tried-and-true answer is: wipe and restore all your systems. However, that's not possible for everyone to do right now, so here is what you can do.
The first step is updating creating a new salt-master.
Install all updates for your distro, and then test to see if your new salt master is still vulnerable.
If it is, take a look at "Tips for Updating Your Master"
Next, determine which version of the RAT you have.
If you didn't remove all instances of the binary by May 3, 4:15 AM PST you likely have version 5 or a hybrid payload, as the RAT binary auto-updates.
However, please find salt-store in /var/tmp and match the md5sum against one of the four in the section titled "Version Breakdown"
All you need to do is kill all salt-store and salt-minions processes, then reboot.
This is outdated information.
An attacker can pretend their payload is salt-store but really run a second, sneakier payload in the background.
If you didn't catch it in time, meaning before May 3, at 4:15 AM PST, assume you were updated to the latest version that was available at the time you froze your minions.
Additionally, if you didn't catch the payload before May 5th, 7PM PST, assume exfil has happened.
It's also possible exfil happened before then, but that's the earliest it has been reported so far.
You will need to determine what else besides updating the master and deleting the binaries you need to fix for your own systems.
Please see a list of changes and side effects in the sections labeled "There are additional system side-effects" and "Big Red Flags" to determine what steps you need to take.
Many people are having trouble removing version 4 of the malware. It's much trickier than versions 1-3, and has better persistence mechanisms.
On each minion, you'll need to look for a very weirdly named process, it's usually got 5 characters long of a random string in the middle, a la NXiQS, and kill it.
Then remove everything else (salt-minions and salt-store) again.
That process is watching the other binaries and replacing them.
Also, delete /tmp/.ICEd-unix/ if you have it, then reboot your machine.
Please report in the slack channel if these steps do not work for you.
You will need to determine what else besides updating the master and deleting the binaries you need to fix for your own systems.
Please see a list of changes and side effects in the sections labeled "There are additional system side-effects" and "Big Red Flags" to determine what steps you need to take.
Note: Many people first found they had the malware installed at this iteration. This is the last version of the malware you can be reasonably certain did not exfiltrate your data (again, with a very notable exception detailed below). However, as with version 1, a new attacker may install version 4 to cover their own, sneakier tracks. If you did not completely remove the malware by May 4, at 2:57 PM PST, consider your system unrecoverable. At this point, version 5 came out, and version 4 began to install the worm as well. Do some work to find out what may have been taken and modified, and feel free to contact me on Slack if you think you found something new, but DO NOT try to continue using these systems.
No known steps are yet available for version 5 as it has not yet been investigated.
No remediation steps will be given.
If you suspect you have version 5 or some other variant, please shut down all potentially affected systems immediately.
The malware behaves as a worm, reaching out all other hosts it can find to infect everything in sight.
## Personally, I'm seeing this event as a nice big vaccine.
A little pinch at the beginning, but nothing really bad happened.
Yes it was bad.
It was embarrassing.
But would you rather they have taken customer data and/or your private SSH keys?
What if he had taken all minions offline, or destroyed and replaced system binaries?
So far, we have no evidence to suggest data exfiltration has occurred, with one significant exception, detailed below.
Many sysadmins were lucky. Many more were not. This quickly became very bad. The end goal may still be to mine crypto, but the methods have changed. Unsecured systems are being compromised right and left.
As per @Exordian, some official repos do not yet have the fix upstream for salt-master. Even after updating your whole system, you may not be running a patched version of the salt-master. It's important to verify you've updated to the latest release, which may be more recent than what's available latest. Look under "Additional Help" to find some verification tools.
You may want to look into using SaltStack's repos instead of your distro's to get the patch sooner.
Fedora Users: This link may be helpful to you.
Note you must restart salt-master for changes to take effect after updating.
No software patches are required minion-side~, but you still need to rotate keys.~ No longer relevant as advice is now to wipe and recreate any and all minions which were affected.)
In order to run unauthenticated Salt commands, you must first expose the Salt Master's Key. This means it's possible (and very likely) the attacker may have a large database of IP/Key pairings. ROTATE YOUR KEYS!!! It's possible to continue to send commands against an updated salt-master if you already have salt's private key (because it actually would be "authenticated" by the stolen key.)
Take a look at this rekey repo (credit to @dwoz) and seriously consider rekeying your master and minions. Note that you may lose access to rekeyed minions not connected to the master during the rekey event. Please plan accordingly.
Note: timeline from March 16-April 30 sourced from F-Secure and is abridged.
March 16, 2020: F-Secure discloses vulnerability
March 24, 2020: SaltStack confirms issue, begins patching
April 23, 2020: SaltStack announces vulnerability, cautions users
April 29, 2020: SaltStack publishes patches to official SaltStack repos
April 30, 2020: F-Secure publishes additional advisory
Saturday, May 2, 2020:
- ~6:18 PM PST:
salt-store(version 1/v1) payload is uploaded to Selectel CDN, attack begins - 6:30 PM PST: payload is uploaded to Bitbucket
- ~6:45 PM PST: Github issue opened by @xiaopanggege , many other devs with monitoring tools follow suit
- ~7:00 PM PST: @aidanstevens29 correctly identifies this as an exploit of CVE-2020-11651 and CVE-2020-11652
- Initial testing and debugging occurs. Several developers in the GH issue claim complete malware removal.
- Midnight: Atlassian is notified by @taigrr of the malicious usage. Highest-level support ticket opened.
Sunday, May 3, 2020:
- 4:15 AM PST: salt-store v2 uploaded to Bitbucket
- 6:07 AM PST: salt-store v3 uploaded to Bitbucket
- 6:11 AM PST: salt-store v4 uploaded to Bitbucket
- 11:12 AM PST: Initial Slack Channel created to collaborate and discuss troubleshooting (35 developers joined)
- 12:39 PM PST: New, public Slack Channel created, 400+ developers joined and rising (many through the bridged IRC channel)
- 4:43 PM PST: Atlassian removes
salt-storev4 from Bitbucket; script now falls back to Selectel
Monday, May 4, 2020:
- 1:00 AM PST: @taigrr sounds the alarm that patched, official Docker images for Saltstack have not been built or released
- 2:35 AM PST: New docker images available (thanks S0undt3ch!)
- ~1:00 AM PST: Another sample exploit released
- ~11:00 AM PST: @itskenny0 posted a cleanup script to assist recovering affected minions
- 2:30 PM PST: First report of
dropper.sh, a malicious worm designed to steal SSH keys and add another backdoor - 2:57 PM PST: salt-store v5 uploaded to Bitbucket
- 4:50 PM PST: New Bitbucket Repo detected by @jscottsf -- now defunct
- 4:51 PM PST: New repo reported to Atlassian by @taigrr
- 4:57 PM PST: Atlassian removes
salt-storev5 from Bitbucket; falls back to a VPS - 5:00 PM PST: More POCs surface on GitHub (seriously, when will people learn?)
- 5:13 PM PST: Abuse reported to UAServers for malware distribution
- 5:15 PM PST: @jblac begins initial testing of v5
Tuesday, May 5, 2020:
- ~9:03 AM PST: Selectel alerted to affected host by Ivan
- ~9:20 AM PST: Alias Landergate, and @taigrr sent abuse reports to SelectelCDN, BunnyCDN
- 9:22 AM PST: No response from UAServers, but affected host removed
- 9:22 AM PST: New host reported to UAServers by @taigrr
- 9:32 AM PST: Team voice notified of affected host
- 9:53 AM PST: New host reported to UAServers by @taigrr
- 11:50 AM PST: Bunny CDN mitigates
- 11:54 AM PST: Grandcosmetic notified of affected host by @taigrr
- 12:19 PM PST: salt-store uploaded to Bitbucket (Credit Zachary for find)
- 12:48 PM PST: Serverio technologijos notified of affected host by @taigrr
- 12:48 PM PST: Dedic notified of affected host by @taigrr
- 12:51 PM PST: Atlassian notified of new repo by @taigrr
- 12:56 PM PST: Selectel notified of additional affected host by @taigrr
- 1:11 PM PST: Atlassian removes new repo; takes additional countermeasures
- 2:22 PM PST: neohost notified of affected host by @taigrr
- 2:27 PM PST: alibaba notified of affected host by @taigrr
- 7:32 PM PST: tzulo notified of affected host by @taigrr
- 7:40 PM PST: webnic.cc notified of affected host by @taigrr
Version 1 of the malware was not persistent after deleting /var/tmp/salt-store and /tmp/salt-minions
There have been 4 5 versions of the malware so far, and the malware has auto-update capabilities.
It seems to check for updates regularly (perhaps as often as every 2 minutes).
Updating your salt-master won't prevent this, as the binary can update itself.
Version 1 is not persistent. This does not mean all you have to do is delete the two files mentioned above. Note the list of side-effects mentioned below.
- md5sum:
8ec3385e20d6d9a88bc95831783beaeb
Version 2: Very likely is not persistent
- I suspect this payload patched the mining application to mine with multiple cores instead of just one. Unclear.
- md5sum:
defc9efd1b430aea797e573922fa49ae
Version 3 sloppily attempts persistence! Look at your crontabs!
- md5sum: unknown
Version 4 is slightly more sophisticated. See the bottom of the post (script by kenny, and "Clearing out v4") for more information.
- Also makes use of crontabs
- md5sum:
2c5cbc18d1796fd64f377c43175e79a3
Version 5 is very sophisticated and includes a worm to attack machines not using salt as well as those that are. Will likely require a complete system wipe.
- md5sum:
33140982ace71281c40d0dab0e9d69b8
Version 6 seems to be similar to version 5, but with some of the malware's bugs fixed.
- md5sum:
ccec8781e629304b387c17a72cb22ec2
The crontab job performed in versions 3 and 4 does not yet do much.
It looks like a placeholder for when the malware will do something nastier later.
But keep in mind, this script can (and likely will) be updated at any point and will run arbitrary shell scripts.
The crontab job in versions 3 and 4 now acts as a backdoor. If you didn't clear out version 4 before May 4, 2020, at 1:00 PM PST, you may have been infected with version 5. SHUT DOWN ALL AFFECTED SYSTEMS ASAP! Version 5 contains a worm, which attempts to replicate across every other host it can contact. Hopefully, you have firewalls set up through your hosting provider, and not just on your host. If you do, that will help limit the spread, "flatten the curve" so to speak. The current behavior of the script inserts a public key as root into your system so a third party can log in remotely without salt. It also finds every host on your network, and tries all private keys it can find against them in attempts to spread. This just turned really ugly.
01f221de0585e96a5eca04c96d624a6d099efe116c6140481ab364c1fb42f5960f3c1bed6400a00eda99193a0c22917d12475ed076e46c7578b7e2abe330d53e2c5cbc18d1796fd64f377c43175e79a333140982ace71281c40d0dab0e9d69b8352418d6fdf7911194605059697d9ec9470c7a9811c580f3b3ad9176f07022f347f650f4e506ae0d17683f07e4b3ce724a9edeacb40d49b57a94f88cd384f8824fd0273c950888bd6d67878b89ddf23552d7311ec94b66a4933751f8013b8dbe52f0f887b6f628db44ef4ffc39f2e5cc55fa7cdaf916605f70a78e0b8a7f22aa77169fc274b26751b324c6bf5f8f02f9827eaf2bf470155950f570d8437db8d8839cf1f82704bc7685308ce6e025c6518ec3385e20d6d9a88bc95831783beaeb96f6ed310405fc12feb7daf6cac1980ca28ded80d7ab5c69d6ccde4602eef861aa145dec736e85cb59ff2da83f4a734cac289bc68c08eb997f81293c16959046ad26246ea1d6f44c3555aeb578eda231b356144fe73ad516fb38ae46fb944593b889bfa84be8bb87e333daa5cb965e7ab90946fcf2b163ca534a6fcff74e3761ccec8781e629304b387c17a72cb22ec2d41d8cd98f00b204e9800998ecf8427ed770672e4c03b2596163fa28c7ec9f99dc5b2fa7f85aa0bbd226afe3b5fcaeeadefc9efd1b430aea797e573922fa49aee056e3de909f03b7c0fd8e32dbb4c968e4788174b361508b0391e6507c64d904e600632da9a710bba3c53c1dfdd7bac1ee623c606c8f18591f5a1714c8f3f0c0f208c67c47f913e08a3df71e0ea6b1f3fa733d1857fd04d90e9b5cfdc3221d8f
There are now many, many strains out in the wild. I believe this is in part due to the various exploit POCs out there on the web.
Publishing POCs when you know people are still patching is really irresponsible. Putting together the Web API on saltexploit.com only took two hours to put together. I thought very carefully about the behavior and access methods to make sure it couldn't be used for evil. If I put my POC code into a loop with a simple bash script, it would take a script-kiddie literally minutes to start deploying payloads across the web.
Please please please, next time people are caught unpatched with a level 10 CVE, don't go pursue clicks and or publish newsworthy code at the expense of others, it's very unethical.
At time of writing, over 350 developers have sought out help in the Slack channel (some via the bridged IRC) and any one of them would likely express similar thoughts.
Some payloads are installing themselves in parallel with the mining application (any version) and some actively take measures to uninstall the mining application so they themselves can fly under the radar. Therefore, if you don't see any symptoms on your minions, take precautions anyway.
Some data we have seen exfiltrated includes, but is not limited to:
- Entire home folders for all users
- SSH keys
- Environment variables
- mining wallets
- bash history files
- remote desktop config files and cookies
- (this means even you windows hosts could be broken into at a later date if you have credentials saved!)
- bash profile scripts
- all salt keys and execution histories
- redis databases
- shadow files
- man pages for any in-house software
- custom kernel modules
- your /opt folder
- lots of other things
- Ran
salt-storeit in a container and viewed all modified files, no persistence mechanisms I could see there. (You can reproduce my tests withvirus.zip, pinned in the Slack channel sidebar. - Checked file modification timestamps on the rootfs (ext4 fs)
- I'm aware this is not a reliable indicator. You can cover your tracks with touch, but just I'm listing it here anyway.
- diffed my
/etcfolder (I have etckeeper installed) and saw only my selinux config was changed - Checked installed systemd units
- checked crontab entries
- checked my kernel wasn't patched (easy on Archlinux!)
- My infections (5 different unrelated salt-masters, and a ton of minions for each) were resolved over
36 hours4 days ago with no recurrences. - Other users, including @jblac confirm they have not seen any recurrences. Larry Cashdollar has been reviewing and tearing down the binaries, performing independent research as well. He has a blog post, which you may read here.
Take additional precautions if you did not manage to kill and remove the salt-store(r) binary before 4:15AM PST, May 3rd. I have not personally run the above tests on any versions past v1, as my issue was already resolved, though others I trust have. Other developers including Justin Black and Larry Cashdollar have performed their own independent testing.
Again, note this is v1. v5 and variants thereof are live as of writing.
You are not alone. Literally hundreds of developers are struggling with this right now.
-
Slack Channel Join Link (join the community and then visit
#salt-store-miner-public)! -
IRC Channel (bridged to the Slack channel). Checkout #salt-store-miner-public on freenode!
Please use caution when patching your salt masters, but do it quickly!
saltexploit.com is hosted by Tai Groot and you can follow me on Twitter for significant updates to this page.
Please note, I am not currently affiliated with SaltStack in any way, and I am just volunteering to help out.
To paraphrase the MIT liscense:
THIS INFORMATION IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE INFORMATION OR THE USE OR OTHER DEALINGS IN THE INFORMATION.
I am trying to help, please don't hurt me for it :)
Solely because people have asked, here is a donation link.
A big thank you to Justin Black, Daniel, Avasc, Jedd, Zachary, and everyone else who provided code samples and trace information in the Slack channel for helping with early troubleshooting and testing!
Additionally, thank you to those a part of the official SaltStack team who were quick to respond to questions and requests for help.
[rekey] That's a really round-about way of doing: