postmortem of a hack

linode327459.md

[xxxxxxx] ToS Violation - Malicious Activity

We take the integrity of our network very seriously [...] appreciate your cooperation in investigating this activity [...] Please keep us updated.

address under your control appears to have attacked one of our customers as part of a coordinated DDoS botnet. [...] UDP packets with a one-byte payload [...] and captured by our router during the attack.

08:42:35.965313 IP (tos 0x0, ttl 53, id 0, offset 0, flags [DF], proto UDP (17), length 29) 66.228.62.237.51282 > 1.2.3.4.80: UDP, length 1
0x0000: 4500 001d 0000 4000 3511 07d7 42e4 3eed E.....@.5...B.>.
0x0010: 4296 7992 c852 0050 0009 c93f 3000 0000 B.y..R.P...?0...
0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
08:42:35.965316 IP (tos 0x0, ttl 53, id 0, offset 0, flags [DF], proto UDP (17), length 29) 66.228.62.237.51282 > 1.2.3.4.80: UDP, length 1
0x0000: 4500 001d 0000 4000 3511 07d7 42e4 3eed E.....@.5...B.>.
0x0010: 4296 7992 c852 0050 0009 c93f 3000 0000 B.y..R.P...?0...
0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
08:42:35.966231 IP (tos 0x0, ttl 53, id 0, offset 0, flags [DF], proto UDP (17), length 29) 66.228.62.237.51282 > 1.2.3.4.80: UDP, length 1
0x0000: 4500 001d 0000 4000 3511 07d7 42e4 3eed E.....@.5...B.>.
0x0010: 4296 7992 c852 0050 0009 c93f 3000 0000 B.y..R.P...?0...
0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
08:42:35.967151 IP (tos 0x0, ttl 53, id 0, offset 0, flags [DF], proto UDP (17), length 29) 66.228.62.237.51282 > 1.2.3.4.80: UDP, length 1
0x0000: 4500 001d 0000 4000 3511 07d7 42e4 3eed E.....@.5...B.>.
0x0010: 4296 7992 c852 0050 0009 c93f 3000 0000 B.y..R.P...?0...
0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
08:42:35.967181 IP (tos 0x0, ttl 53, id 0, offset 0, flags [DF], proto UDP (17), length 29) 66.228.62.237.51282 > 1.2.3.4.80: UDP, length 1
0x0000: 4500 001d 0000 4000 3511 07d7 42e4 3eed E.....@.5...B.>.
0x0010: 4296 7992 c852 0050 0009 c93f 3000 0000 B.y..R.P...?0...
0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............

Analysis (UDP flood)

• On November 30 I added a service account for testing ZNC
• Should have been a nologin

Nov 30 09:44:10 (none) useradd[6457]: new user: name=znc, UID=1000, GID=1000, home=/home/znc, shell=/bin/bash

• Hostile(s) were brute forcing SSH -- 39802 failed login attempts in < 40 days
• List of unique addresses from attack
• At the very most ~5 of these are me

103.16.46.66
103.23.244.22
106.187.36.78
106.187.54.43
107.20.246.150
108.59.9.10
109.73.147.253
113.105.144.52
113.106.201.38
113.108.211.131
113.140.31.230
113.162.154.21
113.162.157.137
114.80.217.238
114.80.246.132
114.80.246.154
114.80.246.159
114.80.246.178
114.80.246.194
114.80.246.203
115.239.195.208
116.228.164.3
116.254.206.83
117.21.127.215
119.120.92.204
119.147.141.88
119.84.63.30
120.150.60.109
120.151.38.43
121.12.173.173
122.141.243.215
123.30.238.220
124.162.54.171
124.193.109.6
125.211.218.228
125.64.130.131
14.63.196.196
150.162.37.3
151.11.201.3
162.216.0.117
163.27.94.130
163.43.48.227
165.138.88.108
174.136.15.97
174.142.68.106
176.34.48.61
177.11.212.40
177.137.128.6
180.153.154.83
180.166.105.76
183.129.197.227
183.129.249.106
183.129.249.98
184.106.189.106
188.165.173.230
188.212.103.108
188.27.72.85
192.170.153.234
195.199.219.105
198.211.49.244
199.71.214.66
199.83.94.85
2.228.35.18
200.51.45.162
201.244.16.146
202.103.218.67
202.153.43.22
202.166.199.74
206.210.124.37
208.84.220.147
209.255.116.35
209.67.232.86
210.118.190.194
210.13.73.30
210.209.88.124
211.141.113.237
211.141.34.111
211.162.119.163
211.202.2.135
211.215.19.231
211.95.76.242
212.126.29.113
216.69.177.62
216.99.159.114
217.153.86.163
218.188.16.150
218.68.3.136
218.94.106.247
219.139.105.144
219.232.240.12
221.11.64.118
221.12.12.3
222.135.144.90
222.175.114.132
222.175.114.134
222.175.114.75
222.187.220.228
222.189.239.124
222.189.239.133
222.189.239.138
222.189.239.14
222.189.239.70
222.189.239.72
222.189.239.75
222.189.239.83
222.189.239.9
222.41.52.88
222.43.96.226
223.4.27.247
223.4.55.49
32.65.224.34
32.65.240.27
37.247.103.107
37.54.23.193
37.55.59.124
42.117.7.53
42.120.4.116
42.51.145.13
46.137.105.194
46.182.30.46
46.201.175.195
46.4.20.7
5.135.183.46
50.112.117.253
54.243.38.84
58.177.198.73
58.215.133.47
58.215.133.51
58.215.16.142
58.215.240.108
58.250.71.43
58.64.156.46
59.124.124.9
60.10.203.18
60.173.10.252
60.174.38.31
60.191.45.248
60.234.45.178
61.147.103.138
61.147.103.161
61.147.107.102
61.147.107.109
61.147.107.86
61.147.113.182
61.147.113.93
61.147.116.13
61.147.116.24
61.147.116.5
61.147.116.51
61.147.116.57
61.147.116.62
61.147.119.106
61.147.74.223
61.155.153.145
61.160.212.66
61.160.213.78
61.160.251.139
61.160.251.141
61.34.216.219
61.55.191.148
62.225.71.18
62.39.125.217
64.251.15.167
67.228.76.210
70.104.144.30
70.34.231.6
71.207.229.18
71.9.152.134
75.126.32.162
76.76.116.30
77.68.62.168
82.211.31.178
82.221.102.182
87.230.77.26
88.150.229.252
88.190.63.53
91.121.50.112
91.223.25.95
94.100.83.54
94.178.110.206
94.23.27.179
94.242.255.60
95.132.233.193
95.132.244.64
95.132.29.66
95.132.57.239
95.132.84.64
95.211.241.93

• Less than 24 hours after the service account was added they hit it

Dec  1 00:26:14 (none) sshd[11433]: Accepted password for znc from 108.59.9.10 port 53226 ssh2
Dec  1 12:29:40 (none) sshd[14449]: Accepted password for znc from 85.214.26.10 port 40999 ssh2
Dec  1 21:45:58 (none) sshd[14758]: Accepted password for znc from 85.214.26.10 port 42535 ssh2
Dec 19 17:34:31 (none) sshd[13570]: Accepted password for znc from 213.181.206.157 port 50421 ssh2

• Short duration logins are most likely automations

znc      pts/1        office.salesguar Thu Dec 19 17:34 - 17:34  (00:00)
znc      pts/0        clipr.eu         Sun Dec  1 21:45 - 21:46  (00:00)
znc      pts/1        clipr.eu         Sun Dec  1 12:29 - 12:29  (00:00)

• Hostile / bot added update scripts to cron for znc

auth.log.2:Dec  8 04:47:01 (none) CRON[20384]: pam_unix(cron:session): session opened for user znc by (uid=0)
auth.log.2:Dec  8 04:47:01 (none) CRON[20384]: pam_unix(cron:session): session closed for user znc
auth.log.2:Dec  8 04:48:01 (none) CRON[20387]: pam_unix(cron:session): session opened for user znc by (uid=0)
auth.log.2:Dec  8 04:48:01 (none) CRON[20387]: pam_unix(cron:session): session closed for user znc
auth.log.2:Dec  8 04:49:01 (none) CRON[20390]: pam_unix(cron:session): session opened for user znc by (uid=0)
auth.log.2:Dec  8 04:49:01 (none) CRON[20390]: pam_unix(cron:session): session closed for user znc
auth.log.2:Dec  8 04:50:01 (none) CRON[20393]: pam_unix(cron:session): session opened for user znc by (uid=0)
auth.log.2:Dec  8 04:50:01 (none) CRON[20393]: pam_unix(cron:session): session closed for user znc

• ZMEU sig in crontab

chunk:crontabs dusty$ pwd
/Users/dusty/Desktop/linode/var/spool/cron/crontabs

chunk:crontabs dusty$ cat znc
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (zmeu.cron installed on Sun Dec  1 21:46:07 2013)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
* * * * * /var/tmp/.b/update >/dev/null 2>&1

• Perl script was masquerading as klogd
• lon1.stuartmacfarlane.co.uk may be command & control box
• I hit the C&C on 6667, was password protected

root@(none):/proc/14869# ps aux | grep znc

znc 14869 0.0 0.4 32768 4400 ? S Dec01 5:01 /sbin/klogd -c 1 -x -x

• Script likely exists on remote box
• Probably piped via curl

root@(none):/proc/14869# lsof -p14869

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
/sbin/klo 14869 znc cwd DIR 202,0 4096 2 /
/sbin/klo 14869 znc rtd DIR 202,0 4096 2 /
/sbin/klo 14869 znc txt REG 202,0 10456 7726 /usr/bin/perl
/sbin/klo 14869 znc mem REG 202,0 80712 14340 /lib/x86_64-linux-gnu/libresolv-2.13.so
/sbin/klo 14869 znc mem REG 202,0 22928 14223 /lib/x86_64-linux-gnu/libnss_dns-2.13.so
/sbin/klo 14869 znc mem REG 202,0 47616 14253 /lib/x86_64-linux-gnu/libnss_files-2.13.so
/sbin/klo 14869 znc mem REG 202,0 35256 693 /usr/lib/perl/5.14.2/auto/Socket/Socket.so
/sbin/klo 14869 znc mem REG 202,0 18704 700 /usr/lib/perl/5.14.2/auto/IO/IO.so
/sbin/klo 14869 znc mem REG 202,0 3014000 25791 /usr/lib/locale/locale-archive
/sbin/klo 14869 znc mem REG 202,0 35104 14370 /lib/x86_64-linux-gnu/libcrypt-2.13.so
/sbin/klo 14869 znc mem REG 202,0 1595408 14239 /lib/x86_64-linux-gnu/libc-2.13.so
/sbin/klo 14869 znc mem REG 202,0 131107 14267 /lib/x86_64-linux-gnu/libpthread-2.13.so
/sbin/klo 14869 znc mem REG 202,0 530736 14217 /lib/x86_64-linux-gnu/libm-2.13.so
/sbin/klo 14869 znc mem REG 202,0 14768 14205 /lib/x86_64-linux-gnu/libdl-2.13.so
/sbin/klo 14869 znc mem REG 202,0 1574680 652740 /usr/lib/libperl.so.5.14.2
/sbin/klo 14869 znc mem REG 202,0 136936 14260 /lib/x86_64-linux-gnu/ld-2.13.so
/sbin/klo 14869 znc 0u CHR 136,0 0t0 3 /dev/pts/0 (deleted)
/sbin/klo 14869 znc 1u CHR 136,0 0t0 3 /dev/pts/0 (deleted)
/sbin/klo 14869 znc 2u CHR 136,0 0t0 3 /dev/pts/0 (deleted)
/sbin/klo 14869 znc 3u IPv4 90513 0t0 TCP li319-237.members.linode.com:58068->lon1.stuartmacfarlane.co.uk:ircd (ESTABLISHED)

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp        0      0 66.228.62.237:58068     164.177.151.142:6667    ESTABLISHED 1000       90513       14869/klogd -c 1 -x

• Found two other similar attacks -- one dating back to January 2013 -- involving lon1.stuartmacfarlane.co.uk (possibly the C&C box)

http://pastebin.com/NWFCbKLB & http://pastebin.com/ZrtpsEZG

• No evidence they got root
• Imaged the disks for local analysis
• SERVER HAS BEEN WIPED
• Was unable to recover update or daemon script
• Taking steps to ensure other boxes are not similarly compromised

TL;DR

• Use public key authentication
• Or stronger passwords

chunk:~ dusty$ openssl rand -base64 12
Qw6eoJrSlyZBEvsE

• White lists
• Failed login notifications
• Rate limiting
• Detect and NULL route dictionary attacks

RE: http://www.youtube.com/watch?v=b2OYNMO_mNw

-- Dusty

Interesting stuff, thanks for sharing