Skip to content

Instantly share code, notes, and snippets.

@jatrost

jatrost/troubleshooting-mhn.md

Last active August 29, 2015 14:03
Show Gist options
  • Save jatrost/99e36bdb1e952a098716 to your computer and use it in GitHub Desktop.
Save jatrost/99e36bdb1e952a098716 to your computer and use it in GitHub Desktop.
Download ZIP
This is a Work in Progress troubleshooting guide to MHN, compiled from responses to questions to the MHN alias (mhn@threatstream.com).
Raw
troubleshooting-mhn.md

MHN Troubleshooting

Note: This will likely be taken down soon since it has been posted on the MHN wiki here: https://github.com/threatstream/mhn/wiki/MHN-Troubleshooting-Guide

This is a Work in Progress troubleshooting guide to MHN, compiled from responses to questions to the MHN alias (mhn@threatstream.com).

I can't login to the MHN server web app

Can you describe the error you're encountering when logging in? Password failure? Connection failure?

If password failure, can you try this (you should see similar output to mine below, if not, it indicates a problem). The select statement is useful to verify that the user you expect to exist does in fact exist.

# cd $MHN_HOME/server/

# sqlite3 mhn.db
SQLite version 3.7.9 2011-11-01 00:52:41
Enter ".help" for instructions
Enter SQL statements terminated with a ";"

sqlite> select * from user;
1|USERNAME@SITE.com|sklfdjhkasdlfhklsadhfklasdhfkldsahklsd|1|

sqlite> .quit

# cd $MHN_HOME/
# . env/bin/activate
# cd server
# python manual_password_reset.py

Walk through the password reset and make sure you see user found, updating password at the end.

Troubleshooting the Honeypot side

  • I don't see events flowing back to my MHN server even though I am scanning/wget'ing my sensor

Can you run these commands from your honeypot system and provide the output?

Before running these commands start a script to wget against your honeypot.

$ sudo netstat -luntp
$ sudo supervisorctl status
$ sudo iptables -L
$ sudo tcpdump -nnNN tcp port 10000

I am expecting output like this:

$ sudo netstat -luntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      561/sshd        
tcp6       0      0 :::21                   :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::22                   :::*                    LISTEN      561/sshd        
tcp6       0      0 :::1433                 :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::443                  :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::445                  :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::5060                 :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::5061                 :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::135                  :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::3306                 :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::42                   :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::80                   :::*                    LISTEN      3763/dionaea    
udp        0      0 0.0.0.0:68              0.0.0.0:*                           464/dhclient3   
udp        0      0 0.0.0.0:40077           0.0.0.0:*                           3763/dionaea    
udp6       0      0 :::5060                 :::*                                3763/dionaea    
udp6       0      0 :::69                   :::*                                3763/dionaea    

 
$ sudo supervisorctl status
dionaea                          RUNNING    pid 3763, uptime 3 days, 22:31:19


$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

$ sudo tcpdump -nnNN tcp port 10000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:46:14.009646 IP 1.2.3.4.42873 > 5.6.6.8.10000: Flags [P.], seq 1180349317:1180349611, ack 2474834734, win 913, options [nop,nop,TS val 85084174 ecr 169636000], length 294
16:46:14.012967 IP 5.6.6.8.10000 > 1.2.3.4.42873: Flags [.], ack 294, win 162, options [nop,nop,TS val 169661444 ecr 85084174], length 0

Dionaea Honeypot troubleshooting (same symptoms)

Can you log into your dionaea box and look at its config (/etc/dionaea/dionaea.conf)? Scroll to the hpfeeds section.
Your should see something like this:

hpfeeds = {
        hp1 = {
                server = "1.2.3.4"
                port = "10000"
                ident = "XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX"
                secret = "XXXXXXXXXXXXXXXX"
                // dynip_resolve: enable to lookup the sensor ip through a webservice
                dynip_resolve = "http://queryip.net/ip/"
        }

Is your "server" variable correct? It should be the IP of your MHN server. If so, can you try this (from your dionaea box)?

nc -vvv <YOUR-SERVER-IP> 10000

Do you see something like this? Or does it just hang?

$ nc -vvv <YOUR-SERVER-IP> 10000
Connection to <YOUR-SERVER-IP> 10000 port [tcp/webmin] succeeded!
@hp2?Z

If it hangs either your hpfeeds service isn't running on your server or there may be a firewall blocking it or there may be another network connectivity issue.

Troubleshooting from the MHN Server

  • I don't see events flowing back to my MHN server even though I am scanning/wget'ing my sensor

On your MHN server box, as root run these commands. You should get similar out to mine. If your output differs, this could be the problem.

# netstat -luntp | grep 10000
tcp        0      0 0.0.0.0:10000           0.0.0.0:*
LISTEN      980/python

# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

# cd /var/log
# tail -f hpfeeds-broker.err
DEBUG:root:broker publish to dionaea.connections by ('XXX.XXX.XXX.XXX', 49187)
DEBUG:root:broker publish to geoloc.events by ('127.0.0.1', 42282)
DEBUG:root:broker publish to dionaea.connections by ('XXX.XXX.XXX.XXX', 39176)
DEBUG:root:broker publish to geoloc.events by ('127.0.0.1', 42282)
DEBUG:root:broker publish to dionaea.connections by ('XXX.XXX.XXX.XXX', 42873)
DEBUG:root:broker publish to geoloc.events by ('127.0.0.1', 42282)
DEBUG:root:broker publish to dionaea.connections by ('XXX.XXX.XXX.XXX', 42873)
DEBUG:root:broker publish to geoloc.events by ('127.0.0.1', 42282)
DEBUG:root:broker publish to dionaea.connections by ('XXX.XXX.XXX.XXX', 46710)
DEBUG:root:broker publish to geoloc.events by ('127.0.0.1', 42282)

# mongo hpfeeds
MongoDB shell version: 2.6.2
connecting to: hpfeeds
> db.auth_key.find()
{ "_id" : ObjectId("YYYYYYYYYYYYYYYYYY"), "identifier" : "mnemosyne",
"subscribe" : [ "conpot.events", "thug.events", "beeswarm.hive",
"dionaea.capture", "dionaea.connections", "thug.files",
"beeswarn.feeder", "cuckoo.analysis", "kippo.sessions",
"glastopf.events", "glastopf.files", "mwbinary.dionaea.sensorunique",
"snort.alerts" ], "secret" : "XXXXXXXXXXXXXX", "publish" : [ ] }
{ "_id" : ObjectId("YYYYYYYYYYYYYYYYYY"), "identifier" : "honeymap",
"subscribe" : [ "geoloc.events" ], "secret" : "XXXXXXXXXXXXXX",
"publish" : [ ] }
{ "_id" : ObjectId("YYYYYYYYYYYYYYYYYY"), "identifier" : "geoloc",
"subscribe" : [ "dionaea.connections", "dionaea.capture",
"glastopf.events", "beeswarm.hive", "kippo.sessions", "conpot.events",
"snort.alerts" ], "secret" : "XXXXXXXXXXXXXX", "publish" : [
"geoloc.events" ] }
{ "_id" : ObjectId("YYYYYYYYYYYYYYYYYY"), "subscribe" : [ ], "secret"
: "XXXXXXXXXXXXXX", "identifier" :
"XXXXXX-XXXXXXXX-XXXXXXXX-XXXXXXXX-XXXXXXXX", "publish" : [
"mwbinary.dionaea.sensorunique", "dionaea.capture",
"dionaea.capture.anon", "dionaea.caputres", "dionaea.connections" ] }
Type "it" for more
>

The last entry is the entry for one of our honeypots. The identifier and secret need to match what is in your honeypot's /etc/dionaea/dionaea.conf config under the hpfeeds section. If these don't match your honeypot will not be able to talk with the server.

Running the honeypot on a private network

Because your honeypot is on a private network, you will almost certainly not see any events show up on the Honeymap since it only renders events that can be IP Geo located and private IPs do not get geolocated by Maxmind.

Can you confirm that you see no events on http://YOUR-SERVER/ui/attacks/ and no counts on the base page? If so, can you run this command? You should have output similar to mine:

# mongo mnemosyne
MongoDB shell version: 2.6.2
connecting to: mnemosyne
> db.getCollectionNames()
[
"daily_stats",
"dork",
"file",
"hpfeed",
"session",
"system.indexes",
"url"
]
> db.session.find()
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" :
"SipCall", "hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"),
"timestamp" : ISODate("2014-05-20T20:12:14.028Z"), "source_ip" :
"192.198.107.186", "source_port" : 5070, "destination_port" : 5060,
"identifier" : "XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" :
"dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" :
"SipSession", "hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"),
"timestamp" : ISODate("2014-06-21T12:25:29.493Z"), "source_ip" :
"209.126.99.96", "source_port" : 5086, "destination_port" : 5060,
"identifier" : "XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" :
"dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T12:25:32.782Z"), "source_ip" : "107.22.35.136",
"source_port" : 55261, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:48.040Z"), "source_ip" : "107.22.35.136",
"source_port" : 52915, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "smbd",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:47.137Z"), "source_ip" : "178.252.183.100",
"source_port" : 3955, "destination_port" : 445, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "httpd",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:51.478Z"), "source_ip" : "115.79.59.61",
"source_port" : 3453, "destination_port" : 80, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "httpd",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:50.476Z"), "source_ip" : "115.79.59.61",
"source_port" : 3454, "destination_port" : 80, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:53.105Z"), "source_ip" : "107.22.35.136",
"source_port" : 52917, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:52.274Z"), "source_ip" : "93.174.93.218",
"source_port" : 52569, "destination_port" : 21320, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "smbd",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:59.602Z"), "source_ip" : "178.252.183.100",
"source_port" : 4053, "destination_port" : 445, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:59.507Z"), "source_ip" : "181.193.66.61",
"source_port" : 43670, "destination_port" : 23, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" :
"SipSession", "hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"),
"timestamp" : ISODate("2014-06-21T13:05:58.327Z"), "source_ip" :
"209.126.99.96", "source_port" : 5068, "destination_port" : 5060,
"identifier" : "XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" :
"dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:58.169Z"), "source_ip" : "107.22.35.136",
"source_port" : 52919, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:04.196Z"), "source_ip" : "27.192.16.128",
"source_port" : 48078, "destination_port" : 4899, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:03.233Z"), "source_ip" : "107.22.35.136",
"source_port" : 52921, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "smbd",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:03.061Z"), "source_ip" : "178.252.183.100",
"source_port" : 4077, "destination_port" : 445, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" :
"SipSession", "hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"),
"timestamp" : ISODate("2014-06-21T13:06:01.699Z"), "source_ip" :
"176.58.113.94", "source_port" : 6072, "destination_port" : 5060,
"identifier" : "XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" :
"dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:05.730Z"), "source_ip" : "27.192.16.128",
"source_port" : 48078, "destination_port" : 4899, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:04.967Z"), "source_ip" : "27.192.16.128",
"source_port" : 48078, "destination_port" : 4899, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:08.296Z"), "source_ip" : "107.22.35.136",
"source_port" : 52923, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment