- What are we doing here?
- What is PSAD?
- What is Logstash?
- Using PSAD and Logstash
- First, configure rsyslog
- Then, configure logstash patterns
- Available log parameters
- Next up: logstash input
- Are you using logstash-forwarder?
- On to: logstash filter
- Roundup
- Links
psad is a collection of three lightweight system daemons (two main daemons and one helper daemon) that run on Linux machines and analyze iptables log messages to detect port scans and other suspicious traffic. A typical deployment is to run psad on the iptables firewall where it has the fastest access to log data.
logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.
Using Logstash, you can process and tag PSAD events (logs) based on their significance. Also, as an alternative to using the PSAD email alerts, you could use one of Logstash' many output filters to send notifications of important events. For example, you could use Nagios to notify you of any port scans PSAD has detected. As the number of servers you install PSAD on grows, receiving email alerts from all of them can become cumbersome and could become a 'boy who cried wolf' situation. Logstash in combination with Kibana will let you chart all port scans and attempted attacks that PSAD can discover in a unified way.
This gist will explain how to configure both PSAD and Logstash to play nice together. This gist is focused on PSAD 2.2.4, Logstash 1.4.2 and Ubuntu Server 14.04; if you're using a different OS or different versions of Logstash/PSAD you may have to tweak the instructions a bit.
This gist assumes you have PSAD and Logstash up and running individually.
The first thing we need to do is configure rsyslog on Ubuntu to write any PSAD events to a separate logfile. Although it is perfectly possible to parse the /var/log/syslog directly for any PSAD messages, I prefer to write PSAD events to its own logfile; it makes it easier to determine its actions and makes our logstash configuration simpler as well, as we don't have to filter out any other kernel messages.
To do this, we need to tell rsyslog to send all messages that originate from PSAD to a different logfile. Create a file called /etc/rsyslog.d/30-psad.conf. The 30 is added to make sure the settings in this configuration file are parsed before the default settings, which start with 50-.
Add the following contents to the file:
if $programname == 'psad' then /var/log/psad.log # redirect psad log line to separate file
& stop # prevent log lines from application psad to be prcessed by any other filtersThe first line ensures that any log lines from the application psad are sent to the file /var/log/psad.log. The second line ensures that any lines matching the previous criteria will not be processed by any other filters. This second line is needed to prevent the log lines from also showing up in your /var/log/messages or /var/log/kern.log file.
Note: with older versions of rsyslog, you may have to replace the word stop on the second line with the ~ character; on newer version of Ubuntu the ~ character as a stop indicator has been deprecated and will cause warning messages when (re)starting the rsyslog service.
Then, restart the rsyslog service to enable the changes:
sudo service rsyslog restartTo make processing PSAD messages easier, I have created a set of GROK patterns which should allow you to detect and act upon the various PSAD messages without too much effort.
First, you need to create a logstash patterns file. Depending on your logstash configuration, the location of these pattern files may vary but I store mine in /etc/logstash/patterns/. Here are the patterns:
# psad grok patterns
PSAD_PORT_RANGE (?:(%{DATA:[psad][start_port]}\-%{DATA:[psad][end_port]})|%{DATA:[psad][scan_port]})
PSAD_SCAN_DETECTED %{SYSLOGBASE} scan detected(:? \(%{DATA:[psad][scan_type]}\))?: %{IPORHOST:src_ip} -> %{IPORHOST:dst_ip} %{WORD:[psad][proto]}: \[%{PSAD_PORT_RANGE}\](:? flags: %{DATA:[psad][flags]})? %{WORD} pkts: %{NUMBER:[psad][proto_num_pkts]} DL: %{NUMBER:[psad][danger_level]}%{GREEDYDATA:message}
PSAD_SIGNATURE_MATCH %{SYSLOGBASE} src: %{IPORHOST:src_ip} signature match: \"%{DATA:[psad][signature]}\" \(sid: %{NUMBER:[psad][signature_id]}\) %{WORD:[psad][proto]} port: %{NUMBER:[psad][scan_port]}
PSAD_AUTO_BLOCK %{SYSLOGBASE} added iptables auto\-block against ${IPORHOST:src_ip} for %{NUMBER:[psad][block_time]} seconds
PSAD_AUTO_UNBLOCK %{SYSLOGBASE} removed iptables auto\-block against ${IPORHOST:src_ip}
PSAD_NOTICE %{SYSLOGBASE} (?=%{GREEDYDATA:message})(?:flushing|imported|received|domain)
PSAD_WARNING %{SYSLOGBASE} %{LOGLEVEL:[psad][log_level]}: (?=%{GREEDYDATA:message})
Let me briefly explain what each pattern does:
This pattern is only used by other patterns to prevent having to duplicate the port range part of the pattern Fields: start_port, end_port, scan_port
Used for port scan detections Fields: scan_type, src_ip, dst_ip, proto, flags, proto_num_pkts, danger_level, message
Used for psad/fwsnort signature matches Fields: src_ip, signature, signature_id, proto, scan_port
Used to detect auto-block actions Fields: src_ip, block_time
Used to detect auto-unblock actions Fields: src_ip
Used to detect PSAD notice messages; usually generated by PSAD upon starting the service or updating the signatures Fields: message
Used to detect PSAD warning messages Fields: log_level, message
Here's a list of all paramters which are available for use in Logstash (or elasticsearch, if that's what you're using to store your logs; these are also the fields you can use to create your dashboards in Kibana).
Start and end port of a port scan range
Port scanned or found in signature match
Indicates the type of scan, if any
Protocol used in scan (TCP/UDP)
Any attack flags found (SYN, etc)
The number of packets received in the attack/scan
The PSAD danger lever assigned to the attack/scan
The psad/fwsnort signature detected
The psad/fwsnort signature ID detected
The time an IP address was auto-blocked
The log level of a PSAD warning message
A generic message for the logline; varies per type of log line
The source IP of the attack/scan
The destination IP of the attack/scan
Note: Why not place the src_ip and dst_ip in the [psad] array as well? Since I use logstash to process almost all my logfiles, I prefer to keep things consistent. This way I can create a separate GEOIP filter to process all fields called src_ip and dst_ip and I don't have to create a separate filter for each logfile.
The logstash input filter will assure that our psad log is parsed by logstash. Here's what it looks like:
input {
file {
path => '/var/log/psad.log' # the location of our psad logfile
type => 'psad' # assign a type so we can easily process this in the filters
}
}No problem, use this instead in your config.json file:
{
"files": [
{
"paths": [ "/var/log/psad.log" ],
"fields": { "type": "psad" }
}
]
}Now that we've made sure logstash has access to our PSAD logfile, we need to tell it what to do with each line. I've placed comments in the code to guide you through each part.
filter {
# only process lines which are of type 'psad'; this is set in the input section
if [type] == "psad" {
# parse for detected scans
grok {
patterns_dir => "/etc/logstash/patterns/" # specify the folder where logstash can find your patterns file
match => [ "message", "%{PSAD_SCAN_DETECTED}" ]
add_tag => [ "psad", "scan-detected" ] # add tags so we can determine what to do with each type of line
tag_on_failure => [] # do not add tags on failure just yet, first check the other line types
}
# parse for signature match
if "psad" not in [tags] {
grok {
patterns_dir => "/etc/logstash/patterns/"
match => [ "message", "%{PSAD_SIGNATURE_MATCH}" ]
add_tag => [ "psad", "signature-match" ]
tag_on_failure => []
}
}
# parse for auto block
if "psad" not in [tags] {
grok {
patterns_dir => "/etc/logstash/patterns/"
match => [ "message", "%{PSAD_AUTO_BLOCK}" ]
add_tag => [ "psad", "iptables-auto-block" ]
tag_on_failure => []
}
}
# parse for auto unblock
if "psad" not in [tags] {
grok {
patterns_dir => "/etc/logstash/patterns/"
match => [ "message", "%{PSAD_AUTO_UNBLOCK}" ]
add_tag => [ "psad", "iptables-auto-unblock" ]
tag_on_failure => []
}
}
# parse for psad warning
if "psad" not in [tags] {
grok {
patterns_dir => "/etc/logstash/patterns/"
match => [ "message", "%{PSAD_WARNING}" ]
add_tag => [ "psad", "warning" ]
tag_on_failure => []
}
}
# parse for psad notice
if "psad" not in [tags] {
grok {
patterns_dir => "/etc/logstash/patterns/"
match => [ "message", "%{PSAD_NOTICE}" ]
add_tag => [ "psad", "service-notice" ]
tag_on_failure => []
}
}
# check for unmatched loglines
# if you find these often, I may have missed a line type and we will need to add a
# pattern for it; please send me any such lines so I can create the grok pattern
if "psad" not in [tags] {
mutate {
add_tag => [ "psad", "unmatched-logline" ]
}
}
}
}Now you've configured rsyslog to write your psad messages to a separate file, told logstash where to find this file and what to do with each type of log line. Use your creativity to send your logstash filtered logs to any of the available outputs; I personally use elasticsearch to store my logs and kibana to visualize them. In a next gist I will likely publish my Kibana PSAD dashboard, but I still need to do some finetuning there first.
If you have any questions, please feel free to contact me: https://www.github.com/netson
Useful links with information about PSAD:
- Official PSAD website
- Official Logstash website
- Puppet module for PSAD: Puppetforge / Github
- Gist: Using PSAD and UFW