Minimizing OSSEC System Update Warnings

README.md

Introduction

Everyone who runs OSSEC on a Unix system has a common problem: you want to follow and apply security udpates closely, but every time you patch, you get a flood of alerts. And this problem quickly grows: if a given package update would result in five alerts, that's fine if you only have one server. But if you have a hundred servers? Five hundred? Five thousand?

So, I've cobbled some stuff together to abuse the OSSEC's Active Response mechanism to not raise an alert when a package is upgraded properly. I've tried to emulate the workflow of a human administrator as closely as closely as possible, but there are definitely some areas that could be handled better -- see Caveats below.

Objectives

Any time I get an OSSEC alert, I'll do any number of things that generally fall into three categories:

1. Validation of automated change: make sure the file was supposed to be upgraded: check the auto-upgrade logs, verify the package that installed/upgraded the file, etc.
2. Validation of expected update: take a look at how many other systems have applied / are applying the same update.
3. Validation of manual change: confirm with my peers / the NOC / Operations that the change was, in fact, supposed to happen, check deploy logs, check orchestration / config management logs, etc.

The idea is to automate as much of this work as possible, while still raising anything questionable as a standard alert for human analysis. And to do so without introducing significant additional complexity, or new systems. Thankfully, OSSEC contains pretty much everything we need to do this quite reliably.

It's computationally difficult to check with peers, and computationally challenging to confirm change across multiple systems. But it should be trivial to confirm the change within the context of a single system.

The target audience here are people who are already, or are considering, wiping their syscheck database whenever they do system updates. This approach is considerably more trusted and secure.

Caveats and Drawbacks

As with any automated system, there are a number of possible ways for this to fall apart. I've taken some precautions to ensure that the script fails in a predictable manner, and never silently, but this is always a risk. As well, abusing Active Response like this is a significant deviation from its intended purpose. Ideally, OSSEC would provide an alert filter, where we could plug in our own analysis framework, but this isn't the case today.

In order for this to work, we need a way to verify files. This means that this will only work if your system has a package manager that is able to validate the files on disk against recorded checksums and signatures in the package database. Preferrably, your package database contains some form of cryptographic signatures.

There is one relatively obvious flaw with this approach: single-file changes via a 'valid' package. If, say, your /usr/sbin/sshd is updated via package install, this process will quietly accept this change as valid. Ideally, we would do some form of statistical analysis, and any one-off changes would be raised as alerts requiring investigation. Unfortunately, we can't do that easily with OSSEC. The only way I know to address this would be to deliver all file changes to a centralized location that not only validates the checksums of the new files against a known-good database, but is also able to raise alerts for one-off changes.

As this is just Active Response configuration, this should work without issue for OSSEC in standalone mode, and in agent-server mode as well, although that has not been tested. There is some inherent risk in running this: if your client is compromised, you can't really depend on it to validate its own binaries. However, additional risk is minimal at worst, as you are already trusting client-side input (file metadata reported by the OSSEC agent).

Implementation

Enough with the jabber, let's get this thing going!

There are three separate things to do to get OSSEC to quiet down a bit:

1. A script that understands Active Response and verifies files in a package
2. The Active Response integration
3. Custom OSSEC alerts for Active Response failures

The Script

Customize the script (see below) as you see fit, and drop it into your Active Response binary directory, usually /var/ossec/active-response/bin/. That's about all we have to do here.

Something to note is that dpkg-deb --verify is still a very new command, and you'll need to be running Ubuntu 14.04 (Trusty) or newer, or Debian 7.0 or newer (I think), to have this flag. rpm -f / rpm -V has existed for ages, so if you're using RPMs, you're fine.

The script is not written for RPM package systems. I don't have such a system to test on.

Active Response Integration

Add these rules to your ossec.conf. They're standard, straightforward AR rules:

<command>
  <name>verify-file-deb</name>
  <executable>verify-file-deb.sh</executable>
  <expect></expect>
  <timeout_allowed>no</timeout_allowed>
</command>

<active-response>
  <command>verify-file-deb</command>
  <location>local</location>
  <rules_group>file_verification</rules_group>
  <disabled>no</disabled>
</active-response>

In order for these to work, you have to group the rules together that you want to hook into a file_verification group, like this:

<group name="local,syscheck,">
  <rule id="200001" level="3">
    <if_sid>550</if_sid>
    <description>Integrity checksum changed.</description>
    <group>file_verification,</group>
  </rule>
  <rule id="200002" level="3">
    <if_sid>551</if_sid>
    <description>Integrity checksum changed again (2nd time).</description>
    <group>file_verification,</group>
  </rule>
  <rule id="200003" level="3">
    <if_sid>552</if_sid>
    <description>Integrity checksum changed again (3rd time).</description>
    <group>file_verification,</group>
  </rule>
  <rule id="200004" level="3">
    <if_sid>554</if_sid>
    <description>File added to the system.</description>
    <group>file_verification,</group>
  </rule>
</group>

Monitor for Active Response failures

This part is key: the last thing you want to have happen is your file verifier start failing, but not tell you that it's failing. If you look at the script, it's written to always log succeeded or failed to the Active Response log when it exits. This is a bit of an abuse of the AR log, but it means that we can monitor our own logfile for failures.

What you want to do is add a custom rule that watches for Active Response logs, and raises an alert when the keyword failed is detected. So add this to your local_rules.xml:

<group name="local,ossec,active_response,">
  <rule id="300001" level="10">
    <if_sid>600</if_sid>
    <status>failed</status>
    <description>Failed active response.</description>
  </rule>
</group>

And that's it! You should now have an OSSEC system that doesn't raise alerts when packages are upgraded properly, but still warns you if anything sensitive has changed.

verify-file-deb.sh

#!/bin/sh

set -eE

# Log an active response failure by default
trap "log_failure" EXIT

ACTION="${1}"
USER="${2}"
IP="${3}"
ALERTID="${4}"
RULEID="${5}"

# The location of the active response logfile
ACTIVE_RESPONSE_LOG="/var/ossec/logs/active-responses.log"

# The maximum number of file verifiers to run at any given time
# Without this, on a large update, there's a serious risk of a local resource exhaustion DoS,
# as package database lookups can be expensive
MAX_VERIFIERS=50

HOSTNAME="${$(/bin/hostname -f 2>/dev/null):-unknown}"
SUBJECT="OSSEC Notification - ${HOSTNAME} - File Verification Alert"

log_active_response() {
	local action="${1:-${ACTION}}"
	echo "$(date) ${0} ${action} ${USER} ${IP} ${ALERTID} ${RULEID}" >> ${ACTIVE_RESPONSE_LOG}

	trap - EXIT
	exit ${2:-253}
}

log_failure() {
  log_active_response failed 255
}

send_alert() {
	local sender="ossec@${HOSTNAME}"
	local recipients="foo@example.com,bar@example.com"

	local additional="${1}\n"
	local alert="${2:-Failed to verify file change status against package database.\n\nAlert ID: ${ALERTID}\nAction: ${ACTION}\nUser: ${USER}}"

	/bin/echo -e "From: ${sender}
To: ${recipients}
Subject: ${SUBJECT}

${additional}
${alert}" | /usr/sbin/sendmail -oi -t

	# Make sure we can trust the binaries we've just used (we're kinda screwed on grep and echo)
	echo "${additional}" | grep -q " /usr/sbin/sendmail " && log_failure
	echo "${additional}" | grep -q " $(/usr/bin/realpath /usr/sbin/sendmail) " && log_failure
	echo "${additional}" | grep -q " /usr/bin/realpath " && log_failure

	log_active_response succeeded 0
}

verify_file() {
	test -n "${1}" || send_alert "verify_file: No filename provided for source package identification."
	local fn="${1}"

  # Obtain the package name for the file
  # If using RPMs, you can skip this part, and jump right to file verification with 'rpm -f <filename>'
	local pkgn="$(/usr/bin/dpkg -S ${fn} 2>/dev/null | cut -d: -f1)"

	# dpkg --verify pretty much always returns 0, so we need to examine its output to determine success/failure
	# Basically, if it has output, we have failure.  But this verifies the whole package, so we need to look for just
	# the file in question.
	# If using rpm, you'll need to sort out its error codes / output here
	# Note that at least dpkg doesn't verify configuration files
	local verification="$(/usr/bin/dpkg --verify "${pkgn}" 2>&1 | egrep "[[:space:]]${fn}\$")"
	test -z "${verification}" || send_alert "CRITICAL: Changed file '${fn}' failed package verification for '${pkgn}':\n${verification}" "${ALERT}"
}

test -z "${ALERTID}" && send_alert "No alert ID was provided; unable to determine the impacted file."

ALERT="$(/bin/sed -n "/${ALERTID}: /,/^$/p" "/var/ossec/logs/alerts/alerts.log")"
test -n "${ALERT}" || send_alert "Unable to identify the originating alert based on the alert ID '${ALERTID}'."

INSTANCE_COUNT=$(ps auxw | egrep "${0}" | grep -v grep | wc -l)
test ${INSTANCE_COUNT} -gt ${MAX_VERIFIERS} && send_alert "Too many file verifiers running." "${ALERT}"

CHANGED_FILE="$(echo "${ALERT}" | /usr/bin/awk "BEGIN {FS=\"'\"} ; /^Integrity checksum changed for: |File '.*' was re-added.$|New file '.*' added to the file system.$/ {print \$2}")"
test -n "${CHANGED_FILE}" || send_alert "Unable to identify the changed file." "${ALERT}"
test -e "${CHANGED_FILE}" || send_alert "Changed file does not appear to exist." "${ALERT}"

# Always raise an alert for stuff in certain portions of the filesystem
echo "${CHANGED_FILE}" | egrep -q '^\/(etc|home|root|boot|var\/ossec)\/' && send_alert "File changed in sensitive area of filesystem." "${ALERT}"

verify_file "${CHANGED_FILE}"

# Special handling for files upon which this script depends
# You'll want to add anything else in here that you use
for fn in "/bin/echo" "/usr/sbin/sendmail" "/bin/sed" "/usr/bin/awk" "/usr/bin/dpkg" "/usr/bin/realpath"
do
	rfn="$(/usr/bin/realpath "${fn}")"
	test "${CHANGED_FILE}" = "${fn}" && send_alert "Verifications depends on '${fn}' which has been updated, but passes verification." "${ALERT}"
	test "${rfn}" = "${CHANGED_FILE}" && send_alert "Verification depends on '${fn}', which resolves to '${rfn}', which has been updated, but passes verification." "${ALERT}"
done

# Looks like the file checked out.  Success!
log_active_response succeeded 0

# This should never happen
exit 127