Basic server hardening techniques

hardening.md

Trust

As is outlined in this LiveOverflow video, trusting information is hard.

Note that I disagree with some of the key talking points in the video, as you will see later on in this guide.

My credentials, as of time of writing:

• Security+ certified
• Linux admin for over a decade
• Professional Linux admin for about 6 years

Supported systems

Systems supported in this guide.

Debian

• Debian
• Ubuntu
• etc

Redhat

• Redhat
• Fedora
• CentOS
• etc

I'm setting up a new server

This section is for those who are setting up a new system and want to do some basic hardening on it to prevent most common types of attacks.

Terminal emulator

A good terminal emulator is worth its weight in gold.

Linux client

If you're connecting to your server on a Linux OS, you already have the ssh command and, honestly, that's all you really need.

Windows client

MobaXterm is an excellent terminal emulator. It gives nice baked-in features like CPU/memory/disk usage, a file browser, multi-execution mdes, ssh and X11 forwarding, and much more. Even better, it's free!

Shell

Many people love ZSH, Fish, etc as their preferred shell on their systems. I love it, as well. It's a great little environment with tons of baked-in features and toys. Oh-my-zsh is my favorite thing to use on my laptop.

That said, you will encounter Bash most often when doing anything sysadmin-related in the real world. It's very rare that you will use anything other than Shell or Bash on a production system. It's best to be used to working in Bash.

Updates

Let's run some updates on this new box to make sure we've got the latest and greatest before setting anything else up.

# Ubuntu
apt update && apt -y full-upgrade
# Debian or old Ubuntu
apt-get update && apt-get -y full-upgrade
# Redhat
yum -y upgrade
# Old Redhat, Fedora
dnf -y upgrade

Firewall

Now we set up a proper firewall.

Why firewalls

Two main reasons:

1. We can control what's actively responding to requests. If someone manages to open a port on your system, the firewall will stop it from being accessible.
2. We're not just enabling firewalls for inbound traffic, but outbound as well. This will help to stop things like reverse shells.

Debian systems

# Ubuntu
apt install -y ufw
# Debian or old Ubuntu
apt-get install -y ufw

Now it's time to put this server into a DMZ. We'll set up UFW to disallow all inbound and outbound connections unless we specifically allow it.

ufw disable
ufw default deny incoming
ufw default deny outgoing
ufw allow out 53 comment "DNS TCP/UDP"
ufw allow out 123/udp comment "NTP UDP"
ufw allow out 80/tcp comment "HTTP TCP"
ufw allow out 443/tcp comment "HTTPS TCP"
ufw allow out 9418/tcp comment "Git TCP"
ufw allow in 22/tcp comment "SSH TCP"

sed '/^COMMIT.*/i -A ufw-before-output -p icmp -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT' /etc/ufw/before.rules
sed '/^COMMIT.*/i -A ufw-before-output -p icmp -m state --state ESTABLISHED,RELATED -j ACCEPT' /etc/ufw/before.rules

Now, open any ports that you need.

# Allow incoming (from the outside world to the server)
ufw allow in <port>/<protocol>
# Allow outgoing (from the server to the outside world)
ufw allow out <port>/<protocol>

eg. if you want a web server on your box:

ufw allow in 80/tcp comment "HTTP TCP"
ufw allow in 443/tcp comment "HTTPS TCP"

eg. for allowing outbound mail.

ufw allow out 25 comment "SMTP TCP/UDP"
ufw allow out 587 comment "SMTP TCP/UDP"
ufw allow out 465 comment "SMTP TCP/UDP"

Enable the firewall.

ufw --force enable

Redhat systems

# Redhat
yum -y install firewalld
# Old Redhat, Fedora
dnf -y install firewalld

TODO: WIP

Apparmor/SELinux

Please, please use Apparmor and SELinux.

Debian/Apparmor

# Ubuntu
apt install -y apparmor apparmor-profiles apparmor-profiles-extra apparmor-utils
# Debian, old Ubuntu
apt-get install -y apparmor apparmor-profiles apparmor-profiles-extra apparmor-utils

if using Apache:

# Ubuntu
apt install -y libapache2-mod-apparmor
# Debian, old Ubuntu
apt-get install -y libapache2-mod-apparmor

Enable apparmor, if disabled.

systemctl enable --now apparmor

Apparmor commands:

# List apparmor statuses
aa-status
# Enforce an apparmor profile
aa-enforce <program>

Redhat/SELinux

Install SELinux if needed

# Redhat
yum install selinux
# Old Redhat, Fedora
dnf install selinux

Set SELinux to enforcing mode.

grep -qoP '^(#\s*)?SELINUX' /etc/selinux/config && sed -i -E 's/^(#\s*)?SELINUX.*/SELINUX=enforcing/' /etc/selinux/config || echo 'SELINUX=enforcing' >> /etc/selinux/config
setenforce 1

Admin user

There are very few reasons to be logged in as root. If you are, then it's time to set up an admin account with sudo access.

TODO: WIP

SSH keys

You should always be using SSH keys. There's no excuse for allowing password logins.

SSH keys vs passwords

There are various points around "if you use a strong password, then.." - but I'll stop it here. There's no point in creating, keeping, and using a strong password, when the average SSH key is about 15 times the length of a decent password and gets used automatically.

Our monkey brains are terrible with passwords, and password managers are painful to use with SSH clients. Do yourself a favor and make good security easy on yourself. Security fatigue is a real thing, and there's no need to get bogged down in yet-more-passwords-to-remember.

TODO: WIP

Fail2ban & alternate SSH ports

Fail2ban is completely useless if you're using SSH keys (which you absolutely should be) and moving your SSH port from the default 22 to 2222 or some other random port is more trouble than it's worth. Security by obscurity is no security at all.

Hiding the SSH port

The best way to "hide" your SSH port from the world - the way the professionals do it - is by putting your server on a network that only VPN access to it.

The idea, here, is that you expose your SSH port to only this VPN network. In order to even access the SSH port, you need to VPN into this network. The VPN would be exposed to the world, but it's an extra (very high-security) layer on top of your SSH requiring keys to log in (you do have SSH keys required, don't you?)

Updates

While unattended/automatic updates may or may not be desirable, ensuring your system is kept up-to-date at least every once in a while is good practice.

I NEED TO LOCK IT DOWN YESTERDAY

This section is for those who have already found themselves in a bad situation. Maybe your server is sending spam e-mails, or maybe your data is being exfiltrated.

CALM

First, be calm. If you can't think clearly then you can't admin clearly. You need a clear head for this, or you're likely to make mistakes and lock yourself out or lock an adversary in with you.

Good advice?

Quick aside: is this short guide actually any good?

The short answer is "generally, yes".

There's explanations for the things outlined here in the guide above. Right now, your system is under attack. Do now, question later.

PLEASE NOTE

This is a guide to getting back up and running quickly. Performing any of these steps will make forensics impossible. If you need to send information to the FBI, etc then at the very least take disk snapshots first.

Users

We need to list (and kick) any currently logged-in users that shouldn't be logged in. If you perform hardening and another admin is logged in they can simply undo your changes.

List current users

First, get a list of users currently logged into the system.

who

List last logged-in

Now, to compare, a list of all the logins, reboots, etc on the box.

last

This gives us a snapshot to quickly see who logged in from where and maybe a hint at what they might have done. We can check other logs later, but for now we're in emergency mode.

List non-system users

eval getent passwd "{$(awk '/^UID_MIN/ {print $2}' /etc/login.defs)..$(awk '/^UID_MAX/ {print $2}' /etc/login.defs)}" | cut -d: -f1

List root users

getent group | grep 'x:0:' /etc/passwd | cut -d: -f1

List sudoers

getent group root wheel adm admin | cut -d: -f4

List users with shells

getent passwd | awk -F/ '$NF != "nologin" && $NF != "false" && $NF != "sync" && $NF != "!"' | cut -d: -f1

Kick users

Time to kick and lock any users that shouldn't be logged into the system. If there's any accounts you're not sure about, lock them out anyway. You can always undo your own damage later. Assume all accounts that are not you are admins with malicious intent, even if you know who they are.

# Lock user
passwd -l <user>
# Kill all running processes from user
pkill -15 -u <user>
# Forcibly kill all running processes from user
pkill -9 -u <user>

List setuid files

Files with setuid allow any user to execute that file as root. While this is neccesary for some files, with others it can be extremely dangerous. It's worth checking ones that look suspicious before continuing.

find / -perm -04000 2>/dev/null

Check open ports

Ports that are open are ports that people can use to access various parts of your server. It would be wise to check them and see if anything is open that shouldn't be.

netstat -peanut | grep LISTEN

If you get the error: netstat: command not found then install net-tools.

# Ubuntu
apt install net-tools
# Debian or old Ubuntu
apt-get install net-tools
# Redhat
yum install net-tools
# Old Redhat, Fedora
dnf install net-tools

Check crontab & services

Cron

Check the root crontab, user crontabs, and the cron directories for anything suspicious. This is one of the many ways persistant access can be attained.

crontab -l -u root
crontab -l -u <user>

ls -la /etc/cron.d/
ls -la /etc/cron.daily/
ls -la /etc/cron.hourly/
ls -la /etc/cron.monthly/
ls -la /etc/cron.weekly/

Services

Get a list of all services, running or not.

Note: System services can live in /etc/systemd/system/, /run/systemd/system/, and /lib/systemd/system/

systemctl list-units --all --type=service --no-pager

Find a service file

find / -name coredns.service 2> /dev/null

Firewall

Time to lock down your firewall. This is important to prevent anything from going out or coming in without your explicit consent.

Please note that we are about to reset your firewall, and any rules you currently have will be erased.

Check firewall

Check which firewall you have.

UFW:

systemctl status ufw

FirewallD:

systemctl status firewalld

IPTables:

iptables --help

ufw systems

Disable firewall (for safety) and enable basic rules.

ufw disable
ufw --force reset
ufw default deny incoming
ufw default deny outgoing
ufw allow out 53 comment "DNS TCP/UDP"
ufw allow out 123/udp comment "NTP UDP"
ufw allow out 80/tcp comment "HTTP TCP"
ufw allow out 443/tcp comment "HTTPS TCP"
ufw allow out 9418/tcp comment "Git TCP"

sed '/^COMMIT.*/i -A ufw-before-output -p icmp -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT' /etc/ufw/before.rules
sed '/^COMMIT.*/i -A ufw-before-output -p icmp -m state --state ESTABLISHED,RELATED -j ACCEPT' /etc/ufw/before.rules

Open ports that you need.

# Allow incoming
ufw allow in <port>/<protocol>
# Allow outgoing
ufw allow out <port>/<protocol>

eg. for allowing SSH access to your box (can be very important).

ufw allow in 22/tcp comment "SSH TCP"

eg. for allowing outbound mail.

ufw allow out 25 comment "SMTP TCP/UDP"
ufw allow out 587 comment "SMTP TCP/UDP"
ufw allow out 465 comment "SMTP TCP/UDP"

Enable the firewall.

ufw --force enable

firewalld systems

TODO: WIP

iptables systems

TODO: WIP

Logs

Now that everything is locked down, it's time to check the logs.

User logs

Who did what, and when?

Debian-based systems.

less /var/log/auth.log | grep -v CRON

Redhat-based systems.

less /var/log/secure | grep -v CRON

Check bash history for users and root.

less /root/.bash_history
less /home/<user>/.bash_history

Harden

Finally, go back to the top of this page and follow the guide to setting up a new system to properly harden this box so this doesn't happen again.