NGINX SSL Reverse Proxy, w/ fail2ban, letsencrypt, and iptables-persistent

nginx-ssl-proxy.md

Installing NGINX SSL Reverse Proxy, w/ fail2ban, letsencrypt, and iptables-persistent.

The steps outlined here make many assumptions about both your operating environment and your understanding of the Linux OS and services running on Linux. I am using the current LTS Ubuntu distribution 16.04 running in the cloud on a DigitalOcean Droplet. These steps should work just fine if followed closely and in order... but you know sh!& never works out the way you plan in Linux. I offer some troubleshooting advice, Google and Stack are your friends ask them for help.

(Entered on your local workstation)

1. ssh-add <path-to-ssh-key> and enter key passphrase when prompted. // ex path... /users/you/.ssh/id_rsa_digital-ocean
2. ssh root@<server-url-or-IP> // should login without prompting for passphrase

(Entered on your remote server)

1. adduser <username> and follow prompts
2. visudo to edit the sudoers file and add a new line

<username>    ALL=(ALL:ALL) ALL

(Edited on remote server)

1. pico /etc/ssh/sshd_config
2. Change PasswordAuthentication no to PasswordAuthentication yes and save.
3. Restart ssh service systemctl restart ssh

(Entered on your local workstation)

1. Copy ssh-key ssh-copy-id <username>@<server-url-or-IP>
2. Log into remote server ssh <username>@<server-url-or-IP>

(Entered on your remote server)

1. Update packages sudo apt-get update
2. Install fail2ban to tighten SSH security sudo apt-get install -y fail2ban
3. Create/edit fail2ban jail.local config file sudo pico /etc/fail2ban/jail.local

[DEFAULT]
bantime = 864000
findtime = 600
maxretry = 3
[sshd]
enabled = true
bantime = 864000

4. Edit SSH service config sudo pico /etc/ssh/sshd_config Note: This step can lock you out of SSH if you are not authenticating with a cert. Have a backup access plan ready.

PermitRootLogin no
PasswordAuthentication no
## Optional if you are not using X11Forwarding
X11Forwarding no

5. Start fail2ban service sudo systemctl start fail2ban.service
6. Stop fail2ban service for the Firewall setup to insure f2b rules are not saved in the base rule set. sudo systemctl stop fail2ban.service
7. Install iptables-persistent sudo apt-get install -y iptables-persistent
8. Setup base firewall rules for IPv4. sudo pico /etc/iptables/rules.v4 replace the contents of rules.v4 with:

*filter
# Allow all outgoing, but drop incoming and forwarding packets by default
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]

# Custom per-protocol chains
:UDP - [0:0]
:TCP - [0:0]
:ICMP - [0:0]

# Acceptable UDP traffic

# Acceptable TCP traffic
-A TCP -p tcp --dport 22 -j ACCEPT
-A TCP -p tcp --dport 80 -j ACCEPT
-A TCP -p tcp --dport 443 -j ACCEPT

# Acceptable ICMP traffic

# Boilerplate acceptance policy
-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-A INPUT -i lo -j ACCEPT

# Drop invalid packets
-A INPUT -m conntrack --ctstate INVALID -j DROP

# Pass traffic to protocol-specific chains
## Only allow new connections (established and related should already be handled)
## For TCP, additionally only allow new SYN packets since that is the only valid
## method for establishing a new TCP connection
-A INPUT -p udp -m conntrack --ctstate NEW -j UDP
-A INPUT -p tcp --syn -m conntrack --ctstate NEW -j TCP
-A INPUT -p icmp -m conntrack --ctstate NEW -j ICMP

# Reject anything that's fallen through to this point
## Try to be protocol-specific w/ rejection message
-A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable
-A INPUT -p tcp -j REJECT --reject-with tcp-reset
-A INPUT -j REJECT --reject-with icmp-proto-unreachable

# Commit the changes
COMMIT

*raw
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT

*security
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT

*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT

8. Test firewall rules sudo iptables-restore -t /etc/iptables/rules.v4 no output is a passing test
9. Make firewall wall rules active sudo iptables-restore -v /etc/iptables/rules.v4
10. Setup base firewall rules for IPv6. Note: we are not using IPv6, so these rules DROP all IPv6 traffic sudo pico /etc/iptables/rules.v6 replace the contents of rules.v6 with:

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
COMMIT

*raw
:PREROUTING DROP [0:0]
:OUTPUT DROP [0:0]
COMMIT

*nat
:PREROUTING DROP [0:0]
:INPUT DROP [0:0]
:OUTPUT DROP [0:0]
:POSTROUTING DROP [0:0]
COMMIT

*security
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
COMMIT

*mangle
:PREROUTING DROP [0:0]
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
:POSTROUTING DROP [0:0]
COMMIT

18. Test firewall rules sudo ip6tables-restore -t /etc/iptables/rules.v6 no output is a passing test
19. Make firewall wall rules active sudo ip6tables-restore -v /etc/iptables/rules.v6
20. Install certbot to automate Let's Encrypt certificate generation sudo apt-get update && sudo apt-get install software-properties-common && sudo add-apt-repository ppa:certbot/certbot && sudo apt-get update && sudo apt-get install certbot

22. Make temporary edit to default site so we can get a server certificate and private key sudo nano /etc/nginx/sites-available/default. Add the following location inside the default server block 23. Save then test your config with sudo nginx -t 24. If your changes are passing, sudo systemctl restart nginx

21. Generate certificates using certbot sudo certbot certonly --standalone -d <server-url> Note: use a -d <server-url> for each url you want to serve on this certificate - follow prompts.
22. Install NGINX web proxy server. sudo apt-get install -y nginx
23. Generate a strong Diffie-Hellman group key sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048
24. Create include snippets for the NGINX ssl configs sudo pico /etc/nginx/snippets/ssl-<server-url>.conf

ssl_certificate /etc/letsencrypt/live/<server-url>/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/<server-url>/privkey.pem;

28. Create include snippet for strong encryption settings sudo pico /etc/nginx/snippets/ssl-params.conf

# from https://cipherli.st/
# and https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_ecdh_curve secp384r1;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;

ssl_dhparam /etc/ssl/certs/dhparam.pem;

21. Move the default website config to a backup and create a new default file. sudo mv /etc/nginx/sites-available/default /etc/nginx/sites-available/default.backup && sudo touch /etc/nginx/sites-available/default
22. Create temporary website config to test ssl sudo pico /etc/nginx/sites-available/default

# Server configuration

# Port 80 redirect and cert renew path
server {
  listen 80 default_server;
 # listen [::]:80 default_server;

 # commented for ssl config.
 #root /var/www/html;

  # Add index.php to the list if you are using PHP
  index index.html index.htm index.nginx-debian.html;

  server_name <your_servers_FQDN>;  ## for example mine is: splunk.samdennon.com

  location '/.well-known/acme-challenge' {
    default_type "text/plain";
    root /tmp/letsencrypt-auto;
  }

  location / {
    return 301 https://$server_name$request_uri;
  }

 #location / {
   # First attempt to serve request as file, then
   # as directory, then fall back to displaying a 404.
 #  try_files $uri $uri/ =404;
 #}
}

## Port 443 config ##
server {

  ## Config block ##
  listen 443 ssl http2 default_server;
  server_name <your_server_FQDN>;   ## for example mine is: splunk.samdennon.com
  large_client_header_buffers 6 16k;
  client_max_body_size 100M;
  include snippets/ssl-<your_server_FQDN>.conf;    ## for example mine is: splunk.samdennon.com
  include snippets/ssl-params.conf;
  root /var/www/html;
  index index.html index.htm;
  server_name _;

  ## Testing location block ##
#  location / {
#    try_files $uri $uri/ =404;
#  }

  ## Proxy config block##
  location / {
    proxy_pass_request_headers on;
    proxy_set_header x-real-IP $remote_addr;
    proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
    proxy_set_header host $host;
    proxy_pass https://<your_server_FQDN>:8000/;  ## looks like this on my server: proxy_pass https://splunk.samdennon.com:8000;
         }


  ## Custom Error Pages ##
  error_page 404 /custom_404.html;
  location = /custom_404.html {
    root /var/www/html;
    internal;
  }

  error_page 500 502 503 504 /custom_50x.html;
  location = /custom_50x.html {
    root /var/www/html;
    internal;
    }
}

23. Restart NGINX sudo systemctl restart nginx
24. Test using browser or openssl openssl s_client -connect <server-url-or-IP>:443
25. The basics of securing the server and getting SSL working with Let's Encrypt and NGINX are done! You can now continue to install and configure other services. Install Splunk>

This code changed in /etc/nginx/snippets/ssl-params.conf to fix server abort error.

OLD

add_header X-Frame-Options DENY;

NEW

add_header X-Frame-Options SAMEORIGIN;

Added this code to /etc/nginx/sites-available/default in the ## Port 443 config ## to fix file upload issues. Mod size if larger files are needed.

client_max_body_size 100M;

New nginx site config file to allow letsencrypt updating. *** Also added custom 50x and 404 error pages ***

# Server configuration

# Port 80 redirect and cert renew path
server {
  listen 80 default_server;
 # listen [::]:80 default_server;

 # commented for ssl config.
 #root /var/www/html;

  # Add index.php to the list if you are using PHP
  index index.html index.htm index.nginx-debian.html;

  server_name <your_servers_FQDN>;  ## for example mine is: splunk.samdennon.com

  location '/.well-known/acme-challenge' {
    default_type "text/plain";
    root /tmp/letsencrypt-auto;
  }

  location / {
    return 301 https://$server_name$request_uri;
  }

 #location / {
   # First attempt to serve request as file, then
   # as directory, then fall back to displaying a 404.
 #  try_files $uri $uri/ =404;
 #}
}

## Port 443 config ##
server {

  ## Config block ##
  listen 443 ssl http2 default_server;
  server_name <your_server_FQDN>;   ## for example mine is: splunk.samdennon.com
  large_client_header_buffers 6 16k;
  client_max_body_size 100M;
  include snippets/ssl-<your_server_FQDN>.conf;    ## for example mine is: splunk.samdennon.com
  include snippets/ssl-params.conf;
  root /var/www/html;
  index index.html index.htm;
  server_name _;

  ## Testing location block ##
#  location / {
#    try_files $uri $uri/ =404;
#  }

  ## Proxy config block##
  location / {
    proxy_pass_request_headers on;
    proxy_set_header x-real-IP $remote_addr;
    proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
    proxy_set_header host $host;
    proxy_pass https://<your_server_FQDN>:8000/;  ## looks like this on my server: proxy_pass https://splunk.samdennon.com:8000;
         }


  ## Custom Error Pages ##
  error_page 404 /custom_404.html;
  location = /custom_404.html {
    root /var/www/html;
    internal;
  }

  error_page 500 502 503 504 /custom_50x.html;
  location = /custom_50x.html {
    root /var/www/html;
    internal;
    }
}

Changed cert gen to "certbot" using the standalone method. It's just easier than messing with all the NGINX config crap.

Add these steps in: https://www.abuseipdb.com/fail2ban.html