Skip to content

Instantly share code, notes, and snippets.

@numberoverzero
Last active June 17, 2022 16:31
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save numberoverzero/556e6908d738ee39c40522731859cd13 to your computer and use it in GitHub Desktop.
Save numberoverzero/556e6908d738ee39c40522731859cd13 to your computer and use it in GitHub Desktop.
hardened nginx conf, multiple subdomains under different certs using SNI

SELinux, oh god.

Context

You generated some certs, some dhparams, set up cloudflare origin certs, copied all the settings below, added some content to /var/www/ and set up your nginx.conf, and nginx -t says everything's fine.

Let's load some content! Nope, just kidding, nothing works.

Debugging

In your browser the page doesn't load and you notice your 404 page is also not loading, but instead causing infinite redirects according to /var/nginx/www/access.log so you check /var/nginx/www/error.log and you find lots of this:

2022/06/07 02:52:13 [crit] 2804#2804: *2 stat() "/var/www/404.html" failed (13: Permission denied), client: 172.68.143.217, server: numberoverzero.com, request: "GET /404 HTTP/2.0", host: "numberoverzero.com", referrer: "https://numberoverzero.com/index.html"
2022/06/07 02:52:13 [crit] 2804#2804: *2 stat() "/var/www/404.html" failed (13: Permission denied), client: 172.68.143.217, server: numberoverzero.com, request: "GET /404 HTTP/2.0", host: "numberoverzero.com", referrer: "https://numberoverzero.com/index.html"
2022/06/07 02:52:20 [crit] 2804#2804: *3 stat() "/var/www/index.html" failed (13: Permission denied), client: 172.68.133.105, server: numberoverzero.com, request: "GET /index.html HTTP/2.0", host: "numberoverzero.com"
2022/06/07 02:52:20 [crit] 2804#2804: *3 stat() "/var/www/404.html" failed (13: Permission denied), client: 172.68.133.105, server: numberoverzero.com, request: "GET /index.html HTTP/2.0", host: "numberoverzero.com"
2022/06/07 02:52:20 [crit] 2804#2804: *3 stat() "/var/www/404.html" failed (13: Permission denied), client: 172.68.133.105, server: numberoverzero.com, request: "GET /404 HTTP/2.0", host: "numberoverzero.com", referrer: "https://numberoverzero.com/index.html"
2022/06/07 02:52:20 [crit] 2804#2804: *3 stat() "/var/www/404.html" failed (13: Permission denied), client: 172.68.133.105, server: numberoverzero.com, request: "GET /404 HTTP/2.0", host: "numberoverzero.com", referrer: "https://numberoverzero.com/index.html"
2022/06/07 02:52:52 [crit] 2876#2876: *1 stat() "/var/www/index.html" failed (13: Permission denied), client: 172.68.132.128, server: numberoverzero.com, request: "GET /index.html HTTP/2.0", host: "numberoverzero.com"
2022/06/07 02:52:52 [crit] 2876#2876: *1 stat() "/var/www/404.html" failed (13: Permission denied), client: 172.68.132.128, server: numberoverzero.com, request: "GET /index.html HTTP/2.0", host: "numberoverzero.com"

You verify that your user owns the directory, but that doesn't seem to matter:

$ ls -lah /var/www/
drwxr-xr-x.  3 crossj crossj   51 Jun  7 02:08 .
drwxr-xr-x. 22 root   root   4.0K Jun  7 02:43 ..
-rw-r--r--.  1 crossj crossj  270 Apr 13  2020 404.html
-rw-rw-r--.  1 crossj crossj    6 Jun  7 02:08 index.html

Before you trip down this massive sinkhole again (again), here's the problem. Again. SELinux.

There's a context httpd_sys_content_t missing on the directory or a file within it, and SELinux prevents the read. When things are set up correctly, it looks like this:

$ ls -Z /var/www/
-rw-r--r--. crossj crossj unconfined_u:object_r:httpd_sys_content_t:s0 404.html
-rw-rw-r--. crossj crossj unconfined_u:object_r:httpd_sys_content_t:s0 index.html

Fixing

Add the context to the directory, then preserve the changes.

sudo semanage fcontext -a -t httpd_sys_content_t "/var/www(/.*)?"
sudo restorecon -R -v /var/www

You should see the /var/www line in this SELinux config file:

$ cat /etc/selinux/targeted/contexts/files/file_contexts.local
# This file is auto-generated by libsemanage
# Do not edit directly.

/etc/(letsencrypt|certbot)/(live|archive)(/.*)?    system_u:object_r:cert_t:s0
/var/www(/.*)?    system_u:object_r:httpd_sys_content_t:s0

See Also

# sudo mkdir -p /etc/nginx/certs/cloudflare
# sudo mkdir -p /etc/nginx/certs/letsencrypt
# sudo wget -O /etc/nginx/certs/letsencrypt/x3-cross-signed.pem https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt
# sudo curl -o /etc/nginx/certs/cloudflare/origin-pull.pem https://developers.cloudflare.com/ssl/static/authenticated_origin_pull_ca.pem
# certbot: learn how to install this elsewhere.
# I used the following commands:
# certbot --nginx -d mydomain.com
# certbot --nginx -d whatever.mydomain.com
# certbot --nginx -d another.mydomain.com
# verify that there's a crontab configured to run `certbot renew`
# I had the following in /etc/crontab:
# 0 0,12 * * * root python -c 'import random; import time; time.sleep(random.random() * 3600)' && certbot renew
# hardening reading material
# https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
# https://support.cloudflare.com/hc/en-us/articles/360006660072
# testing sites
# https://www.ssllabs.com/ssltest
# http://dnsviz.net
# https://dnssec-analyzer.verisignlabs.com
# use these cloudflare crypto settings:
# ssl: strict
# always use https: off # nginx will 301 http anyway
# hsts: off # enable once you've set things up very carefully
# authenticated origin pulls: on
# minimum tls version: 1.2
# opportunistic encryption: on
# onion routing: on
# tls 1.3: on
# automatic https rewrites: off # just hides problems with mixed media
# hey kid, you're running with SELinux, so you're going to want to read about that.
# when in doubt, switch into the permissive context, see if it works, then re-enable enforcing.
# https://web.archive.org/web/20211118000712/https://www.nginx.com/blog/using-nginx-plus-with-selinux/
# https://www.nginx.com/blog/using-nginx-plus-with-selinux/
# sudo semanage permissive -a httpd_t
# sudo semanage permissive -d httpd_t
# while running in permissive, you can tail the following log file to see errors that
# wouldn't be allowed in enforcing mode:
# tail -f /var/log/audit/audit.log
# you'll also want to check out audit2why. consider the following entry in audit.log:
# type=AVC msg=audit(1415714880.156:29): avc: denied { name_connect } for pid=1349 \
# comm="nginx" dest=8080 scontext=unconfined_u:system_r:httpd_t:s0 \
# tcontext=system_u:object_r:http_cache_port_t:s0 tclass=tcp_socket
# to decode this:
# grep 1415714880.156:29 /var/log/audit/audit.log | audit2why
user nginx;
worker_processes auto;
pid /run/nginx.pid;
events {
# 2^16 - previous limit (1024)
worker_connections 64512;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
types_hash_max_size 2048;
# https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options
# https://www.owasp.org/index.php/OWASP_Secure_Headers_Project#xfo
# use "sameorigin" to enable frames on the same domain
add_header X-Frame-Options deny;
# https://www.owasp.org/index.php/OWASP_Secure_Headers_Project#xcto
add_header X-Content-Type-Options nosniff;
# https://www.owasp.org/index.php/OWASP_Secure_Headers_Project#xxxsp
add_header X-XSS-Protection "1; mode=block";
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Logging
# -------
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
# Compression
# -----------
gzip on;
gzip_disable "msie6";
# Hardening
# ---------
server_tokens off;
error_page 401 403 404 /404.html;
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
# Buffers
# -------
client_body_buffer_size 64k;
client_header_buffer_size 1k;
client_max_body_size 128k;
large_client_header_buffers 4 4k;
# =====
# SSL
# =====
# Client TLS
# https://support.cloudflare.com/hc/en-us/articles/204899617
# sudo mkdir -p /etc/nginx/certs/cloudflare
# sudo wget -O /etc/nginx/certs/cloudflare/origin-pull.pem https://support.cloudflare.com/hc/en-us/article_attachments/360044928032/origin-pull-ca.pem
ssl_client_certificate /etc/nginx/certs/cloudflare/origin-pull.pem;
ssl_verify_client on;
# Server Certificates (Let's Encrypt)
# configure these in each /etc/nginx/conf.d/*.conf virtual server
# Keep sessions small
ssl_session_timeout 5m;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
# Long-lived 4096 Diffie-Hellman parameter
# you can either use one dhparam for all servers by putting it here, or
# generate a unique value per virtual server. The following steps generate a
# shared one for all virtual servers:
# mkdir /etc/nginx/certs
# openssl dhparam -out /etc/nginx/certs/dhparam.pem 4096
# then, uncomment the following line:
# ssl_dhparam /etc/nginx/certs/dhparam.pem;
# Protocols
# ---------
ssl_protocols TLSv1.2 TLSv1.3;
# TODO REVIEW THIS LIST PERIODICALLY
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
ssl_prefer_server_ciphers on;
# HSTS
# ----
# https://support.cloudflare.com/hc/en-us/articles/204183088-Understanding-HSTS-HTTP-Strict-Transport-Security-
# TODO configure with DNS provider and cloudflare when everything else is solid.
# TODO don't get it wrong. changes may take 6 months to propagate to clients.
# add_header Strict-Transport-Security max-age=15768000; # 6 months
# OCSP Stapling
# -------------
# sudo mkdir -p /etc/nginx/certs/letsencrypt
# sudo wget -O /etc/nginx/certs/letsencrypt/x3-cross-signed.pem https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/nginx/certs/letsencrypt/x3-cross-signed.pem;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;
# ======================
# Virtual Host Configs
# ======================
# http -> https
# -------------
server {
listen 80 default_server;
listen [::]:80 default_server;
location ^~ /.well-known/acme-challenge {
default_type text/plain;
root /var/www/letsencrypt;
}
return 301 https://$host$request_uri;
}
# here's where you'll add virtual servers. each will be a server{} block
include /etc/nginx/conf.d/*.conf;
}
# assumptions:
# * "example.com" is our root domain
# * "something.example.com" is the subdomain being handled in this server block
# * "another.example.com" exists and is configured in an adjacent config file "another.conf"
# don't forget to make selinux play nice with these ports
# (or just blast open httpd_can_network_connect):
# sudo setsebool httpd_can_network_connect 1 -P
# upstream falconpy {
# server unix:///tmp/uswgi.sock;
# }
# upstream rocket {
# server http://127.0.0.1:9090;
# }
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name something.example.com;
# suggested convention: log <subdomain>.example.com to /var/log/nginx/<subdomain>/{access,error}.log
access_log /var/log/nginx/something/access.log;
error_log /var/log/nginx/something/error.log;
# certbot --nginx -d something.example.com
ssl_certificate /etc/letsencrypt/live/something.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/something.example.com/privkey.pem;
# if you want to use the same dhparams across virtual servers, comment this line
# and uncomment the ssl_dhparam setting in /etc/nginx/nginx.conf
# openssl dhparam -out /etc/nginx/certs/something.example.com/dhparam.pem 4096
ssl_dhparam /etc/nginx/certs/something.example.com/dhparam.pem;
root /var/something;
## the following is preference for cleaning up urls.
## eg. amazon.com/checkout/ --> amazon.com/checkout
# rewrite ^/(.*)/$ /$1 permanent;
## eg. amazon.com/about.html --> amazon.com/about
# rewrite ^/(.*)\.html$ /$1 permanent;
## STATIC CONTENT: my standard search rules.
## 0. name as given (css/normalize.min.css)
## 1. append .html for single pages (/login -> /login.html)
## 2. append /index.html for directories (/users -> /users/index.html)
# location / {
# try_files $uri $uri.html $uri/index.html =404;
# }
## PYTHON PROXY: uwsgi to falconpy
## see also: https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html
## see also: https://falcon.readthedocs.io/en/stable/deploy/nginx-uwsgi.html
# location / {
# include /etc/nginx/uwsgi_params;
# uwsgi_pass falconpy;
# }
## RUST PROXY: nearly the same, include any forward headers you need
# location / {
# proxy_pass rocket;
# ## http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering
# ## proxy_buffering off <-- only releveant in some long-polling scenarios.
# ## http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header
# proxy_set_header Host $host;
# }
}

nginx validation

Check current config and any includes:

sudo nginx -t

Print all included config:

sudo nginx -T

watch those logs

assuming you followed the conventions in nginx.conf:

tail -f /var/logs/nginx/www/access.log
tail -f /var/logs/nginx/www/error.log

apply nginx config changes

For most config changes, use reload. This reloads config without killing the process tree.

sudo service nginx reload

for interface and port changes, use restart. This shuts down the process tree before starting up fresh.

sudo service nginx restart
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment