Skip to content

Instantly share code, notes, and snippets.

@saniaky
Last active October 30, 2024 15:30
Show Gist options
  • Save saniaky/dc75cbf64922e418400b0f54ed5b2c3a to your computer and use it in GitHub Desktop.
Save saniaky/dc75cbf64922e418400b0f54ed5b2c3a to your computer and use it in GitHub Desktop.
Docker + nginx-proxy + let's encrypt + watchtower + fail2ban

Complete solution for websites hosting

This gist contains example of how you can configure nginx reverse-proxy with autmatic container discovery, SSL certificates generation (using Let's Encrypt) and auto updates.

Features:

  • Automatically detect new containers and reconfigure nginx reverse-proxy
  • Automatically generate/update SSL certificates for all specified containers.
  • Watch for new docker images and update them.
  • Ban bots and hackers who are trying to bruteforce your website or do anything suspicious.

Techonolgy stack:

Watchtower

With watchtower you can update the running version of your containerized app simply by pushing a new image to the Docker Hub or your own image registry. Watchtower will pull down your new image, gracefully shut down your existing container and restart it with the same options that were used when it was deployed initially.

In this example we run watchtower with the following command: "--interval 60 --cleanup", so basically we ask watchtower to check for new images every minute and remove old images when udpate is performed.

Fail2Ban

Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs -- too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured.

In this example I created two configuration rules - "wplogin" and "phpmyadmin".

  • "wplogin" rule: if someone will make 3 POST call to "wp-login.php" within 10 minutes - consider it as a brute force attack and block such IP for 10 minutes.
  • "phpmyadmin" rule: I don't host any "phpmyadmin" implementation on the website, so if someone (or some vulnerability scanner to be preciese) will try reach such URL - block his IP for an hour.

How to use fail2ban client?

# check status
$ sudo docker exec -t fail2ban fail2ban-client status
Status
|- Number of jail:	2
`- Jail list:	phpmyadmin, wplogin

# Check status of specific jail
$ sudo docker exec -t fail2ban fail2ban-client status phpmyadmin
Status for the jail: phpmyadmin
|- Filter
|  |- Currently failed:	0
|  |- Total failed:	2
|  `- File list: /container-logs/[....]-json.log
`- Actions
   |- Currently banned:	0
   |- Total banned:	2
   `- Banned IP list:	


# ban specific IP
$ sudo docker exec -t fail2ban fail2ban-client set <JAIL> banip <IP>

# unban specific IP
$ sudo docker exec -t fail2ban fail2ban-client set <JAIL> banip <IP>

# Context - http
# Location insdie docker container - /etc/nginx/conf.d/custom-nginx.conf:ro
client_max_body_size 32m;
gzip on;
# Restore real user IP if using Clouflare
# If you are using Clouflare or any other reverse-proxy - uncomment the following lines.
# Because Clouflare replaces user IP address we need to convert it back for correct work of fail2ban.
#set_real_ip_from 103.21.244.0/22;
#set_real_ip_from 103.22.200.0/22;
#set_real_ip_from 103.31.4.0/22;
#set_real_ip_from 104.16.0.0/12;
#set_real_ip_from 108.162.192.0/18;
#set_real_ip_from 131.0.72.0/22;
#set_real_ip_from 141.101.64.0/18;
#set_real_ip_from 162.158.0.0/15;
#set_real_ip_from 172.64.0.0/13;
#set_real_ip_from 173.245.48.0/20;
#set_real_ip_from 188.114.96.0/20;
#set_real_ip_from 190.93.240.0/20;
#set_real_ip_from 197.234.240.0/22;
#set_real_ip_from 198.41.128.0/17;
#set_real_ip_from 2400:cb00::/32;
#set_real_ip_from 2606:4700::/32;
#set_real_ip_from 2803:f800::/32;
#set_real_ip_from 2405:b500::/32;
#set_real_ip_from 2405:8100::/32;
#set_real_ip_from 2c0f:f248::/32;
#set_real_ip_from 2a06:98c0::/29;
#real_ip_header CF-Connecting-IP;
#real_ip_header X-Forwarded-For;
version: '3'
services:
# NGINX reverse-proxy
nginx-proxy:
image: jwilder/nginx-proxy
container_name: nginx-proxy
restart: unless-stopped
ports: ["80:80", "443:443"]
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
- ./custom-nginx.conf:/etc/nginx/conf.d/custom-nginx.conf:ro
- ./nginx-data/conf.d:/etc/nginx/conf.d
- ./nginx-data/vhost.d:/etc/nginx/vhost.d
- ./nginx-data/html:/usr/share/nginx/html
- ./nginx-data/certs:/etc/nginx/certs:ro
- ./nginx-data/htpasswd:/etc/nginx/htpasswd:ro
labels:
com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: "true"
environment:
HTTPS_METHOD: "noredirect"
logging:
driver: "json-file"
options:
max-size: "5m"
max-file: "10"
# Automatically add SSL certificates from Let's Encrypt
nginx-letsencrypt:
image: jrcs/letsencrypt-nginx-proxy-companion
container_name: nginx-letsencrypt
restart: unless-stopped
environment:
NGINX_PROXY_CONTAINER: nginx-proxy
# Uncoment to test on Staging
#ACME_CA_URI: https://acme-staging.api.letsencrypt.org/directory
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./nginx-data/conf.d:/etc/nginx/conf.d
- ./nginx-data/vhost.d:/etc/nginx/vhost.d
- ./nginx-data/html:/usr/share/nginx/html
- ./nginx-data/certs:/etc/nginx/certs:rw
logging:
driver: "json-file"
options:
max-size: "5m"
max-file: "10"
watchtower:
image: containrrr/watchtower
container_name: watchtower
restart: unless-stopped
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /root/.docker/config.json:/config.json
command: --interval 60 --cleanup
logging:
driver: "json-file"
options:
max-size: "5m"
max-file: "10"
networks:
default:
external:
name: edge
version: '3'
services:
fail2ban:
image: crazymax/fail2ban:latest
container_name: fail2ban
restart: "unless-stopped"
network_mode: "host"
cap_add:
- NET_ADMIN
- NET_RAW
volumes:
- "./fail2ban-data:/data"
- "/var/log:/var/log:ro"
- "/var/lib/docker/containers/:/container-logs/:ro"
env_file:
- "./fail2ban.env"
logging:
driver: "json-file"
options:
max-size: "5m"
max-file: "10"
TZ=America/Chicago
F2B_LOG_TARGET=STDOUT
F2B_LOG_LEVEL=INFO
F2B_DB_PURGE_AGE=1d
F2B_MAX_RETRY=3
F2B_DEST_EMAIL=RECEIPIENT_EMAIL
F2B_SENDER=SENDER_EMAIL
F2B_ACTION=%(action_mwl)s <---- action_mwl - block + send email
F2B_IPTABLES_CHAIN=DOCKER-USER
SSMTP_HOST=smtp.sendgrid.net
SSMTP_PORT=587
SSMTP_USER=apikey
SSMTP_PASSWORD=YOUR_API_KEY
SSMTP_TLS=YES
# File:
# "./fail2ban-data/jail.d/jail.local"
# The DEFAULT allows a global definition of the options.
# They can be override in each jail afterwards.
[DEFAULT]
# Number of seconds that a host is banned.
bantime = 18000 # ban for 5 hours
# "ignoreip" can be an IP address, a CIDR mask or a DNS host.
# Fail2ban will not ban a host which matches an address in this list.
# Several addresses can be defined using space separator.
ignoreip = 127.0.0.1/8 73.176.154.35
# attempts must occur within the 10-minute
findtime = 600
# How many attempts can be made before a ban is imposed
maxretry = 3
[wplogin]
enabled = true
port = http,https
logpath = /container-logs/*/*-json.log
filter = wplogin
bantime = 600
maxretry = 3
[phpmyadmin]
enabled = true
port = http,https
logpath = /container-logs/*/*-json.log
filter = phpmyadmin
bantime = 3600
maxretry = 1
# File:
# ./fail2ban-data/filter.d/wplogin.conf
[Definition]
failregex = {"log":"<HOST> -.*phpmyadmin.*
ignoreregex =
# Sample service
# VIRTUAL_HOST is used by "jwilder/nginx-proxy" to redirect traffic to right docker container.
# LETSENCRYPT_HOST and LETSENCRYPT_EMAIL are used by "jrcs/letsencrypt-nginx-proxy-companion" to create SSL certificates.
version: '3'
services:
nginx:
image: nginx
container_name: my_super_service
restart: unless-stopped
environment:
VIRTUAL_HOST: my-domain.com
LETSENCRYPT_HOST: my-domain.com
LETSENCRYPT_EMAIL: no-reply@my-domain.com
networks:
default:
external:
name: edge
# File:
# ./fail2ban-data/filter.d/wplogin.conf
[Definition]
failregex = {"log":"<HOST> -.*POST.*wp-login.php.*
ignoreregex =
@alexpovel
Copy link

Thanks for the write-up, looking very good!

You might want to add your user to the docker group so you don't have to prefix sudo on every command:

$ sudo docker ...

becomes just

$ groups
... docker ...
$ docker ...

@daveschafer
Copy link

daveschafer commented Feb 10, 2021

I had the problem that fail2ban banned IPs and added a correct iptables rule, though the rule was not processed correctly - banned IPs came through anyway. It seemed like the rule was not in the correct order. The solution for me was to specifically configuring the chain DOCKER-USER to be used in the fail2ban jail like this:

[FILTERNAME]
enabled = true  
chain = DOCKER-USER

Source: fail2ban/fail2ban#2292

PS: thanks for this awesome guide which really helped me for my project!

@mayanez
Copy link

mayanez commented Jun 9, 2021

Incredibly helpful!

For me instead of defining F2B_ACTION and F2B_IPTABLES_CHAIN. I used the following in my applications jail.d (eg. app.jail)

action = iptables[name=app, port=https, protocol=tcp, chain=FORWARD]

@phatpaul
Copy link

Thanks for the inspiration, but this seems to have each jail process the logs for All docker containers on the system. That seems wasteful, and would lead to an infinite loop, since fail2ban docker will be processing it's own logs??

Also, in my case, one of my dockers had a different timezone configured, which seems to mess up all of the jails.

2021-08-10 21:43:33,649 fail2ban.filter         [1]: WARNING [nginx-http-auth] Simulate NOW in operation since found time has too large deviation None ~ 1628646213.6496086 +/- 60
2021-08-10 21:43:33,650 fail2ban.filter         [1]: WARNING [nginx-http-auth] Please check jail has possibly a timezone issue. Line with odd timestamp: {"log":"2021-08-10 21:43:33,641 fail2ban.filter         [1]: ERROR   Failed to process line: '{\"log\":\"Initializing files and folders...\\\\n\",\"stream\":\"stdout\",\"time\":\"2021-08-11T01:43:32.295184913Z\"}', caught exception: IndexError('string index out of range')\n","stream":"stdout","time":"2021-08-11T01:43:33.642316898Z"}

How can I configure the jails to only process the appropriate log file?

@mfittko
Copy link

mfittko commented Oct 6, 2021

Thanks for the inspiration, but this seems to have each jail process the logs for All docker containers on the system. That seems wasteful, and would lead to an infinite loop, since fail2ban docker will be processing it's own logs??

Logs are placed on the docker host under /var/lib/docker/containers/CONTAINER_ID/CONTAINER_ID-json.log so I guess the easiest fix for making fail2ban process its own logs is using a different logging driver (like local or syslog) for the fail2ban container (see https://docs.docker.com/config/containers/logging/configure/#supported-logging-drivers)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment