Block globally reported hack attempts using your local iptables firewall rules

sync-fail2ban.sh

#!/bin/bash

##  Update fail2ban iptables with globally known attackers.
##  Actually, runs 100% independently now, without needing fail2ban installed.
##
##  /etc/cron.daily/sync-fail2ban
##
## Author: Marcos Kobylecki <fail2ban.globalBlackList@askmarcos.com>
## http://www.reddit.com/r/linux/comments/2nvzur/shared_blacklists_from_fail2ban/


## Quit if fail2ban is missing.  Maybe this fake requirement can be skipped? YES.
#PROGRAM=/etc/init.d/fail2ban
#[ -x $PROGRAM ] || exit 0

datadir=/etc/fail2ban
[[ -d "$datadir" ]] || datadir=/tmp

## Get default settings of fail2ban (optional?)
[ -r /etc/default/fail2ban ] && . /etc/default/fail2ban

umask 000
blacklistf=$datadir/blacklist.blocklist.de.txt

mv -vf  $blacklistf  $blacklistf.last

badlisturls="http://antivirus.neu.edu.cn/ssh/lists/base_30days.txt http://lists.blocklist.de/lists/ssh.txt  http://lists.blocklist.de/lists/bruteforcelogin.txt"


 iptables -vN fail2ban-ssh   # Create the chain if it doesn't exist. Harmless if it does.
  
# Grab list(s) at https://www.blocklist.de/en/export.html .  Block.
echo "Adding new blocks:"
 time  curl -s http://lists.blocklist.de/lists/ssh.txt  http://lists.blocklist.de/lists/bruteforcelogin.txt \
  |sort -u \
  |tee $blacklistf \
  |grep -v '^#\|:' \
  |while read IP; do iptables -I fail2ban-ssh 1 -s $IP -j DROP; done 



# Which listings had been removed since last time?  Unblock.
echo "Removing old blocks:"
if [[ -r  $blacklistf.diff ]]; then
  #       comm  is brittle, cannot use sort -rn 
 time  comm -23 $blacklistf.last  $blacklistf \
   |tee $blacklistf.delisted \
   |grep -v '^#\|:' \
   |while read IP; do  iptables -w -D fail2ban-ssh -s $IP -j DROP || iptables -wv -D fail2ban-ssh -s $IP -j LOGDROP; done 

fi


# prepare for next time.
	diff -wbay $blacklistf.last $blacklistf  > $blacklistf.diff 





# Saves a copy of current iptables rules, should you like to check them later.
(set -x; iptables -wnv -L --line-numbers; iptables -wnv -t nat -L --line-numbers) &> /tmp/iptables.fail2ban.log &


exit 

# iptables v1.4.21: host/network `2a00:1210:fffe:145::1' not found
# So weed out IPv6, try |grep -v ':' 

## http://ix.io/fpC

 
# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionban = iptables -I fail2ban-<name> 1 -s <ip> -j <blocktype># Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionunban = iptables -D fail2ban-<name> -s <ip> -j <blocktype>

Described at http://serverfault.com/a/648087/104682
and https://plus.google.com/102676720348185803844/posts/XoDRGQU4i4A

Issue #874: fail2ban/fail2ban#874

To install:
(as root)

curl -s https://gist.githubusercontent.com/klepsydra/ecf975984b32b1c8291a/raw > /etc/cron.daily/sync-fail2ban

chmod a+x /etc/cron.daily/sync-fail2ban

• Optional. Initial run manually:

  time /etc/cron.daily/sync-fail2ban

  Tomorrow, check your /tmp/iptables.fail2ban.log file to see who's been blocked.
  The lists you get are stored locally for now at /etc/fail2ban/blacklist.*

Hi, get error an message, I believe it is on line 49 & 49 & 62: ... iptables -w .... What is this argument suppose to do? It gives me an error on Ubuntu, the argument is not valid. I am using iptables 1.4.12. Also, googling around, I cannot see that this argument is valid for any platform. Please advise how to correct this.

Edit: later versions of iptables have the this argument.. time to upgrade.
Edit2: upgraded to Ubuntu 14.x, everything is OK!

Thanks /Olof

@offetoffe
Yes, thanks for your findings re: earlier versions of iptables.
Feel free to remove the -w for installations where it's not supported.

The -w param to iptables was a later development I noticed helped, when elsewhere on the same system, there'd be another long-running iptables command (possibly due to -v or not using -n, so 1000's of DNS queries would have to be resolved first). In those cases, before the -w usage, iptables would error and fail rather than wait for its turn to run.

From the man page (Ubuntu 14.04 ):

   -w, --wait
          Wait  for  the xtables lock.  To prevent multiple instances of the program from running concurrently, an attempt will be made to obtain an exclusive lock at
          launch.  By default, the program will exit if the lock cannot be obtained.  This option will make the program wait until the exclusive lock can be obtained.

Is there a reason for using the default name for the Chain fail2ban-ssh? That is, this is a common chain in jail.local. Would it be better for any reason to rename this default to something like fail2ban-blacklist?

The script does not seem to enforce persistence across booting. I ran it once after a reboot, and it just complained a lot since the previous blacklist.blocklist.de* files were still there, but the new rules still were inserted OK.

Thanks.

Lester

Hrm ; this doesn't support v6 ip's by the looks of it and should probably use ipset

ipv6 works fine: Just create a parallel file:
sync-fail2ban6.sh

iptables -> ip6tables
blacklist -> blacklist6
etc.

Use a source, e.g., http://lists.blocklist.de/lists/bruteforcelogin.txt
inside modify the grep lines:
|grep ':'
|grep -v '^#' \

This seems to work fine.

There are a lot of duplicates added on each run, so I modified
|while read IP; do /sbin/iptables -I fail2ban-sshBL 1 -s $IP -j DROP; done
to
|while read IP; do /etc/fail2ban/xiptables_check_add $IP ; done
which calls a script with
set IPTEST = /sbin/iptables -L fail2ban-sshBL -n | /bin/grep "$1" | sed -e '1s/^\(DROP\).*$/\1/'
if ( "$IPTEST" == "DROP" ) then
...
endif

Hi,

this is a version that works with IPv4 and IPv6 and does not overload your iptables chains by using ipset and one single iptables rule (and one ip6tables rule) for all the 40.000 something IP addresses in the all list of https://www.blocklist.de/en/export.html

This will run (when you comment out the sleep call) in about a second and activates the new block list atomically. It also does not need any state or diff files to work correctly.

#!/bin/bash

# updates the block list once daily. Make this file executable and place it in
#   /etc/cron.daily/sync-blocklist
#   
#  requires the ipset program, install it e.g. via
#  apt-get install ipset
# 
# inspired by https://gist.github.com/klepsydra/ecf975984b32b1c8291a
# but uses ipset because directly using iptables / ip6tables does not scale

PATH="/sbin:/bin:/usr/bin"

if test ! -x /sbin/ipset; then
  echo "ipset not installed"
  exit 1
fi

# sleep for up to 30 seconds to not overload blocklist.de on midnight
sleep $[ ( $RANDOM % 30 )  + 1 ]s

SET_TYPE="hash:ip"
SET_CONFIG="hashsize 16384 maxelem 131072"

# initialize the iptables integration if it is not already present
ipset -exist create blacklist-ip4 $SET_TYPE family inet $SET_CONFIG
ipset -exist create blacklist-ip6 $SET_TYPE family inet6 $SET_CONFIG
iptables -wn -L INPUT | fgrep -q 'match-set blacklist-ip4 src' || iptables -w -I INPUT -m set --match-set blacklist-ip4 src -j DROP
ip6tables -wn -L INPUT | fgrep -q 'match-set blacklist-ip6 src' || ip6tables -w -I INPUT -m set --match-set blacklist-ip6 src -j DROP

# create the new lists
ipset create new-blacklist-ip4 $SET_TYPE family inet $SET_CONFIG
ipset create new-blacklist-ip6 $SET_TYPE family inet6 $SET_CONFIG

# fill the new lists
#   this does the following:
#    1. get the file https://lists.blocklist.de/lists/all.txt
#    2. stream every line to the grep command that finds lines 
#       that only have a IPv4 or IPv6 address on them (actually something 
#       like 999.999.999.999 will match, too but we do not mind) to filter out 
#       comments and shell injection attacks
#    3. removes duplicate IP addresses
#    4. prefixes "add new-blacklist-ip6" or "add new-blacklist-ip4" to the line
#       depending on wheter there is a : in the line (IPv6 addresses always have one)
#    5. feed these to ipset in one single call
curl -s https://lists.blocklist.de/lists/all.txt \
  | grep -Pxe '((?:[0-9]{1,3}\.){3}[0-9]{1,3}|(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])))' \
  | sort -u \
  | awk '/:/{ print "add new-blacklist-ip6 " $0 } !/:/{ print "add new-blacklist-ip4 " $0 }' \
  | ipset restore

# swap the new and the old lists
ipset swap blacklist-ip4 new-blacklist-ip4
ipset swap blacklist-ip6 new-blacklist-ip6


# remove the old lists (they have the new lists names now)
ipset destroy new-blacklist-ip4
ipset destroy new-blacklist-ip6

Re: using ipset: On boot, this does not see create the new chains, so the rest of it does not work. I had to first use
/sbin/iptables -vN fail2ban-sshBL
/sbin/ip6tables -vN fail2ban-sshBL6
where I am using different names for my chains.

Hi,

I'm using your script because ipset doesn't work in OpenVZ but i get the following errors:

/usr/local/bin/sync-blocklist: line 85: syntax error near unexpected token newline' /usr/local/bin/sync-blocklist: line 85:actionban = iptables -I fail2ban- 1 -s -j # Option: actionunban'
'/etc/fail2ban/blacklist.txt' -> '/etc/fail2ban/blacklist.txt.last'
/usr/local/bin/sync-blocklist: line 32: iptables: command not found
Adding new blocks:
/usr/local/bin/sync-blocklist: line 41: iptables: command not found
/usr/local/bin/sync-blocklist: line 41: iptables: command not found

The last lines repeat for every ip added.
The rules are added though.

joaomourato:

In a login shell, check
which iptables
you should get
/sbin/iptables

Then, in your script, explicitly change
iptables -> /sbin/iptables

Do the same for ip6tables and ipset.

I wish the ipset method worked, but it just does not on my Ubuntu 14.04.04 x64 VPS.

@d--j
Hi thanks for the code, I'm using your script, but sometimes I get some errors

/etc/cron.daily/sync_blocklist:
ipset v6.23: Set cannot be created: set with the same name already exists
ipset v6.23: Set cannot be created: set with the same name already exists
/etc/cron.daily/sync_blocklist: line 49: unexpected EOF while looking for matching `''
/etc/cron.daily/sync_blocklist: line 61: syntax error: unexpected end of file
run-parts: /etc/cron.daily/sync_blocklist exited with return code 2 

any ideas on how to fix it?

Hi @fwsl

sorry I haven't seen your question until now. I currently use the following version:

#!/bin/bash

# updates the block list once daily. Make this file executable and place it in
#   /etc/cron.daily/sync-blocklist
#
#  requires the ipset program, install it e.g. via
#  apt-get install ipset
#
# inspired by https://gist.github.com/klepsydra/ecf975984b32b1c8291a
# but uses ipset because directly using iptables / ip6tables does not scale

PATH="/sbin:/usr/sbin:/bin:/usr/bin"

if test ! -x /sbin/ipset && test ! -x /usr/sbin/ipset; then
  echo "ipset not installed"
  exit 1
fi

if test ! -x /usr/bin/sipcalc && test ! -x /bin/sipcalc; then
  echo "sipcalc not installed"
  exit 1
fi

# sleep for up to 30 seconds to not overload blocklist.de on midnight
sleep $(( ( RANDOM % 30 )  + 1 ))s

SET_TYPE="hash:ip"
SET_CONFIG="hashsize 16384 maxelem 131072"
IPTABLES_WAIT=""
if iptables --help | fgrep -q -- '--wait'; then
  IPTABLES_WAIT="-w" # use --wait when iptables supports it
fi

# initialize the iptables integration if it is not already present
ipset -exist create blacklist-ip4 $SET_TYPE family inet $SET_CONFIG
ipset -exist create blacklist-ip6 $SET_TYPE family inet6 $SET_CONFIG
iptables -n $IPTABLES_WAIT -L INPUT | fgrep -q 'match-set blacklist-ip4 src' || iptables $IPTABLES_WAIT -I INPUT -m set --match-set blacklist-ip4 src -j DROP
ip6tables -n $IPTABLES_WAIT -L INPUT | fgrep -q 'match-set blacklist-ip6 src' || ip6tables $IPTABLES_WAIT -I INPUT -m set --match-set blacklist-ip6 src -j DROP

# create the new lists
ipset create new-blacklist-ip4 $SET_TYPE family inet $SET_CONFIG
ipset create new-blacklist-ip6 $SET_TYPE family inet6 $SET_CONFIG

# fill the new lists
#   this does the following:
#    1. get the file https://lists.blocklist.de/lists/all.txt
#    2. stream every line to the grep command that finds lines
#       that only have a IPv4 or IPv6 address on them (actually something
#       like 999.999.999.999 will match, too but we do not mind) to filter out
#       comments and shell injection attacks
#    3. Canonicalize (v6) and validate (v4 and v6) IP adreses via sipcalc
#    4. removes duplicate IP addresses (with canonicalizing IPv6 addresses first)
#    5. Remove IP addresses that would block too much or unwanted targets
#    6. prefixes "add new-blacklist-ip6" or "add new-blacklist-ip4" to the line
#       depending on wheter there is a : in the line (IPv6 addresses always have one)
#    7. feed these to ipset in one single call
curl -s https://lists.blocklist.de/lists/all.txt \
  | grep -Pxe '((?:[0-9]{1,3}\.){3}[0-9]{1,3}|(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])))' \
  | xargs sipcalc \
  | awk -F '- ' '/Expanded Address/ { print $2 } /Host address\s+-/ { print $2 }' \
  | sort -u \
  | grep -ve '^0.0.0.0$' \
  | awk '/:/{ print "add new-blacklist-ip6 " $0 } !/:/{ print "add new-blacklist-ip4 " $0 }' \
  | ipset restore

# swap the new and the old lists – but only if there is something in the new lists
if [[ $(ipset list new-blacklist-ip4 | wc -l) -gt 10 ]]; then # header is 6 or 7, so list needs more than 3 entries
  ipset swap blacklist-ip4 new-blacklist-ip4
fi
if [[ $(ipset list new-blacklist-ip6 | wc -l) -gt 10 ]]; then
  ipset swap blacklist-ip6 new-blacklist-ip6
fi


# remove the old lists (they have the new lists names now)
ipset destroy new-blacklist-ip4
ipset destroy new-blacklist-ip6

It uses sipcalc (apt-get install sipcalc) to validate and canonicalize the IP-adresses before feeding them to ipset.

The code also does a very crude check if the new blacklist has any entries in it. If the new list is empty it does not replace it. Maybe that helps with your problem.

Hi,
firts of all thank you both for your scripts.
I have a problem i dont understand, in debian8 the script made by d__j starts then exit with the error
ipset v6.23: Error in line 1: The set with the given name does not exist
what can I investigate to fix it?
BR