Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Block globally reported hack attempts using your local iptables firewall rules
#!/bin/bash
## Update fail2ban iptables with globally known attackers.
## Actually, runs 100% independently now, without needing fail2ban installed.
##
## /etc/cron.daily/sync-fail2ban
##
## Author: Marcos Kobylecki <fail2ban.globalBlackList@askmarcos.com>
## http://www.reddit.com/r/linux/comments/2nvzur/shared_blacklists_from_fail2ban/
## Quit if fail2ban is missing. Maybe this fake requirement can be skipped? YES.
#PROGRAM=/etc/init.d/fail2ban
#[ -x $PROGRAM ] || exit 0
datadir=/etc/fail2ban
[[ -d "$datadir" ]] || datadir=/tmp
## Get default settings of fail2ban (optional?)
[ -r /etc/default/fail2ban ] && . /etc/default/fail2ban
umask 000
blacklistf=$datadir/blacklist.blocklist.de.txt
mv -vf $blacklistf $blacklistf.last
badlisturls="http://antivirus.neu.edu.cn/ssh/lists/base_30days.txt http://lists.blocklist.de/lists/ssh.txt http://lists.blocklist.de/lists/bruteforcelogin.txt"
iptables -vN fail2ban-ssh # Create the chain if it doesn't exist. Harmless if it does.
# Grab list(s) at https://www.blocklist.de/en/export.html . Block.
echo "Adding new blocks:"
time curl -s http://lists.blocklist.de/lists/ssh.txt http://lists.blocklist.de/lists/bruteforcelogin.txt \
|sort -u \
|tee $blacklistf \
|grep -v '^#\|:' \
|while read IP; do iptables -I fail2ban-ssh 1 -s $IP -j DROP; done
# Which listings had been removed since last time? Unblock.
echo "Removing old blocks:"
if [[ -r $blacklistf.diff ]]; then
# comm is brittle, cannot use sort -rn
time comm -23 $blacklistf.last $blacklistf \
|tee $blacklistf.delisted \
|grep -v '^#\|:' \
|while read IP; do iptables -w -D fail2ban-ssh -s $IP -j DROP || iptables -wv -D fail2ban-ssh -s $IP -j LOGDROP; done
fi
# prepare for next time.
diff -wbay $blacklistf.last $blacklistf > $blacklistf.diff
# Saves a copy of current iptables rules, should you like to check them later.
(set -x; iptables -wnv -L --line-numbers; iptables -wnv -t nat -L --line-numbers) &> /tmp/iptables.fail2ban.log &
exit
# iptables v1.4.21: host/network `2a00:1210:fffe:145::1' not found
# So weed out IPv6, try |grep -v ':'
## http://ix.io/fpC
# Option: actionban
# Notes.: command executed when banning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: See jail.conf(5) man page
# Values: CMD
#
actionban = iptables -I fail2ban-<name> 1 -s <ip> -j <blocktype># Option: actionunban
# Notes.: command executed when unbanning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: See jail.conf(5) man page
# Values: CMD
#
actionunban = iptables -D fail2ban-<name> -s <ip> -j <blocktype>
@klepsydra

This comment has been minimized.

@klepsydra

This comment has been minimized.

Copy link
Owner Author

commented Dec 1, 2014

To install:
(as root)

curl -s https://gist.githubusercontent.com/klepsydra/ecf975984b32b1c8291a/raw > /etc/cron.daily/sync-fail2ban

chmod a+x /etc/cron.daily/sync-fail2ban

  • Optional. Initial run manually:

    time /etc/cron.daily/sync-fail2ban

    Tomorrow, check your /tmp/iptables.fail2ban.log file to see who's been blocked.
    The lists you get are stored locally for now at /etc/fail2ban/blacklist.*

@offetoffe

This comment has been minimized.

Copy link

commented Jan 13, 2015

Hi, get error an message, I believe it is on line 49 & 49 & 62: ... iptables -w .... What is this argument suppose to do? It gives me an error on Ubuntu, the argument is not valid. I am using iptables 1.4.12. Also, googling around, I cannot see that this argument is valid for any platform. Please advise how to correct this.

Edit: later versions of iptables have the this argument.. time to upgrade.
Edit2: upgraded to Ubuntu 14.x, everything is OK!

Thanks /Olof

@klepsydra

This comment has been minimized.

Copy link
Owner Author

commented Apr 9, 2015

@offetoffe
Yes, thanks for your findings re: earlier versions of iptables.
Feel free to remove the -w for installations where it's not supported.

The -w param to iptables was a later development I noticed helped, when elsewhere on the same system, there'd be another long-running iptables command (possibly due to -v or not using -n, so 1000's of DNS queries would have to be resolved first). In those cases, before the -w usage, iptables would error and fail rather than wait for its turn to run.

From the man page (Ubuntu 14.04 ):

   -w, --wait
          Wait  for  the xtables lock.  To prevent multiple instances of the program from running concurrently, an attempt will be made to obtain an exclusive lock at
          launch.  By default, the program will exit if the lock cannot be obtained.  This option will make the program wait until the exclusive lock can be obtained.
@ingber

This comment has been minimized.

Copy link

commented Apr 9, 2015

Is there a reason for using the default name for the Chain fail2ban-ssh? That is, this is a common chain in jail.local. Would it be better for any reason to rename this default to something like fail2ban-blacklist?

The script does not seem to enforce persistence across booting. I ran it once after a reboot, and it just complained a lot since the previous blacklist.blocklist.de* files were still there, but the new rules still were inserted OK.

Thanks.

Lester

@aenertia

This comment has been minimized.

Copy link

commented Apr 9, 2015

Hrm ; this doesn't support v6 ip's by the looks of it and should probably use ipset

@ingber

This comment has been minimized.

Copy link

commented Jul 28, 2015

ipv6 works fine: Just create a parallel file:
sync-fail2ban6.sh

iptables -> ip6tables
blacklist -> blacklist6
etc.

Use a source, e.g., http://lists.blocklist.de/lists/bruteforcelogin.txt
inside modify the grep lines:
|grep ':'
|grep -v '^#' \

This seems to work fine.

There are a lot of duplicates added on each run, so I modified
|while read IP; do /sbin/iptables -I fail2ban-sshBL 1 -s $IP -j DROP; done
to
|while read IP; do /etc/fail2ban/xiptables_check_add $IP ; done
which calls a script with
set IPTEST = /sbin/iptables -L fail2ban-sshBL -n | /bin/grep "$1" | sed -e '1s/^\(DROP\).*$/\1/'
if ( "$IPTEST" == "DROP" ) then
...
endif

@d--j

This comment has been minimized.

Copy link

commented Dec 9, 2015

Hi,

this is a version that works with IPv4 and IPv6 and does not overload your iptables chains by using ipset and one single iptables rule (and one ip6tables rule) for all the 40.000 something IP addresses in the all list of https://www.blocklist.de/en/export.html

This will run (when you comment out the sleep call) in about a second and activates the new block list atomically. It also does not need any state or diff files to work correctly.

#!/bin/bash

# updates the block list once daily. Make this file executable and place it in
#   /etc/cron.daily/sync-blocklist
#   
#  requires the ipset program, install it e.g. via
#  apt-get install ipset
# 
# inspired by https://gist.github.com/klepsydra/ecf975984b32b1c8291a
# but uses ipset because directly using iptables / ip6tables does not scale

PATH="/sbin:/bin:/usr/bin"

if test ! -x /sbin/ipset; then
  echo "ipset not installed"
  exit 1
fi

# sleep for up to 30 seconds to not overload blocklist.de on midnight
sleep $[ ( $RANDOM % 30 )  + 1 ]s

SET_TYPE="hash:ip"
SET_CONFIG="hashsize 16384 maxelem 131072"

# initialize the iptables integration if it is not already present
ipset -exist create blacklist-ip4 $SET_TYPE family inet $SET_CONFIG
ipset -exist create blacklist-ip6 $SET_TYPE family inet6 $SET_CONFIG
iptables -wn -L INPUT | fgrep -q 'match-set blacklist-ip4 src' || iptables -w -I INPUT -m set --match-set blacklist-ip4 src -j DROP
ip6tables -wn -L INPUT | fgrep -q 'match-set blacklist-ip6 src' || ip6tables -w -I INPUT -m set --match-set blacklist-ip6 src -j DROP

# create the new lists
ipset create new-blacklist-ip4 $SET_TYPE family inet $SET_CONFIG
ipset create new-blacklist-ip6 $SET_TYPE family inet6 $SET_CONFIG

# fill the new lists
#   this does the following:
#    1. get the file https://lists.blocklist.de/lists/all.txt
#    2. stream every line to the grep command that finds lines 
#       that only have a IPv4 or IPv6 address on them (actually something 
#       like 999.999.999.999 will match, too but we do not mind) to filter out 
#       comments and shell injection attacks
#    3. removes duplicate IP addresses
#    4. prefixes "add new-blacklist-ip6" or "add new-blacklist-ip4" to the line
#       depending on wheter there is a : in the line (IPv6 addresses always have one)
#    5. feed these to ipset in one single call
curl -s https://lists.blocklist.de/lists/all.txt \
  | grep -Pxe '((?:[0-9]{1,3}\.){3}[0-9]{1,3}|(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])))' \
  | sort -u \
  | awk '/:/{ print "add new-blacklist-ip6 " $0 } !/:/{ print "add new-blacklist-ip4 " $0 }' \
  | ipset restore

# swap the new and the old lists
ipset swap blacklist-ip4 new-blacklist-ip4
ipset swap blacklist-ip6 new-blacklist-ip6


# remove the old lists (they have the new lists names now)
ipset destroy new-blacklist-ip4
ipset destroy new-blacklist-ip6
@ingber

This comment has been minimized.

Copy link

commented Jul 2, 2016

Re: using ipset: On boot, this does not see create the new chains, so the rest of it does not work. I had to first use
/sbin/iptables -vN fail2ban-sshBL
/sbin/ip6tables -vN fail2ban-sshBL6
where I am using different names for my chains.

@joaomourato

This comment has been minimized.

Copy link

commented Jul 5, 2016

Hi,

I'm using your script because ipset doesn't work in OpenVZ but i get the following errors:

/usr/local/bin/sync-blocklist: line 85: syntax error near unexpected token newline' /usr/local/bin/sync-blocklist: line 85:actionban = iptables -I fail2ban- 1 -s -j # Option: actionunban'
'/etc/fail2ban/blacklist.txt' -> '/etc/fail2ban/blacklist.txt.last'
/usr/local/bin/sync-blocklist: line 32: iptables: command not found
Adding new blocks:
/usr/local/bin/sync-blocklist: line 41: iptables: command not found
/usr/local/bin/sync-blocklist: line 41: iptables: command not found

The last lines repeat for every ip added.
The rules are added though.

@ingber

This comment has been minimized.

Copy link

commented Jul 6, 2016

joaomourato:

In a login shell, check
which iptables
you should get
/sbin/iptables

Then, in your script, explicitly change
iptables -> /sbin/iptables

Do the same for ip6tables and ipset.

@ingber

This comment has been minimized.

Copy link

commented Jul 10, 2016

I wish the ipset method worked, but it just does not on my Ubuntu 14.04.04 x64 VPS.

@fwsl

This comment has been minimized.

Copy link

commented Dec 20, 2016

@d--j
Hi thanks for the code, I'm using your script, but sometimes I get some errors

/etc/cron.daily/sync_blocklist:
ipset v6.23: Set cannot be created: set with the same name already exists
ipset v6.23: Set cannot be created: set with the same name already exists
/etc/cron.daily/sync_blocklist: line 49: unexpected EOF while looking for matching `''
/etc/cron.daily/sync_blocklist: line 61: syntax error: unexpected end of file
run-parts: /etc/cron.daily/sync_blocklist exited with return code 2 

any ideas on how to fix it?

@d--j

This comment has been minimized.

Copy link

commented Mar 27, 2017

Hi @fwsl

sorry I haven't seen your question until now. I currently use the following version:

#!/bin/bash

# updates the block list once daily. Make this file executable and place it in
#   /etc/cron.daily/sync-blocklist
#
#  requires the ipset program, install it e.g. via
#  apt-get install ipset
#
# inspired by https://gist.github.com/klepsydra/ecf975984b32b1c8291a
# but uses ipset because directly using iptables / ip6tables does not scale

PATH="/sbin:/usr/sbin:/bin:/usr/bin"

if test ! -x /sbin/ipset && test ! -x /usr/sbin/ipset; then
  echo "ipset not installed"
  exit 1
fi

if test ! -x /usr/bin/sipcalc && test ! -x /bin/sipcalc; then
  echo "sipcalc not installed"
  exit 1
fi

# sleep for up to 30 seconds to not overload blocklist.de on midnight
sleep $(( ( RANDOM % 30 )  + 1 ))s

SET_TYPE="hash:ip"
SET_CONFIG="hashsize 16384 maxelem 131072"
IPTABLES_WAIT=""
if iptables --help | fgrep -q -- '--wait'; then
  IPTABLES_WAIT="-w" # use --wait when iptables supports it
fi

# initialize the iptables integration if it is not already present
ipset -exist create blacklist-ip4 $SET_TYPE family inet $SET_CONFIG
ipset -exist create blacklist-ip6 $SET_TYPE family inet6 $SET_CONFIG
iptables -n $IPTABLES_WAIT -L INPUT | fgrep -q 'match-set blacklist-ip4 src' || iptables $IPTABLES_WAIT -I INPUT -m set --match-set blacklist-ip4 src -j DROP
ip6tables -n $IPTABLES_WAIT -L INPUT | fgrep -q 'match-set blacklist-ip6 src' || ip6tables $IPTABLES_WAIT -I INPUT -m set --match-set blacklist-ip6 src -j DROP

# create the new lists
ipset create new-blacklist-ip4 $SET_TYPE family inet $SET_CONFIG
ipset create new-blacklist-ip6 $SET_TYPE family inet6 $SET_CONFIG

# fill the new lists
#   this does the following:
#    1. get the file https://lists.blocklist.de/lists/all.txt
#    2. stream every line to the grep command that finds lines
#       that only have a IPv4 or IPv6 address on them (actually something
#       like 999.999.999.999 will match, too but we do not mind) to filter out
#       comments and shell injection attacks
#    3. Canonicalize (v6) and validate (v4 and v6) IP adreses via sipcalc
#    4. removes duplicate IP addresses (with canonicalizing IPv6 addresses first)
#    5. Remove IP addresses that would block too much or unwanted targets
#    6. prefixes "add new-blacklist-ip6" or "add new-blacklist-ip4" to the line
#       depending on wheter there is a : in the line (IPv6 addresses always have one)
#    7. feed these to ipset in one single call
curl -s https://lists.blocklist.de/lists/all.txt \
  | grep -Pxe '((?:[0-9]{1,3}\.){3}[0-9]{1,3}|(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])))' \
  | xargs sipcalc \
  | awk -F '- ' '/Expanded Address/ { print $2 } /Host address\s+-/ { print $2 }' \
  | sort -u \
  | grep -ve '^0.0.0.0$' \
  | awk '/:/{ print "add new-blacklist-ip6 " $0 } !/:/{ print "add new-blacklist-ip4 " $0 }' \
  | ipset restore

# swap the new and the old lists – but only if there is something in the new lists
if [[ $(ipset list new-blacklist-ip4 | wc -l) -gt 10 ]]; then # header is 6 or 7, so list needs more than 3 entries
  ipset swap blacklist-ip4 new-blacklist-ip4
fi
if [[ $(ipset list new-blacklist-ip6 | wc -l) -gt 10 ]]; then
  ipset swap blacklist-ip6 new-blacklist-ip6
fi


# remove the old lists (they have the new lists names now)
ipset destroy new-blacklist-ip4
ipset destroy new-blacklist-ip6

It uses sipcalc (apt-get install sipcalc) to validate and canonicalize the IP-adresses before feeding them to ipset.

The code also does a very crude check if the new blacklist has any entries in it. If the new list is empty it does not replace it. Maybe that helps with your problem.

@stefanogatto

This comment has been minimized.

Copy link

commented May 24, 2017

Hi,
firts of all thank you both for your scripts.
I have a problem i dont understand, in debian8 the script made by d__j starts then exit with the error
ipset v6.23: Error in line 1: The set with the given name does not exist
what can I investigate to fix it?
BR

@RobbieTheK

This comment has been minimized.

Copy link

commented Dec 6, 2017

Is there a way to do this if iptables is disabled and just running firewalld on Fedora 26?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.