Skip to content
Create a gist now

Instantly share code, notes, and snippets.

Make one large blocklist from the bluetack lists on iblocklist.com
#!/usr/bin/env sh
# Download lists, unpack and filter, write to stdout
curl -s https://www.iblocklist.com/lists.php \
| sed -n "s/.*value='\(http:.*=bt_.*\)'.*/\1/p" \
| xargs wget -O - \
| gunzip \
| egrep -v '^#'
@studgeek

This can easily be added to cron also:
0 3 * * 0 curl -s http://www.iblocklist.com/lists.php | sed -n "s/.*value='\(http:.*=bt_.*\)'.*/\1/p" | xargs wget -O - | gunzip | egrep -v '^#' > ~/Library/Application\ Support/Transmission/blocklists/generated.txt

@ArtemGordinsky

Hi! I'm getting these errors trying to execute from the Terminal:
"xargs: wget: No such file or directory
gzip: stdin: unexpected end of file"

Am I doing anything wrong? Thanks!

@krono
krono commented Oct 11, 2012

replace wget -O - by curl in the script.

@panosmdma

Hi and thanks for script. I run from terminal (i have replace wget -0 - with curl) and all is:
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
And then error:
gzip: stdin: unexpected end of file
Thanks!

@jamesstout

Replace wget -O - with curl -Ls

@ArtemGordinsky

Hi @jamesstout,
Would you please post the entire terminal command with your correction?
When I run:

sudo 0 3 * * 0 curl -s http://www.iblocklist.com/lists.php | sed -n "s/.*value='\(http:.*=bt_.*\)'.*/\1/p" | xargs curl -Ls | gunzip | egrep -v    '^#' > ~/Library/Application\ Support/Transmission/blocklists/generated.txt

I get:

-bash: 0: command not found

@prehensilecode

The "0 3 * * 0 ..." is a crontab entry. Just type "curl -s ...."

@tlongren

@ArtemGordinsky you're getting -bash: 0: command not found because 0 isn't a command. It's a crontab entry, as @prehensilecode mentioned.

@fortran01

Updated for Mac:

curl -s https://www.iblocklist.com/lists.php | sed -n "s/.*value='\(http:.*=bt_.*\)'.*/\1/p" | sed "s/\&/\&/g" | sed "s/http/\"http/g" | sed "s/gz/gz\"/g" | xargs curl -L | gunzip | egrep -v '^#' > ~/Library/Application\ Support/Transmission/blocklists/generated.txt.bin
@kilolima

The blocklist URL has changed to HTTPS. The updated url on line 1 is https://www.iblocklist.com/lists.php (as above but it wasn't updated in the download). Thanks for this great script!

@GithubIsAgeist

iblocklist.com is bigoted hate website.

@johntyree
Owner

@kilolima updated.

@hedgehoggski

Hi folks. I've analysed the code for @fortran01's mac version (thanks v. much @fortran01)
and posted a plain english translation of what each bit does below.
Hope it's accurate but if I have made any errors please correct me! :-)
Hope this helps people tweak the code to do other similar stuff

For more clarity as to what's going on, I've put each piped segment of the command on seperate lines,
but remember it's all one big line.

 curl -s https://www.iblocklist.com/lists.php
 | sed -n "s/.*value='\(http:.*=bt_.*\)'.*/\1/p"
 | sed "s/\&/\&/g"
 | sed "s/http/\"http/g"
 | sed "s/gz/gz\"/g"
 | xargs curl -L
 | gunzip
 | egrep -v '^#' > ~/Library/Application\ Support/Transmission/blocklists/generated.txt.bin

Plain english Explanation of each bit of the command does:

grab the webpage "https://www.iblocklist.com/lists.php" in silent mode (no progress bar or error messages)

search each line of this webpage, looking for lines containing text of the form
(anything)value=http:(anything)=bt_(anything)
chop out and dump the first bit ( (anything)value= ) from each of the lines you find

in the resultant lines, change all occurrences of &amp to &

in the resultant lines change all occurrences of the string http to "http

in the resultant lines, change all occurrences of the string gz to gz"

feed the resultant lines one by one to the curl command (-L means curl will automatically redo the grab if server says any file resource has moved)

feed each file downloaded by curl to gunzip program (uncompress it)

write only the lines from each file that don't start with a # (i.e. that are not comments) into the file
"~/Library/Application Support/Transmission/blocklists/generated.txt.bin"

All this ultimately results in the following lines being fed one by one by xargs to the curl command:

"http://list.iblocklist.com/?list=bt_level1&fileformat=p2p&archiveformat=gz"
"http://list.iblocklist.com/?list=bt_level2&fileformat=p2p&archiveformat=gz"
"http://list.iblocklist.com/?list=bt_level3&fileformat=p2p&archiveformat=gz"
"http://list.iblocklist.com/?list=bt_edu&fileformat=p2p&archiveformat=gz"
"http://list.iblocklist.com/?list=bt_rangetest&fileformat=p2p&archiveformat=gz"
"http://list.iblocklist.com/?list=bt_bogon&fileformat=p2p&archiveformat=gz"
"http://list.iblocklist.com/?list=bt_ads&fileformat=p2p&archiveformat=gz"
"http://list.iblocklist.com/?list=bt_spyware&fileformat=p2p&archiveformat=gz"
"http://list.iblocklist.com/?list=bt_proxy&fileformat=p2p&archiveformat=gz"
"http://list.iblocklist.com/?list=bt_templist&fileformat=p2p&archiveformat=gz"
"http://list.iblocklist.com/?list=bt_microsoft&fileformat=p2p&archiveformat=gz"
"http://list.iblocklist.com/?list=bt_spider&fileformat=p2p&archiveformat=gz"
"http://list.iblocklist.com/?list=bt_hijacked&fileformat=p2p&archiveformat=gz"
"http://list.iblocklist.com/?list=bt_dshield&fileformat=p2p&archiveformat=gz"

This should help clarify exactly what pattern is being searched for by sed and what the sed filtering actually does
e.g.
"&amp" becomes "&"
inverted commas placed before each http and after each gz

You could of course feed these lines manually to curl if you just want to grab individual zipped blocklists etc.

If doing this, don't forget to filter out the comment lines with egrep and write to a .bin file for transmission :-)

@ronnicek

aww.. they changed it again :/

@Piezoid
Piezoid commented Jun 4, 2014

I've just wrote a simple Haskell program to sort and merge IP ranges (overlapping and adjacent IP ranges are fused, empty or commented lines are removed) : https://gist.github.com/Piezoid/ee43be6e5eebd6aa9bac

Use stdin and stdout with gz format :
./fuseblkl < inList.p2p.gz > outList.p2p.gz

@BBcan177

I wrote a script that downloads 50 different Blocklists and performs Duplication checks and other Repeated Offender Queries to compact the IP Reputation from a Threat Source of over 700,000 IPs to approx 2-300,000 IPs. Its primarily for pfSense Firewall Blocking but can be adapted for other needs. Any comments appreciated via email.

https://gist.github.com/BBcan17/67e8c456cb399fbe02ee

@rwilhelm

Here's another variant: it downloads only lists with a rating of >= 4. You could remove grep -A1 'star_[45]' to get all lists.

curl -sL 'https://www.iblocklist.com/lists.php' | egrep -A1 'star_[45]' | egrep -o '[a-z]{20}' | sort -u | while read -r blocklist; do curl -sL "http://list.iblocklist.com/?list=${blocklist}&fileformat=p2p&archiveformat=gz" | gunzip -q > ~/Library/Application\ Support/Transmission/blocklists/$blocklist; done

https://trac.transmissionbt.com/wiki/Blocklists:

When transmission starts, it scans this directory for files not ending in ".bin" and tries to parse them.

@ip2k
ip2k commented Jul 30, 2014

[ Edited 08/17 to no longer use a temp file]

I re-wrote this to work with their current site and made the resulting daily-updated list available at http://ip2k.com/list.gz

(note that this requires hxwls which should be in the 'html-xml-utils' package or similar in your distro)


#!/bin/bash

for url in $(curl -s https://www.iblocklist.com/lists.php \
| hxwls \
| grep -v png \
| grep 'list=' \
| sed 's|/list.php?list=||g' \
| sed 's|^|http://list.iblocklist.com/?list=|g' \
| sed 's|$|\&fileformat=p2p\&archiveformat=gz|g'); do wget --no-verbose "${url}" -O - | gunzip |egrep -v '^#' >> list; done

gzip list
echo "DONE"
ls -lah list.gz

@dkavraal
dkavraal commented Aug 2, 2014

This one is for uTorrent on MacOSX, if anyone needs:

#!/bin/bash
DIR="~/Library/Application Support/uTorrent"
LIST_COUNT=$(wget -q -O - http://www.iblocklist.com/lists.php | sed -n "s/.*value='\(http:.*list=.*\)'.*/\1/p" | wc -l )
CURRENT_LIST_ID=0
mv ipfilter.dat ipfilter.`date +%Y%m%d_%s`.dat 2>/dev/null
>"${DIR}/ipfilter.dat"
STARS_PRINTED=0
echo -n -e "..."
wget -q -O - http://www.iblocklist.com/lists.php | sed -n "s/.*value='\(http:.*list=.*\)'.*/\1/p" | while read LIST_URL; do
    CURRENT_LIST_ID=$[${CURRENT_LIST_ID}+1]
    STARS=$[20*${CURRENT_LIST_ID}/${LIST_COUNT}]
    STARS_LEFT=$[${STARS}-${STARS_PRINTED}]
    STARS_PRINTED=$STARS
    for ((;STARS_LEFT>0; STARS_LEFT--)); do echo -n -e "*"; done
    URL=$LIST_URL sh -c 'wget -q -O - "$URL" | gunzip -c | egrep -v "^#" | cut -d":" -f2 | egrep -v "^$" ' >> "${DIR}/ipfilter.dat" 2>/dev/null
done
echo
@enricobacis

@ip2k nice script, you just have one typo:

for url in $(cat bl); do

should be

for url in $(cat bls); do

@ip2k
ip2k commented Aug 17, 2014

@enricobacis thanks for pointing that out :) I fixed the script to not use a temp file at all now, and a daily-updated list is at http://ip2k.com/list.gz

@sudhirkhanger

@ip2k Just found out about the this list. Once I add the list you generated to the Transmission, it will get automatically updated as frequently are yours. Meaning I will not have to do anything except add the link to Transmission. Is that right?

@philippemorin123

ip2k, is your list still maintained?

@cinus
cinus commented Feb 3, 2015

Hey, looks like a script pasting party!
https://github.com/cinus/iblocklist/blob/master/get_lists.sh

@negativeions

When I paste http://ip2k.com/list.gz into transmission I only get 389,000+ IPs.. I thought it was supposed to be something like 2 billion. Is something wring with this list? or what?

@LinixLinux

@negativeions There are 389,000 ranges, or, as Transmission calls it, rules. 389,000 ranges = two billion individual IPs ;).

|Source:|
https://forum.transmissionbt.com/viewtopic.php?t=16987

@koesherbacon

Do you have any idea why all torrents from LinuxTracker.org (IP: 66.135.33.31, found from linuxtracker.org.ipaddress.com) are being blocked by your list? I downloaded the list and opened it in Notepad++; however, I could not find either the URL and IP address when I searched for them, so I'm not certain why they would be blocked.

Any ideas you might have on how to fix this issue?

@ip2k
ip2k commented Jul 2, 2015

http://ip2k.com/list.gz is updated daily via crontab; yes.

@Luen
Luen commented Aug 3, 2015

@koesherbacon iblocklist.com is in charge of which ip's are blocked. It might be that the ip address is within an ip range that is blocked.

@johntyree
Owner

@ip2k I'm getting literally thousands of hits a day on this and haven't looked at it in at least a year. I'm considering just 301 redirecting to yours. Are you OK with that?

@madhouserevival

@ip2k, so do I open it with a txt editor and copy into the ipfilter.dat file? Looks like Dave's Ipfilter hasn't been updated in almost a month. Miss the normal ipfilter updater which no longer works.

@decabyte

@freed00m is not DEAD ... it is just returning an empty list due to some URL schema changes on the original website.

To fix it just run the ip2k script changing:

| sed 's|/list.php?list=||g' \

with:

| sed 's|/list?list=||g' \

and you will get a long blocklist. :)

@petehalatsis

So, I have some other OS that I will not name. I was just downloading the list.gz file, unzipping, and importing to my bittorrent client. I assume there is a better way to do this? Also, just downloading the list right now results in an empty file... Please assist!

@sandstrom

For anyone in search of a blocklist for Transmission etc. (ending up here via Google), here is a list: http://www.wael.name/wael.list.p2p.gz

(note that it's not related to the script in this gist)

@timfam
timfam commented Jan 29, 2016

@sandstrom How often is your list updated?

@jult
jult commented Feb 14, 2016

Mine's updated nightly: https://gist.github.com/jult/e76c628899bd5aa3c33a
I use https://jult.net/bloc.txt.gz in Tixati and it shows me quite a lot of matches so it's working really well!

@dementio
dementio commented Mar 2, 2016

@jult I'm trying to modify yours to also use the I-Blocklist lists and add the &username=USERNAME&pin=PIN without having to put a separate line for each list but keep coming up short on how to achieve this. Any ideas?

@jult
jult commented Mar 8, 2016

@dementio How many lists are there for your login? You need to auth yourself, for every list, I don't see how that would be easier in one go. Why bother with shortening that?

@Zulith
Zulith commented Mar 21, 2016

Thank you jult, I've plugged your blocklist url into transmission and it has nearly 115k entries (for the tixati list), is this the one you would recommend for most users? some of your other lists have nearly 500k entries according to transmission, but I see that they have way more cruft in them. Are they both equally as effective?

@rtinikan

Hi all, @jult Thanks for the script

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.