Skip to content

Instantly share code, notes, and snippets.

@johntyree
Last active June 4, 2024 12:30
Show Gist options
  • Save johntyree/3331662 to your computer and use it in GitHub Desktop.
Save johntyree/3331662 to your computer and use it in GitHub Desktop.
Make one large blocklist from the bluetack lists on iblocklist.com
#!/usr/bin/env sh
# Download lists, unpack and filter, write to stdout
curl -s https://www.iblocklist.com/lists.php \
| sed -n "s/.*value='\(http:.*=bt_.*\)'.*/\1/p" \
| xargs wget -O - \
| gunzip \
| egrep -v '^#'
@divadsn
Copy link

divadsn commented Jun 5, 2021

Funny enough I got now a takedown request from iblocklist.com as I have a copy of the biglist on my mirror, even though those lists aren't even authored by them...

You are redistributing iblocklist.com data without permission.

Please take down this file;

Should I just ignore them?

@fonic
Copy link

fonic commented Jun 5, 2021

You are redistributing iblocklist.com data without permission.
Please take down this file;

Should I just ignore them?

No, you shouldn't. You should take it down. They asked you kindly not to redistribute data they provide without their permission. They have every right to do that.

That's why people like myself opt to create and provide scraper apps/scripts instead of distributing actual data.

@divadsn
Copy link

divadsn commented Jun 5, 2021

That's why people like myself opt to create and provide scraper apps/scripts instead of distributing actual data.

What's interesting, there are plenty of repos on GitHub containing same list files, yet those are up longer than my private mirror does. And the free lists they distribute can be sourced from other sites than iblocklist.com, so I'm more curious whether other owners have received such an email.

@fonic
Copy link

fonic commented Jun 5, 2021

What's interesting, there are plenty of repos on GitHub containing same list files, yet those are up longer than my private mirror does. And the free lists they distribute can be sourced from other sites than iblocklist.com, so I'm more curious whether other owners have received such an email.

Well, might just be coincidence, might be that the others received takedown notices too. You'll probably never know. Doesn't change the fact that they contacted you and expect you to act, though.

@divadsn
Copy link

divadsn commented Jun 6, 2021

@fonic I just wrote them to go away and pulled off one of their lists that my script was downloading, but it's still not ok how they communicate because instead of writing me directly what the problem is, I had to figure it out on my own. However, what annoys me the most is that they ignored my questions and threatened to reveal my name and answers to their emails on their site, very unprofessional.

@divadsn
Copy link

divadsn commented Jun 7, 2021

Screenshot_20210607_131259

Oh, how ironic...

@fonic
Copy link

fonic commented Jun 7, 2021

What site is that screenshot from?

@divadsn
Copy link

divadsn commented Jun 7, 2021

@sahsu
Copy link

sahsu commented Jul 1, 2021

just warm remind to all here's message from iblocklist to me they asked shutdown my github repo and here's why:

We don't mind sharing scripts that access our service.
We don't want the list data to be redistributed.

I hope all of you if redistributed now, shutdown it now, script are fine I think.

@divadsn
Copy link

divadsn commented Jul 1, 2021

@sahsu they can go somewhere else, they don't even own the data except their one free list.

They even threatened me without explaining why then other websites like firehol.org actually provide a copy to be re-distributed...

@mantrasiva
Copy link

Pretty soon http://john.bitsurge.net/public/biglist.p2p.gz is likely to disappear anyway because my host is discontinuing the service plan that made hosting this file affordable. Even now, with it being way past its prime, it's still quite a lot of network egress per month.

Even if my hosting costs weren't changing, as this script gets more and more out of date it becomes less and less responsible for me to keep letting people rely on it. Maybe if their clients start failing to download they will update their configs to point to something more recent. There are lots of alternatives.

@johntyree
hi
sadly now the link is down , is there a mirror?
thanks

@divadsn
Copy link

divadsn commented Jul 2, 2021

Pretty soon http://john.bitsurge.net/public/biglist.p2p.gz is likely to disappear anyway because my host is discontinuing the service plan that made hosting this file affordable. Even now, with it being way past its prime, it's still quite a lot of network egress per month.
Even if my hosting costs weren't changing, as this script gets more and more out of date it becomes less and less responsible for me to keep letting people rely on it. Maybe if their clients start failing to download they will update their configs to point to something more recent. There are lots of alternatives.

@johntyree
hi
sadly now the link is down , is there a mirror?
thanks

Use iplists.firehol.org or build the list yourself. I actually provide a iblocklist free list on my mirror: https://mirror.codebucket.de/transmission/

@johntyree
Copy link
Author

Dear everyone,

The file is no longer hosted at john.bitsurge.net and never will be again. It languished unmaintained for a very long time anyway. My suggestion to you is to run the script(s) yourself now and then. Anyone hosting iblocklists lists will eventually be asked to stop.

@YesTrustMe
Copy link

Thanks John for your efforts all these years. All the best.

@mantrasiva
Copy link

. It languished unmaintained for a very long time anyway. My suggestion to you is to run the script(s) yourself now and then. Anyone hosting iblocklists lists will eventually be asked to stop.

@johntyree
Hi
may i ask you a question ?
may i know which filters have you used in the past to build your biglist.p2p.gz ?
thanks

@mrglutton
Copy link

Dear everyone,

The file is no longer hosted at john.bitsurge.net and never will be again. It languished unmaintained for a very long time anyway. My suggestion to you is to run the script(s) yourself now and then. Anyone hosting iblocklists lists will eventually be asked to stop.

Thanks.

@YesTrustMe
Copy link

may i ask you a question ?

you may not. john has left.

@YesTrustMe
Copy link

This thread should be locked.

@zfkdnop
Copy link

zfkdnop commented Aug 1, 2021

curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" -s https://www.iblocklist.com/lists.php \
 | sed -n "s/.*value='\(http:.*\)'.*/\1/p" \
 | xargs wget -O - \
 | gunzip \
 | egrep -v '^#'

@johntyree
Copy link
Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment