Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@johntyree

johntyree/getBlockLists.sh

Last active March 9, 2024 12:32
Show Gist options
  • Save johntyree/3331662 to your computer and use it in GitHub Desktop.
Save johntyree/3331662 to your computer and use it in GitHub Desktop.
Download ZIP
Make one large blocklist from the bluetack lists on iblocklist.com
Raw
getBlockLists.sh
#!/usr/bin/env sh
# Download lists, unpack and filter, write to stdout
curl -s https://www.iblocklist.com/lists.php \
| sed -n "s/.*value='\(http:.*=bt_.*\)'.*/\1/p" \
| xargs wget -O - \
| gunzip \
| egrep -v '^#'
@YesTrustMe
Copy link

YesTrustMe commented Jun 16, 2020 •
edited

@Koeroesi86 you're right, they started checking at some point. It's an easy fix and I've done so, but this is a good example of why people should be looking for alternatives. If it breaks again, people could end up downloading an empty list and not realize it.<

Besides a VPN and a subscription to a service like iblocklist what do you recommend? Your list has been awesome and while it's not fool proof it's often better than nothing. If anything it weeds out a lot of crap.

@YesTrustMe
Copy link

i'm curious what people still using this think it's preventing? let's assume it's > 0% effective at going "unnoticed" by entity's collecting connected peer ips, but obviously fall short of 100% effectiveness. is it that you're still reducing visibility that you care about blocklists? i mean, if you really care about reduced visibility, you'll likely use a vpn, even a free one

i used a blocklist for years before i thought "..wait, what am i thinking/doing"

People use it to supplement their VPN. Many have received a copyright infringement email while using peer block lists.

@virtuarytours
Copy link

Hi all - first I must say I agree with fonic about etiquette on here. We all have stepped on people's toes before and the point I'm trying to make is that many contributeers here are providing an awesome service for free (iblocklist.com is no longer free, btw) and you don't push things upon them like asking them to "give" you more stuff on top of the already-generous packages of great stuff.

Now, WHAT is the big whoop here lol? Transmission and @fonic 's scripts and some of the other gz files on here, as far as all that--if you already have a program that accepts lists (peerBlock, Transmish, Adguard, w.e. etc.,) and you are here on Github. that means you at least have "some" tech level, so why can't you figure out how to add more Domains or addresses rather than tell people that have created something to create something more for you, or why are you pointing out what's wrong with it all?
Find lists. Find how to Export. Save. Paste new ones. Import. Done.

And as for all the salivating users (millennials) here asking @fonic to work for you because ''his lists are out-dated", as I previously stated, add more new and current addresses, and if you don't know how to figure that out, just log out, take open palm, thrust at face, and smash your facial structure lol.

Oh, and where are you going to find list for free that are current up to the minute, you say???

https://www.reddit.com/r/pfBlockerNG/comments/ed702i/dbloisdnl_internets_1_domain_blocklist/

Ok, I kid. I kid. But for real -- -- Read the ^ url. Digest. Copy. Paste. Heed the warnings. You'll be just fine.

Your welcome, is what people that help and give stuff to you say-- which means....... be nice, and enjoy.

@divadsn
Copy link

divadsn commented Jun 5, 2021 •
edited

Funny enough I got now a takedown request from iblocklist.com as I have a copy of the biglist on my mirror, even though those lists aren't even authored by them...

You are redistributing iblocklist.com data without permission.

Please take down this file;

Should I just ignore them?

@fonic
Copy link

fonic commented Jun 5, 2021 

You are redistributing iblocklist.com data without permission.
Please take down this file;

Should I just ignore them?

No, you shouldn't. You should take it down. They asked you kindly not to redistribute data they provide without their permission. They have every right to do that.

That's why people like myself opt to create and provide scraper apps/scripts instead of distributing actual data.

@divadsn
Copy link

divadsn commented Jun 5, 2021 •
edited

That's why people like myself opt to create and provide scraper apps/scripts instead of distributing actual data.

What's interesting, there are plenty of repos on GitHub containing same list files, yet those are up longer than my private mirror does. And the free lists they distribute can be sourced from other sites than iblocklist.com, so I'm more curious whether other owners have received such an email.

@fonic
Copy link

fonic commented Jun 5, 2021 •
edited

What's interesting, there are plenty of repos on GitHub containing same list files, yet those are up longer than my private mirror does. And the free lists they distribute can be sourced from other sites than iblocklist.com, so I'm more curious whether other owners have received such an email.

Well, might just be coincidence, might be that the others received takedown notices too. You'll probably never know. Doesn't change the fact that they contacted you and expect you to act, though.

@divadsn
Copy link

divadsn commented Jun 6, 2021 •
edited

@fonic I just wrote them to go away and pulled off one of their lists that my script was downloading, but it's still not ok how they communicate because instead of writing me directly what the problem is, I had to figure it out on my own. However, what annoys me the most is that they ignored my questions and threatened to reveal my name and answers to their emails on their site, very unprofessional.

@divadsn
Copy link

divadsn commented Jun 7, 2021

Screenshot_20210607_131259

Oh, how ironic...

@fonic
Copy link

fonic commented Jun 7, 2021

What site is that screenshot from?

@divadsn
Copy link

divadsn commented Jun 7, 2021

From https://iplists.firehol.org

@sahsu
Copy link

sahsu commented Jul 1, 2021

just warm remind to all here's message from iblocklist to me they asked shutdown my github repo and here's why:

We don't mind sharing scripts that access our service.
We don't want the list data to be redistributed.

I hope all of you if redistributed now, shutdown it now, script are fine I think.

@divadsn
Copy link

divadsn commented Jul 1, 2021 •
edited

@sahsu they can go somewhere else, they don't even own the data except their one free list.

They even threatened me without explaining why then other websites like firehol.org actually provide a copy to be re-distributed...

@mantrasiva
Copy link

Pretty soon http://john.bitsurge.net/public/biglist.p2p.gz is likely to disappear anyway because my host is discontinuing the service plan that made hosting this file affordable. Even now, with it being way past its prime, it's still quite a lot of network egress per month.

Even if my hosting costs weren't changing, as this script gets more and more out of date it becomes less and less responsible for me to keep letting people rely on it. Maybe if their clients start failing to download they will update their configs to point to something more recent. There are lots of alternatives.

@johntyree
hi
sadly now the link is down , is there a mirror?
thanks

@divadsn
Copy link

divadsn commented Jul 2, 2021

Pretty soon http://john.bitsurge.net/public/biglist.p2p.gz is likely to disappear anyway because my host is discontinuing the service plan that made hosting this file affordable. Even now, with it being way past its prime, it's still quite a lot of network egress per month.
Even if my hosting costs weren't changing, as this script gets more and more out of date it becomes less and less responsible for me to keep letting people rely on it. Maybe if their clients start failing to download they will update their configs to point to something more recent. There are lots of alternatives.

@johntyree
hi
sadly now the link is down , is there a mirror?
thanks

Use iplists.firehol.org or build the list yourself. I actually provide a iblocklist free list on my mirror: https://mirror.codebucket.de/transmission/

@johntyree
Copy link
Author

Dear everyone,

The file is no longer hosted at john.bitsurge.net and never will be again. It languished unmaintained for a very long time anyway. My suggestion to you is to run the script(s) yourself now and then. Anyone hosting iblocklists lists will eventually be asked to stop.

@YesTrustMe
Copy link

Thanks John for your efforts all these years. All the best.

@mantrasiva
Copy link

. It languished unmaintained for a very long time anyway. My suggestion to you is to run the script(s) yourself now and then. Anyone hosting iblocklists lists will eventually be asked to stop.

@johntyree
Hi
may i ask you a question ?
may i know which filters have you used in the past to build your biglist.p2p.gz ?
thanks

@mrglutton
Copy link

Dear everyone,

The file is no longer hosted at john.bitsurge.net and never will be again. It languished unmaintained for a very long time anyway. My suggestion to you is to run the script(s) yourself now and then. Anyone hosting iblocklists lists will eventually be asked to stop.

Thanks.

@YesTrustMe
Copy link

may i ask you a question ?

you may not. john has left.

@YesTrustMe
Copy link

This thread should be locked.

@zfkdnop
Copy link

zfkdnop commented Aug 1, 2021 

curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" -s https://www.iblocklist.com/lists.php \
 | sed -n "s/.*value='\(http:.*\)'.*/\1/p" \
 | xargs wget -O - \
 | gunzip \
 | egrep -v '^#'

@johntyree
Copy link
Author

@YesTrustMe 🥂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment