Pi-hole Youtube ad blocking

filter-youtube-domains.sh

#!/bin/sh
# This script will fetch the Googlevideo ad domains and append them to the Pi-hole block list.
# Run this script daily with a cron job (don't forget to chmod +x)
# More info here: https://discourse.pi-hole.net/t/how-do-i-block-ads-on-youtube/253/136

# File to store the YT ad domains
FILE=/etc/pihole/youtube.hosts

# Fetch the list of domains, remove the ip's and save them
curl 'https://api.hackertarget.com/hostsearch/?q=googlevideo.com' \
| awk -F, 'NR>1{print $1}' \
| grep -vE "redirector|manifest" > $FILE

# Replace r*.sn*.googlevideo.com URLs to r*---sn-*.googlevideo.com
# and add those to the list too
cat $FILE | sed -r 's/(^r[[:digit:]]+)(\.)(sn)/\1---\3-/' >> $FILE

# Scan log file for previously accessed domains
grep '^r.*googlevideo\.com' /var/log/pihole*.log \
| awk '{print $8}' \
| grep -vE "redirector|manifest" \
| sort | uniq >> $FILE

# Add to Pi-hole adlists if it's not there already
if ! grep $FILE < /etc/pihole/adlists.list; then echo "file://$FILE" >> /etc/pihole/adlists.list; fi;

To get more subdomains from Wolfram, add this after line 16:
curl 'http://api.wolframalpha.com/v2/query?input=googlevideo.com&appid=$APPID$&format=plaintext&podstate=WebSiteStatisticsPod:InternetData__Subdomains&podstate=WebSiteStatisticsPod:InternetData__Subdomains_More'
| grep -Eo "r\d+---sn-.+.googlevideo.com" >> $FILE

Replace $APPID$ with your AppId from Wolfram https://developer.wolframalpha.com/portal/signin.html

Guys, I have a problem here.
My drive only has 20 gb storage...., the youtube.hosts file eats up all space.
The other day I had to do a trick to find the source of the problem
which was youtube.hosts
What is the solution for that problem? At the pihole forum other people claim they have this situation to.
But the pihole group responds logically 'Sorry, youtube.hosts is not our department...'

Besides that, all respect for this interesting script. Nowadays, I still find youtube adds.
Is there new development to handle 2019?

Thanks for all your effort.

Hey @leeuwtjex,
I had the same problem and fixed it by deleting the old youtube.hosts and editing the sed line to this:
sed -i $FILE -re 's/(^r[[:digit:]]+)(\.)(sn)/\1---\3-/'
I hope that helps you!

Also had to edit the regex at the wolfram grep since it didn't return anything (according to the website https://regexr.com/ your grep should work though). For me grep -Eo "r\d+---sn-.+.googlevideo.com" didn't find e.g. r1---sn-oguelned.googlevideo.com so I edited the grep to grep -Eo "r.+---sn-.+googlevideo.com" which works.

Thanks for sharing the script, it is working perfectly fine for over a year now.

Thanks Kapujino, I found this mod exactly like you said in another post to in a reddit post somewhere.
It works! I still have 19 gb free now!.

My I ask what does that wolfram API call do? I'm trying to make this happen without registering at wolfram. Thanks.

You need the $APPID$ for Wolfram to work

Replace $APPID$ with your AppId from Wolfram https://developer.wolframalpha.com/portal/signin.html

@ericknorr, Thanks mate, I appreciate. But that was not what I'm asking, I know how to use APIs. My questions was, rather, what is wolfram alpha is doing here. You know, the first API call gets the googlevideo subdomains from hackertarget, then with some awk and sed magic it transforms them to these ---sn like format but then, what is the purpose of call to wolfram services?

@Pouya-moh WA is able to retrieve more domains than hackertarget. But don't bother anymore, as this script is deprecated because YT changed how they serve ads. More info in this thread

This is a long going thread, however the solution (script) works well.
I applied some changes:
#!/bin/sh
# More info here: https:#discourse.pi-hole.net/t/how-do-i-block-ads-on-youtube/253/136
# File to store the YT ad domains
FILE=/etc/pihole/youtube.hosts

# Fetch the list of domains, remove the ip's and save them
curl 'https:#api.hackertarget.com/hostsearch/?q=googlevideo.com' \
| awk -F, 'NR>1{print $1}' \
| grep -vE "redirector|manifest" > $FILE

# Replace r*.sn*.googlevideo.com URLs to r*---sn-*.googlevideo.com
# and add those to the list too

sed -i $FILE -re 's/(^r[[:digit:]]+)(\.)(sn)/\1---\3-/'

# Scan log file for previously accessed domains
grep '^r.*googlevideo\.com' /var/log/pihole*.log \
| awk '{print $8}' \
| grep -vE "redirector|manifest" \
| sort | uniq >> $FILE

# Update Gravity
pihole -g

I added the list to the list group using the URL: file:///etc/pihole/youtube.hosts
I then edit my cron jobs
sudo crontab -e
adding
*/10 * * * * /etc/pihole/youtube-adblock.sh
This executes the script every 10 minutes, replacing the file (youtube.hosts) with the latest domains.
Works well - I only get Ad's one time, after this they are blocked.

@dev-2-4-h would you mind doing a step by step guide for those who are not particularly familiar with bash scripting? I already have my pi hole in place but I still see the same youtube ads as usual.

I get this error when I try to run the CURL
curl: (6) Could not resolve host: https

@dev-2-4-h would you mind doing a step by step guide for those who are not particularly familiar with bash scripting? I already have my pi hole in place but I still see the same youtube ads as usual.

I get this error when I try to run the CURL
curl: (6) Could not resolve host: https

I just changed "curl 'https:#api.hackertarget.com/hostsearch/?q=googlevideo.com' " to "curl 'https://api.hackertarget.com/hostsearch/?q=googlevideo.com' "

Im running pih-hole v5.2.2 and I seem to be missing something as the AdList is not updating with the YouTube hosts (domains).

I have the script in my /etc/pihole folder as per above revised by @dev-2-4-h last post here

It Successfully creates a youtube.hosts file in the /etc/pihole folder
It Successfull call pihole -g and the update appears to run with success into GravityDB, but no new domains are added.

Looking at the script above I see no obvious way the youtube.hosts file is added to Pi-Hole automatically. Also checked the gravity.sh script Pi-Hole uses when executes the pihole -g command and that doesnt appear to look for .hosts files on updating GravityDB.

Therefore what am I missing, how does pihole -g suppose to add the contents of the auto-generated youtube.hosts file to Pi-Hole?

Cleaned up the script a little bit by replacing the curl/awk/grep/sed sequence with just curl and sed.

#!/bin/sh
# More info here: https:#discourse.pi-hole.net/t/how-do-i-block-ads-on-youtube/253/136
# File to store the YT ad domains
FILE=/etc/pihole/youtube.hosts

# Fetch the list of domains, remove the ip's and save them
curl 'https://api.hackertarget.com/hostsearch/?q=googlevideo.com' | \
sed -E '1d;/redirector|manifest/d;s/(^r[0-9]+)\.(sn)(.*)(,.*)/\1---\2-\3/' > $FILE

# Scan log file for previously accessed domains
grep '^r.*googlevideo\.com' /var/log/pihole*.log \
| awk '{print $8}' \
| grep -vE "redirector|manifest" \
| sort | uniq >> $FILE

# Update Gravity
pihole -g

maybe this is neccesary to append the youtube domains list to the other already validated, works for me .

bash -c 'cat youtube.hosts >> list.1.raw.githubusercontent.com.domains'

i believe this would be a solution.
`
FILE=/etc/pihole/youtube.hosts

curl 'https://api.hackertarget.com/hostsearch/?q=googlevideo.com'
| awk -F, 'NR>1{print $1}'
| grep -vE "redirector|manifest" > $FILE

sed -i $FILE -re 's/(^r[[:digit:]]+)(.)(sn)/\1---\3-/'

grep '^r.googlevideo.com' /var/log/pihole.log
| awk '{print $8}'
| grep -vE "redirector|manifest"
| sort | uniq >> $FILE

bash -c 'cat /etc/pihole/youtube.hosts >> list.1.raw.githubusercontent.com.domains'

pihole -g`

An alternative may be to add the following regex on the blacklist:

^r.+\.googlevideo.com$

Seems to work for me locally. The ad segments still appear but it's just a spinner for 5 seconds then 'Skip Ads' appear.

rjhancock,

Thanks for the idea, I just tried it cause I didn't understand it. After adding the regex ^r.+.googlevideo.com$
all youtube movies are blocked....not one runs.
How local did you mean?

Yea, I removed it last night as it looks like about half the videos are blocked if not more. Bad idea. So I wonder if there is a different pattern that might work instead.