Create a gist now

Instantly share code, notes, and snippets.

Embed
OpenWRT adblock implementation

Others have recently developed packages for this same functionality, and done it better than anything I could do. Use the packages instead of this script:

Description

In its basic usage, this script will modify the router such that blocked addresses are null routed and unreachable. Since the address blocklist is full of advertising, malware, and tracking servers, this setup is generally a good thing. In addition, the router will update the blocklist weekly. However, the blocking is leaky, so do not expect everything to be blocked.

Setup

The script must be copied to an OpenWRT router (gargoyle firmware works fine, too).

For example, if the router is located at 192.168.1.1:

# scp adblock.sh root@192.168.1.1:/etc/adblock.sh

Make the script executable:

# chmod +x /etc/adblock.sh

Basic usage

If you are running the script for the first time:

# sh /etc/adblock.sh -f

There should be status updates in the output, but there should be no errors. If these commands complete without errors, the adblocking is active. You can test it by looking up, say, google analytics.

Whitelists and blacklists

The script supports defining whitelisted urls. That is, urls that will be filtered out of the downloaded blocklists. To whitelist urls, place them (one per line) in /etc/white.list.

Similarly, the script supports defining blacklisted urls - urls that will be added to the downloaded blocklists. To blacklist urls, place them (one per line) in /etc/black.list.

NOTE: The whitelist support is pretty stupid, so don't expect smart filtering (e.g., domain extrapolation). I've found it tedious, but worthwhile, to find the offending url in /etc/block.hosts and copy it to /etc/white.list.

Advanced usage

Toggle on and off

To toggle the blocking on and off, run the script with the -t switch:

# sh /etc/adblock.sh -t

Note: This does not delete the blocklist, whitelist, or blacklist.

Manually update blocklist

To manually update the blocklist, run the script without switches:

# sh /etc/adblock.sh

Full reinstall

To reinstall the current implementation:

# sh /etc/adblock.sh -r

Configuration

The config section of the script has some variables that alter the behaviour of the script.

For example, if you change:

ONLY_WIRELESS="N"

to

ONLY_WIRELESS="Y"

Then only the wireless interface of the router will filter the blocklist.

To change the configuration of an already active installation, I would toggle the adblocking off first, change the script, then toggle it back on. That is,

# sh /etc/adblock.sh -t # turn off
...modify variables...
# sh /etc/adblock.sh -t # turn on

However, if you change certain variables, you must re-update the blocklist because the redirection values will have changed. Other variables require a full restart because they must install or verify dependencies.

Configurable variables:

  • ONLY_WIRELESS (Y/N): Only filter on wireless interface
  • EXEMPT# (Y/N): Exempt ip range from filtering (between START_ RANGE and END_RANGE)
  • IPV6*: (Y/N): Add IPv6 support
  • SSL# (Y/N): Install wget with ssl support (only needed for ssl websites)
  • TRANS#: (Y/N): Modify router web server to server transparent pixel responses for blocked websites
  • ENDPOINT_IP4/IP6*: Define the IP to return for blocked hostnames (IPv4 and IPv6)
  • CRON: The cron line to put in the crontab

*: Requires blocklist update.

: Requires full reinstall

#!/bin/sh
#Put in /etc/adblock.sh
#Block ads, malware, etc.
#### CONFIG SECTION ####
# Only block wireless ads? Y/N
ONLY_WIRELESS="N"
# IPv6 support? Y/N
IPV6="N"
# Need SSL websites?
SSL="N"
# Try to transparently serve pixel response?
# If enabled, understand the consequences and mechanics of this setup
TRANS="N"
# Exempt an ip range
EXEMPT="N"
START_RANGE="192.168.1.0"
END_RANGE="192.168.1.255"
# Redirect endpoint
ENDPOINT_IP4="0.0.0.0"
ENDPOINT_IP6="::"
#Change the cron command to what is comfortable, or leave as is
CRON="0 4 * * 0,3 sh /etc/adblock.sh"
#### END CONFIG ####
#### FUNCTIONS ####
cleanup()
{
#Delete files used to build list to free up the limited space
echo 'Cleaning up...'
rm -f /tmp/block.build.list
rm -f /tmp/block.build.before
}
install_dependencies()
{
#Need iptables-mod-nat-extra installed
if opkg list-installed | grep -q iptables-mod-nat-extra
then
echo 'iptables-mod-nat-extra is installed!'
else
echo 'Updating package list...'
opkg update > /dev/null
echo 'Installing iptables-mod-nat-extra...'
opkg install iptables-mod-nat-extra > /dev/null
fi
#Need iptable-mod-iprange for exemption
if [ "$EXEMPT" = "Y" ]
then
if opkg list-installed | grep -q iptables-mod-iprange
then
echo 'iptables-mod-iprange installed'
else
echo 'Updating package list...'
opkg update > /dev/null
echo 'Installing iptables-mod-iprange...'
opkg install iptables-mod-iprange > /dev/null
fi
fi
#Need wget for https websites
if [ "$SSL" = "Y" ]
then
if opkg list-installed wget | grep -q wget
then
if wget --version | grep -q +ssl
then
echo 'wget (with ssl) found'
else
# wget without ssl, need to reinstall full wget
opkg update > /dev/null
opkg install wget --force-reinstall > /dev/null
fi
else
echo 'Updating package list...'
opkg update > /dev/null
echo 'Installing wget (with ssl)...'
opkg install wget > /dev/null
fi
fi
}
add_config()
{
if [ "$ONLY_WIRELESS" = "Y" ]
then
echo 'Wireless only blocking!'
if [ "$EXEMPT" = "Y" ]
then
echo 'Exempting some ips...'
FW1="iptables -t nat -I PREROUTING -m iprange ! --src-range $START_RANGE-$END_RANGE -i wlan+ -p tcp --dport 53 -j REDIRECT --to-ports 53"
FW2="iptables -t nat -I PREROUTING -m iprange ! --src-range $START_RANGE-$END_RANGE -i wlan+ -p udp --dport 53 -j REDIRECT --to-ports 53"
else
FW1="iptables -t nat -I PREROUTING -i wlan+ -p tcp --dport 53 -j REDIRECT --to-ports 53"
FW2="iptables -t nat -I PREROUTING -i wlan+ -p udp --dport 53 -j REDIRECT --to-ports 53"
fi
else
if [ "$EXEMPT" = "Y" ]
then
echo "Exempting some ips..."
FW1="iptables -t nat -I PREROUTING -m iprange ! --src-range $START_RANGE-$END_RANGE -p tcp --dport 53 -j REDIRECT --to-ports 53"
FW2="iptables -t nat -I PREROUTING -m iprange ! --src-range $START_RANGE-$END_RANGE -p udp --dport 53 -j REDIRECT --to-ports 53"
else
FW1="iptables -t nat -I PREROUTING -p tcp --dport 53 -j REDIRECT --to-ports 53"
FW2="iptables -t nat -I PREROUTING -p udp --dport 53 -j REDIRECT --to-ports 53"
fi
fi
echo 'Updating config...'
#Update DHCP config
uci add_list dhcp.@dnsmasq[0].addnhosts=/etc/block.hosts > /dev/null 2>&1 && uci commit
#Add to crontab
echo "$CRON" >> /etc/crontabs/root
#Update dnsmasq config for Tor
TOR=`uci get tor.global.enabled 2> /dev/null`
if [ "$TOR" == "1" ]
then
TORPORT=`uci get tor.client.dns_port`
TORIP="127.0.0.1:$TORPORT"
uci set dhcp.@dnsmasq[0].noresolv='1' > /dev/null &2>1 && uci commit
uci add_list dhcp.@dnsmasq[0].server="$TORIP" > /dev/null &2>1 && uci commit
fi
# Add firewall rules
echo "$FW1" >> /etc/firewall.user
echo "$FW2" >> /etc/firewall.user
# Provide hint if localservice is 1
LS=`uci get dhcp.@dnsmasq[0].localservice 2> /dev/null`
if [ "$LS" == "1" ]
then
echo "HINT: localservice is set to 1"
echo " Adblocking (and router DNS) over a VPN may not work"
echo " To allow VPN router DNS, manually set localservice to 0"
fi
# Determining uhttpd/httpd_gargoyle for transparent pixel support
if [ "$TRANS" = "Y" ]
then
if [ ! -e "/www/1.gif" ]
then
/usr/bin/wget -O /www/1.gif http://upload.wikimedia.org/wikipedia/commons/c/ce/Transparent.gif > /dev/null
fi
if [ -s "/usr/sbin/uhttpd" ]
then
#The default is none, so I don't want to check for it, so just write it
echo "uhttpd found..."
echo "updating server error page to return transparent pixel..."
uci set uhttpd.main.error_page="/1.gif" && uci commit
elif [ -s "/usr/sbin/httpd_gargoyle" ]
then
# Write without testing
echo "httpd_gargoyle found..."
echo "updating server error page to return transparent pixel..."
uci set httpd_gargoyle.server.page_not_found_file="1.gif" && uci commit
else
echo "Cannot find supported web server..."
fi
fi
}
update_blocklist()
{
#Delete the old block.hosts to make room for the updates
rm -f /etc/block.hosts
# Correct endpoint for transparent pixel response
if [ "$TRANS" = "Y" ] && [ -e "/www/1.gif" ] && ([ -s "/usr/sbin/uhttpd" ] || [ -s "/usr/sbin/httpd_gargoyle" ])
then
ENDPOINT_IP4=$(uci get network.lan.ipaddr)
if [ "$IPV6" = "Y" ]
then
ENDPOINT_IP6=$(uci get network.lan6.ipaddr)
fi
fi
echo 'Downloading hosts lists...'
#Download and process the files needed to make the lists (enable/add more, if you want)
wget -qO- http://www.mvps.org/winhelp2002/hosts.txt| awk -v r="$ENDPOINT_IP4" '{sub(/^0.0.0.0/, r)} $0 ~ "^"r' > /tmp/block.build.list
wget -qO- "http://adaway.org/hosts.txt"|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list
#wget -qO- http://www.malwaredomainlist.com/hostslist/hosts.txt|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list
#wget -qO- "http://hosts-file.net/.\ad_servers.txt"|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list
#Add black list, if non-empty
if [ -s "/etc/black.list" ]
then
echo 'Adding blacklist...'
awk -v r="$ENDPOINT_IP4" '/^[^#]/ { print r,$1 }' /etc/black.list >> /tmp/block.build.list
fi
echo 'Sorting lists...'
#Sort the download/black lists
awk '{sub(/\r$/,"");print $1,$2}' /tmp/block.build.list|sort -u > /tmp/block.build.before
#Filter (if applicable)
if [ -s "/etc/white.list" ]
then
#Filter the blacklist, supressing whitelist matches
# This is relatively slow =-(
echo 'Filtering white list...'
egrep -v "^[[:space:]]*$" /etc/white.list | awk '/^[^#]/ {sub(/\r$/,"");print $1}' | grep -vf - /tmp/block.build.before > /etc/block.hosts
else
cat /tmp/block.build.before > /etc/block.hosts
fi
if [ "$IPV6" = "Y" ]
then
safe_pattern=$(printf '%s\n' "$ENDPOINT_IP4" | sed 's/[[\.*^$(){}?+|/]/\\&/g')
safe_addition=$(printf '%s\n' "$ENDPOINT_IP6" | sed 's/[\&/]/\\&/g')
echo 'Adding ipv6 support...'
sed -i -re "s/^(${safe_pattern}) (.*)$/\1 \2\n${safe_addition} \2/g" /etc/block.hosts
fi
}
restart_firewall()
{
echo 'Restarting firewall...'
if [ -s "/usr/lib/gargoyle/restart_firewall.sh" ]
then
/usr/lib/gargoyle/restart_firewall.sh > /dev/null 2>&1
else
/etc/init.d/firewall restart > /dev/null 2>&1
fi
}
restart_dnsmasq()
{
if [ "$1" -eq "0" ]
then
echo 'Re-reading blocklist'
killall -HUP dnsmasq
else
echo 'Restarting dnsmasq...'
/etc/init.d/dnsmasq restart
fi
}
restart_http()
{
if [ -s "/usr/sbin/uhttpd" ]
then
echo 'Restarting uhttpd...'
/etc/init.d/uhttpd restart
elif [ -s "/usr/sbin/httpd_gargoyle" ]
then
echo 'Restarting httpd_gargoyle...'
/etc/init.d/httpd_gargoyle restart
fi
}
restart_cron()
{
echo 'Restarting cron...'
/etc/init.d/cron restart > /dev/null 2>&1
}
remove_config()
{
echo 'Reverting config...'
# Remove addnhosts
uci del_list dhcp.@dnsmasq[0].addnhosts=/etc/block.hosts > /dev/null 2>&1 && uci commit
# Remove cron entry
sed -i '/adblock/d' /etc/crontabs/root
# Remove firewall rules
sed -i '/--to-ports 53/d' /etc/firewall.user
# Remove Tor workarounds
uci del_list dhcp.@dnsmasq[0].server > /dev/null 2>&1 && uci commit
uci set dhcp.@dnsmasq[0].noresolv='0' > /dev/null 2>&1 && uci commit
# Remove proxying
uci delete uhttpd.main.error_page > /dev/null 2>&1 && uci commit
uci set httpd_gargoyle.server.page_not_found_file="login.sh" > /dev/null 2>&1 && uci commit
}
toggle()
{
# Check for cron as test for on/off
if grep -q "adblock" /etc/crontabs/root
then
# Turn off
echo 'Turning off!'
remove_config
else
# Turn on
echo 'Turning on!'
add_config
fi
# Restart services
restart_firewall
restart_dnsmasq 1
restart_http
restart_cron
}
#### END FUNCTIONS ####
### Options parsing ####
case "$1" in
# Toggle on/off
"-t")
toggle
;;
#First time run
"-f")
install_dependencies
add_config
update_blocklist
restart_firewall
restart_dnsmasq 1
restart_http
restart_cron
cleanup
;;
#Reinstall
"-r")
remove_config
install_dependencies
add_config
update_blocklist
restart_firewall
restart_dnsmasq 1
restart_http
restart_cron
cleanup
;;
#Default updates blocklist only
*)
update_blocklist
restart_dnsmasq 0
cleanup
;;
esac
#### END OPTIONS ####
exit 0
#/etc/black.list
#add some server that the list doesn't block
example1.block.com
#/etc/sysupgrade.conf
#This file is a list of files that should be preserved through upgrades
#OPTIONAL!!!!!
/etc/passwd
/etc/shadow
...
...
/etc/adblock.sh #ADD THIS LINE
/etc/white.list #AND THIS ONE
/etc/block.hosts #AND THIS ONE
/etc/black.list #AND THIS ONE
#/etc/white.list
#Add whitelisted addresses, when appropriate, etc.
a248.e.akamai.net
@stoinov

This comment has been minimized.

Show comment
Hide comment
@stoinov

stoinov Feb 23, 2014

Instead of dnsmasq restart you could just use killall -HUP dnsmasq which will reload the host files as per http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html#lbAF

stoinov commented Feb 23, 2014

Instead of dnsmasq restart you could just use killall -HUP dnsmasq which will reload the host files as per http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html#lbAF

@Cybso

This comment has been minimized.

Show comment
Hide comment
@Cybso

Cybso May 21, 2014

Have a look at my fork at https://gist.github.com/Cybso/bf9b69c6a638ffd68281

  • Use 0.0.0.0 instead of 127.0.0.1 to avoid conflicts with local webservers
  • Added IPv6 support (::, which means "not specified")
  • adaway.org redirects to https, now.

Cybso commented May 21, 2014

Have a look at my fork at https://gist.github.com/Cybso/bf9b69c6a638ffd68281

  • Use 0.0.0.0 instead of 127.0.0.1 to avoid conflicts with local webservers
  • Added IPv6 support (::, which means "not specified")
  • adaway.org redirects to https, now.
@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump May 24, 2014

yeah, I just noticed that adaway thing gotta update it, lol. yeah, 0.0.0.0 would work; i'll update that stuff thanks! =-)

Owner

teffalump commented May 24, 2014

yeah, I just noticed that adaway thing gotta update it, lol. yeah, 0.0.0.0 would work; i'll update that stuff thanks! =-)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump May 24, 2014

Issue about SIGHUP is that it re-reads addnhosts, hosts, etc files, but it doesn't re-read the config. If this is the first time running the script, dnsmasq would not update and read the new addnhost file. I probably should extend the script, so it can use pkill, not simply restarting, when applicable because I agree, re-reading the files is much better than restarting the process.

Owner

teffalump commented May 24, 2014

Issue about SIGHUP is that it re-reads addnhosts, hosts, etc files, but it doesn't re-read the config. If this is the first time running the script, dnsmasq would not update and read the new addnhost file. I probably should extend the script, so it can use pkill, not simply restarting, when applicable because I agree, re-reading the files is much better than restarting the process.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump May 24, 2014

Modified script to use pkill when possible.

Owner

teffalump commented May 24, 2014

Modified script to use pkill when possible.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump May 24, 2014

Added firewall restart if settings changed, since that's required on first run.

Owner

teffalump commented May 24, 2014

Added firewall restart if settings changed, since that's required on first run.

@borore

This comment has been minimized.

Show comment
Hide comment
@borore

borore Jun 17, 2014

Great! thank you!

borore commented Jun 17, 2014

Great! thank you!

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jul 8, 2014

Added checks for pkill command and iptables-mod-nat-extra package.

Owner

teffalump commented Jul 8, 2014

Added checks for pkill command and iptables-mod-nat-extra package.

@ronison

This comment has been minimized.

Show comment
Hide comment
@ronison

ronison Jul 16, 2014

Since Barrier Breaker / Trunk version the pkill command is in procps-pkill package and no more in procps package.

echo 'Installing procps package...'
opkg install procps > /dev/null
opkg install procps-pkill > /dev/null #ADDING THIS LINE WILL SOLVE

ronison commented Jul 16, 2014

Since Barrier Breaker / Trunk version the pkill command is in procps-pkill package and no more in procps package.

echo 'Installing procps package...'
opkg install procps > /dev/null
opkg install procps-pkill > /dev/null #ADDING THIS LINE WILL SOLVE

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jul 17, 2014

Thanks a bunch. Updated. I'm running gargoyle, so I'm somewhat out of step with the main openwrt releases.

Owner

teffalump commented Jul 17, 2014

Thanks a bunch. Updated. I'm running gargoyle, so I'm somewhat out of step with the main openwrt releases.

@troych

This comment has been minimized.

Show comment
Hide comment
@troych

troych Sep 8, 2014

The ipv6 support does not seem to work for me, shouldn't it create the "/tmp/block.build.before" file including "::" and "0.0.0.0" after doing the sed command?

edit: googled a bit and it looks like this could be the problem: http://osdir.com/ml/bug-gnu-utils-gnu/2011-04/msg00002.html and indeed if I change line 77 to this:

sed -i -re 's/^(0\.0\.0\.0) (.*)$/\1 \2\n:: \2/g' /tmp/block.build.before

I get my ipv6 supported blocklist

troych commented Sep 8, 2014

The ipv6 support does not seem to work for me, shouldn't it create the "/tmp/block.build.before" file including "::" and "0.0.0.0" after doing the sed command?

edit: googled a bit and it looks like this could be the problem: http://osdir.com/ml/bug-gnu-utils-gnu/2011-04/msg00002.html and indeed if I change line 77 to this:

sed -i -re 's/^(0\.0\.0\.0) (.*)$/\1 \2\n:: \2/g' /tmp/block.build.before

I get my ipv6 supported blocklist

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Sep 30, 2014

@troych: Thanks for this. I'll update it. Was working for me, but, hey, more readable that way, too. Tons of switches squashed together... eww!

Owner

teffalump commented Sep 30, 2014

@troych: Thanks for this. I'll update it. Was working for me, but, hey, more readable that way, too. Tons of switches squashed together... eww!

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Sep 30, 2014

Updated.

Owner

teffalump commented Sep 30, 2014

Updated.

@norman25

This comment has been minimized.

Show comment
Hide comment
@norman25

norman25 Oct 31, 2014

great job.
works fine on my router.
is possible to separate the ad-blocking in the lan and wifi?
for example no ad-block on the LAN and enable ad-blocking in the WIFI?
thank you

great job.
works fine on my router.
is possible to separate the ad-blocking in the lan and wifi?
for example no ad-block on the LAN and enable ad-blocking in the WIFI?
thank you

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Nov 26, 2014

@norman25: Sorry about the delay responding! I think you are saying that you want no ad-blocking on the wired internet, but blocking on wireless?

Owner

teffalump commented Nov 26, 2014

@norman25: Sorry about the delay responding! I think you are saying that you want no ad-blocking on the wired internet, but blocking on wireless?

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Nov 26, 2014

Updated. Added wireless only support.

Owner

teffalump commented Nov 26, 2014

Updated. Added wireless only support.

@norman25

This comment has been minimized.

Show comment
Hide comment
@norman25

norman25 Dec 2, 2014

@teffalump: you want no ad-blocking on the wired internet, but blocking on wireless?
yes is exactly that i need!!!
@teffalump: Updated. Added wireless only support.
well i'm testing now thx.

norman25 commented Dec 2, 2014

@teffalump: you want no ad-blocking on the wired internet, but blocking on wireless?
yes is exactly that i need!!!
@teffalump: Updated. Added wireless only support.
well i'm testing now thx.

@norman25

This comment has been minimized.

Show comment
Hide comment
@norman25

norman25 Dec 2, 2014

well I tried the script and it works but recently in firefox, opera and chrome blocking are skipped. (with internet explorer and slimbrowser working very well).
any idea why this happens?

norman25 commented Dec 2, 2014

well I tried the script and it works but recently in firefox, opera and chrome blocking are skipped. (with internet explorer and slimbrowser working very well).
any idea why this happens?

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Dec 8, 2014

@norman25: Sorry again about delay, busy IRL. Do you mean the wireless-only portion is broken? Or blocking in general is broken?

Owner

teffalump commented Dec 8, 2014

@norman25: Sorry again about delay, busy IRL. Do you mean the wireless-only portion is broken? Or blocking in general is broken?

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Dec 8, 2014

Updated. Disabled some hosts urls. Moved ipv6 support after whitelist function.

Owner

teffalump commented Dec 8, 2014

Updated. Disabled some hosts urls. Moved ipv6 support after whitelist function.

@norman25

This comment has been minimized.

Show comment
Hide comment
@norman25

norman25 Dec 15, 2014

@teffalump: Sorry again about delay, busy IRL. Do you mean the wireless-only portion is broken? Or blocking in general is broken?
nope your modified wireless version is working fine. there is a problem in some pages that use https. example, (https://myadvert.ads) seems not blocked correctly, anyway only occurs in some pages is not a big deal.
thx a lot.

@teffalump: Sorry again about delay, busy IRL. Do you mean the wireless-only portion is broken? Or blocking in general is broken?
nope your modified wireless version is working fine. there is a problem in some pages that use https. example, (https://myadvert.ads) seems not blocked correctly, anyway only occurs in some pages is not a big deal.
thx a lot.

@gregoirefavre

This comment has been minimized.

Show comment
Hide comment
@gregoirefavre

gregoirefavre Dec 15, 2014

Tremendously well done : thank you very much !!!
Would there be an easy way to not have all blocked url go to an empty gif like in http://pakitong.blogspot.ch/2014/02/openwrt-installing-adblock-with.html (but without creating another interface) ?

Tremendously well done : thank you very much !!!
Would there be an easy way to not have all blocked url go to an empty gif like in http://pakitong.blogspot.ch/2014/02/openwrt-installing-adblock-with.html (but without creating another interface) ?

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Dec 22, 2014

@norman25: Odd. Maybe explicitly add them to the blacklist? Hehe, or ignore them.

Owner

teffalump commented Dec 22, 2014

@norman25: Odd. Maybe explicitly add them to the blacklist? Hehe, or ignore them.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Dec 22, 2014

@gregoirefavre: I think you are asking, is there a simple way to return an empty file instead of returning a spoofed DNS response (like, 0.0.0.0)? Put simply, it's complicated (as you can tell from the article). In this case, the adblocking is done on the router by returning an empty page, which complicates the setup. However, one could run a pixelserv setup on the personal computer and do the same thing. The bigger issue seems to be how to accomplish the best adblocking setup. Adblock Edge, ABP, etc. work very well in the browser, although there are downsides. Then there is this way for the router. Then there is running dnsmasq + pixelserv on the computer. Etc. A complete solution probably incorporates some of everything. I'll look into simpler setups to return an empty page, but I think any solution will be somewhat inelegant. With the spoofed DNS response, I haven't run into performance issues - curl 0.0.0.0:80 or 0.0.0.0:443 immediately refuse connections, they do not timeout. To me, this setup seems good enough.... What is it? The best is the enemy of the good.

Owner

teffalump commented Dec 22, 2014

@gregoirefavre: I think you are asking, is there a simple way to return an empty file instead of returning a spoofed DNS response (like, 0.0.0.0)? Put simply, it's complicated (as you can tell from the article). In this case, the adblocking is done on the router by returning an empty page, which complicates the setup. However, one could run a pixelserv setup on the personal computer and do the same thing. The bigger issue seems to be how to accomplish the best adblocking setup. Adblock Edge, ABP, etc. work very well in the browser, although there are downsides. Then there is this way for the router. Then there is running dnsmasq + pixelserv on the computer. Etc. A complete solution probably incorporates some of everything. I'll look into simpler setups to return an empty page, but I think any solution will be somewhat inelegant. With the spoofed DNS response, I haven't run into performance issues - curl 0.0.0.0:80 or 0.0.0.0:443 immediately refuse connections, they do not timeout. To me, this setup seems good enough.... What is it? The best is the enemy of the good.

@flashjh

This comment has been minimized.

Show comment
Hide comment
@flashjh

flashjh Dec 25, 2014

This is really awesome! I used opkg install wget and then everything worked perfectly. I have this combined with OpenDNS on my Dlink DIR-825 with OpenWRT. Finally, a decent solution for some control of the home network. Thanks!!

flashjh commented Dec 25, 2014

This is really awesome! I used opkg install wget and then everything worked perfectly. I have this combined with OpenDNS on my Dlink DIR-825 with OpenWRT. Finally, a decent solution for some control of the home network. Thanks!!

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Jan 2, 2015

Hi, really appreciate your work on this. I have managed to implement it such that the script works, however i am having trouble with the white.list implementation.
This is ok because i CURRENTLY have no use for this, but in the future it may be a problem.

I find that when i have a file called white.list in /etc then the script still executes without error, however block.hosts ends up completely empty.
If white.list is not present, block.hosts populates as expected.

I have no experience with shell scripting so i unfortunately can't add any insight.
Regards

Hi, really appreciate your work on this. I have managed to implement it such that the script works, however i am having trouble with the white.list implementation.
This is ok because i CURRENTLY have no use for this, but in the future it may be a problem.

I find that when i have a file called white.list in /etc then the script still executes without error, however block.hosts ends up completely empty.
If white.list is not present, block.hosts populates as expected.

I have no experience with shell scripting so i unfortunately can't add any insight.
Regards

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jan 2, 2015

@lantis1008: Yeah, I think what happens is that grep matches white characters (like space) in the blocked urls, so every line is white-listed. I've added a regex to hopefully remove all whitespace only lines. I'll try to test it. =-/

@flashjh: Glad you like it! Keep me up-to-date on the openwrt stuff, I run gargoyle, so I'm a little out-of-step, so if anything breaks, tell me. =-)

Owner

teffalump commented Jan 2, 2015

@lantis1008: Yeah, I think what happens is that grep matches white characters (like space) in the blocked urls, so every line is white-listed. I've added a regex to hopefully remove all whitespace only lines. I'll try to test it. =-/

@flashjh: Glad you like it! Keep me up-to-date on the openwrt stuff, I run gargoyle, so I'm a little out-of-step, so if anything breaks, tell me. =-)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jan 4, 2015

Added GNU wget check (not Busybox) and install.

Owner

teffalump commented Jan 4, 2015

Added GNU wget check (not Busybox) and install.

@rodtrevizan

This comment has been minimized.

Show comment
Hide comment
@rodtrevizan

rodtrevizan Jan 5, 2015

Hi, I'm using OpenWrt r36088 and when I run " sh /etc/adblock.sh " i get:
" /etc/adblock.sh: line 132: syntax error: unexpected end of file (expecting "then") "
Any tips?
Thanks

Hi, I'm using OpenWrt r36088 and when I run " sh /etc/adblock.sh " i get:
" /etc/adblock.sh: line 132: syntax error: unexpected end of file (expecting "then") "
Any tips?
Thanks

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jan 5, 2015

@rodtrevizan: Hehe, I've had that before, too. I believe that the error occurs when the file is saved in dos format, not unix. Try running something like this:

root@192.168.1.1: /etc# sed -i -re 's/\r$//g' adblock.sh

The command removes the dos end-of-line character (\r). If that doesn't work, fiddle around with trying to convert the file to unix format. Hope that helps.

Owner

teffalump commented Jan 5, 2015

@rodtrevizan: Hehe, I've had that before, too. I believe that the error occurs when the file is saved in dos format, not unix. Try running something like this:

root@192.168.1.1: /etc# sed -i -re 's/\r$//g' adblock.sh

The command removes the dos end-of-line character (\r). If that doesn't work, fiddle around with trying to convert the file to unix format. Hope that helps.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jan 5, 2015

Removed pkill, just use killall - fewer package issues.

Owner

teffalump commented Jan 5, 2015

Removed pkill, just use killall - fewer package issues.

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Jan 5, 2015

Thanks for updating it to remove those package issues. just updated my gargoyle to 1.7.0 and it spat issues at me until this change :)

Thanks for updating it to remove those package issues. just updated my gargoyle to 1.7.0 and it spat issues at me until this change :)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jan 5, 2015

@lantis1008: Yeah, same thing, lol. =-)

Owner

teffalump commented Jan 5, 2015

@lantis1008: Yeah, same thing, lol. =-)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jan 5, 2015

Tried to better handle wget installation and checking. IPV6 support optional now.

Owner

teffalump commented Jan 5, 2015

Tried to better handle wget installation and checking. IPV6 support optional now.

@gregoirefavre

This comment has been minimized.

Show comment
Hide comment
@gregoirefavre

gregoirefavre Jan 6, 2015

Thanks for all, and it's true they aren't any perfomance issue I can when using it (which is all the time) :-)

Thanks for all, and it's true they aren't any perfomance issue I can when using it (which is all the time) :-)

@pkrexer

This comment has been minimized.

Show comment
Hide comment
@pkrexer

pkrexer Jan 7, 2015

Hello, I'm running on Chaos Calmer r43823 and I'm running to this error when executing the script:

* satisfy_dependencies_for: Cannot satisfy the following dependencies for iptables-mod-nat-extra:
*      kernel (= 3.14.27-1-7c18d64c08713497f02c692f0b8572ab) *
* opkg_install_cmd: Cannot install package iptables-mod-nat-extra.

However, it still seems to be working? I've tested a number of websites and the ads are indeed being blocked.

pkrexer commented Jan 7, 2015

Hello, I'm running on Chaos Calmer r43823 and I'm running to this error when executing the script:

* satisfy_dependencies_for: Cannot satisfy the following dependencies for iptables-mod-nat-extra:
*      kernel (= 3.14.27-1-7c18d64c08713497f02c692f0b8572ab) *
* opkg_install_cmd: Cannot install package iptables-mod-nat-extra.

However, it still seems to be working? I've tested a number of websites and the ads are indeed being blocked.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jan 8, 2015

@pkrexer: You tried looking at development package snapshots? Also, /etc/opkg.conf should show where the packages are coming from... and since you are running a very new version, may not match with the package release. shoulder shrug No idea, other than that. Hehe, if it's working... is kind of annoying though. It sounds like it's saying it isn't installed, but is... =-/

Owner

teffalump commented Jan 8, 2015

@pkrexer: You tried looking at development package snapshots? Also, /etc/opkg.conf should show where the packages are coming from... and since you are running a very new version, may not match with the package release. shoulder shrug No idea, other than that. Hehe, if it's working... is kind of annoying though. It sounds like it's saying it isn't installed, but is... =-/

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jan 27, 2015

Use restart_firewall.sh for those Gargoyle users - nice script that won't mess up other things. Thanks @lantis1008.

Owner

teffalump commented Jan 27, 2015

Use restart_firewall.sh for those Gargoyle users - nice script that won't mess up other things. Thanks @lantis1008.

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Jan 28, 2015

great solution! :)

great solution! :)

@triplefb

This comment has been minimized.

Show comment
Hide comment
@triplefb

triplefb Feb 6, 2015

This is awesome, thanks so much. Has anyone had any issues with Chrome on IOS while running this script? It seems to just hang, both on my phone & ipad. (I'd much rather use Safari with this than Chrome without though!)

triplefb commented Feb 6, 2015

This is awesome, thanks so much. Has anyone had any issues with Chrome on IOS while running this script? It seems to just hang, both on my phone & ipad. (I'd much rather use Safari with this than Chrome without though!)

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Feb 7, 2015

@teffalump: i'm looking at implementing this into a gargoyle plugin with your permission.

@teffalump: i'm looking at implementing this into a gargoyle plugin with your permission.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Feb 16, 2015

@lantis1008: Yeah, go right ahead. It was on my to-do list, but... yeah. Lol. =-)

Owner

teffalump commented Feb 16, 2015

@lantis1008: Yeah, go right ahead. It was on my to-do list, but... yeah. Lol. =-)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Feb 16, 2015

@triplefb: I've had reports about some issues. I guess I don't know, really. I wonder if the browser must contact certain servers...? I really don't know.

Owner

teffalump commented Feb 16, 2015

@triplefb: I've had reports about some issues. I guess I don't know, really. I wonder if the browser must contact certain servers...? I really don't know.

@norman25

This comment has been minimized.

Show comment
Hide comment
@norman25

norman25 Mar 2, 2015

@teffalump
i see you change the script, any improvements?

norman25 commented Mar 2, 2015

@teffalump
i see you change the script, any improvements?

@tinyema

This comment has been minimized.

Show comment
Hide comment
@tinyema

tinyema Mar 18, 2015

@teffalump
great job, it works perfectly!
i found a couple of mistakes...
Filtering white list:
egrep -v "^[[:space:]]$" /etc/white.list | awk '/^[^#]/ {sub(/\r$/,"");print $1}' | grep -vf - /tmp/block.build.list > /etc/block.hosts
must be:
egrep -v "^[[:space:]]
$" /etc/white.list | awk '/^[^#]/ {sub(/\r$/,"");print $1}' | grep -vf - /tmp/block.build.before > /etc/block.hosts
and you forgot to delete /tmp/block.build.before in cleaning up's section:
rm -f /tmp/block.build.before

tinyema commented Mar 18, 2015

@teffalump
great job, it works perfectly!
i found a couple of mistakes...
Filtering white list:
egrep -v "^[[:space:]]$" /etc/white.list | awk '/^[^#]/ {sub(/\r$/,"");print $1}' | grep -vf - /tmp/block.build.list > /etc/block.hosts
must be:
egrep -v "^[[:space:]]
$" /etc/white.list | awk '/^[^#]/ {sub(/\r$/,"");print $1}' | grep -vf - /tmp/block.build.before > /etc/block.hosts
and you forgot to delete /tmp/block.build.before in cleaning up's section:
rm -f /tmp/block.build.before

@prouderthings

This comment has been minimized.

Show comment
Hide comment
@prouderthings

prouderthings Mar 20, 2015

First, thanks for the script! With my router (TL-WR184N), I had trouble when it got to the installing wget (with ssl) part (not enough memory). I changed the https to http and it retrieved the list without error. I do have a question though... instead of ads, I get the "Server not found". Was that how it is suppose to be or is it suppose to return a blank placeholder?

First, thanks for the script! With my router (TL-WR184N), I had trouble when it got to the installing wget (with ssl) part (not enough memory). I changed the https to http and it retrieved the list without error. I do have a question though... instead of ads, I get the "Server not found". Was that how it is suppose to be or is it suppose to return a blank placeholder?

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Mar 20, 2015

to add to @tinyema's suggestion, the else statement following that line should also have "/tmp/block.build.before" if i'm not mistaken.

@prouderthings yes it is supposed to show "server not found". There are ways of returning a blank placeholder,

@teffalump
Is it possible to use a variable instead of "0.0.0.0" in case i wish to specify a destination other than this for redirecting? i suck with awk and all my attempts so far don't work. my main issue is the pattern matching at the end "/^x.x.x.x/".

EDIT -> got it. check out my fork if you like

to add to @tinyema's suggestion, the else statement following that line should also have "/tmp/block.build.before" if i'm not mistaken.

@prouderthings yes it is supposed to show "server not found". There are ways of returning a blank placeholder,

@teffalump
Is it possible to use a variable instead of "0.0.0.0" in case i wish to specify a destination other than this for redirecting? i suck with awk and all my attempts so far don't work. my main issue is the pattern matching at the end "/^x.x.x.x/".

EDIT -> got it. check out my fork if you like

@triplefb

This comment has been minimized.

Show comment
Hide comment
@triplefb

triplefb Mar 24, 2015

That works well for me @lantis1008, thanks! It really seems to pound the cpu on my router though. (wndr3700v2, gargoyle 1.7.0.) Sometimes pages display the login screen to my router instead of the ad element too. (Which is still better than the ad, ha ha.) Thanks again all!

That works well for me @lantis1008, thanks! It really seems to pound the cpu on my router though. (wndr3700v2, gargoyle 1.7.0.) Sometimes pages display the login screen to my router instead of the ad element too. (Which is still better than the ad, ha ha.) Thanks again all!

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Mar 25, 2015

@triplefb if its displaying the login screen, did you correctly make the substitution in /etc/config/httpd_gargoyle option error_page_not_found_file to "/1.gif". you'll then need to wget the 1.gif image into your /www/ directory. check out my fork for more details :)

please be aware that in the next gargoyle version (1.7.2) it is likely that this substitution will have to be changed as it is being migrated from httpd to uhttpd. i'll keep you posted.

@triplefb if its displaying the login screen, did you correctly make the substitution in /etc/config/httpd_gargoyle option error_page_not_found_file to "/1.gif". you'll then need to wget the 1.gif image into your /www/ directory. check out my fork for more details :)

please be aware that in the next gargoyle version (1.7.2) it is likely that this substitution will have to be changed as it is being migrated from httpd to uhttpd. i'll keep you posted.

@triplefb

This comment has been minimized.

Show comment
Hide comment
@triplefb

triplefb Mar 25, 2015

Whoops I didn't read closely enough, missed the part about updating /etc/config/httpd_gargoyle. Thank you!

Whoops I didn't read closely enough, missed the part about updating /etc/config/httpd_gargoyle. Thank you!

@keiler3000

This comment has been minimized.

Show comment
Hide comment
@keiler3000

keiler3000 Mar 26, 2015

There is something you have to keep in mind when you use this script:
It redirects DNS requests and filters out the balck listed hosts.

However, if you need to update bind zones, e.g. because you are running a ddns setup from or behind the router using the adblock script, you may screw up your dyndns updates.

In order to let dns updates through, something like

BIND_REQUESTS="Y"
BIND_REQUEST_HOST=192.168.0.2 # host issuing ddns updates
...
and
if [ "$BIND_REQUESTS" == "Y" ]
then
FW1="iptables -t nat -I PREROUTING ! -s $BIND_REQUEST_HOST -p tcp --dport 53 -j REDIRECT --to-ports 53"
FW2="iptables -t nat -I PREROUTING ! -s $BIND_REQUEST_HOST -p udp --dport 53 -j REDIRECT --to-ports 53"

or something like this.

@lantis1008: Does your fork include @tinyema's changes?

There is something you have to keep in mind when you use this script:
It redirects DNS requests and filters out the balck listed hosts.

However, if you need to update bind zones, e.g. because you are running a ddns setup from or behind the router using the adblock script, you may screw up your dyndns updates.

In order to let dns updates through, something like

BIND_REQUESTS="Y"
BIND_REQUEST_HOST=192.168.0.2 # host issuing ddns updates
...
and
if [ "$BIND_REQUESTS" == "Y" ]
then
FW1="iptables -t nat -I PREROUTING ! -s $BIND_REQUEST_HOST -p tcp --dport 53 -j REDIRECT --to-ports 53"
FW2="iptables -t nat -I PREROUTING ! -s $BIND_REQUEST_HOST -p udp --dport 53 -j REDIRECT --to-ports 53"

or something like this.

@lantis1008: Does your fork include @tinyema's changes?

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Mar 27, 2015

@keiler3000 yes my fork includes the changes from @tinyema
I also changed the firewall restart script section so that it only applies to gargoyle. I basically removed the if statement.

Also I've had no issues with my ddns updating, but if you can sell me a good reasoning behind it I'll try adding it in.

@keiler3000 yes my fork includes the changes from @tinyema
I also changed the firewall restart script section so that it only applies to gargoyle. I basically removed the if statement.

Also I've had no issues with my ddns updating, but if you can sell me a good reasoning behind it I'll try adding it in.

@keiler3000

This comment has been minimized.

Show comment
Hide comment
@keiler3000

keiler3000 Mar 27, 2015

Well, the reasoning is:

I have a setup like this:

client -//WiFi/LAN//- router -//Internet//- self hosted domain host

After activating adblock.sh on the router I was unable to update zone entries using nsupdate from the client on the host. I always got TSIG errors and in the end the update failed with BADKEY. From other hosts, e.g. outside my LAN, there was no problem. After hours of tying I realized that the script might be the cause as it, obviously, acts on port 53. After including the exception for the client IP, the problem was gone.

Well, the reasoning is:

I have a setup like this:

client -//WiFi/LAN//- router -//Internet//- self hosted domain host

After activating adblock.sh on the router I was unable to update zone entries using nsupdate from the client on the host. I always got TSIG errors and in the end the update failed with BADKEY. From other hosts, e.g. outside my LAN, there was no problem. After hours of tying I realized that the script might be the cause as it, obviously, acts on port 53. After including the exception for the client IP, the problem was gone.

@9M2PJU

This comment has been minimized.

Show comment
Hide comment
@9M2PJU

9M2PJU Mar 27, 2015

Thanks

9M2PJU commented Mar 27, 2015

Thanks

@timche

This comment has been minimized.

Show comment
Hide comment
@timche

timche Mar 31, 2015

Thanks for the script. I've installed it on my TL-WR1034ND with ONLY_WIRELESS="Y". It works fine, but my LAN devices are affected as well and I don't know why.

timche commented Mar 31, 2015

Thanks for the script. I've installed it on my TL-WR1034ND with ONLY_WIRELESS="Y". It works fine, but my LAN devices are affected as well and I don't know why.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Mar 31, 2015

Hey guys, sorry I've been out of it. IRL stuff. Let me get those edits started. Lol. And thanks @lantis1008 again. =-)

Owner

teffalump commented Mar 31, 2015

Hey guys, sorry I've been out of it. IRL stuff. Let me get those edits started. Lol. And thanks @lantis1008 again. =-)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Mar 31, 2015

@norman25: Thought I responded back then. But anyways, nothing really major. I'll take the suggestion and add the config for the error page, I think. =-)

Owner

teffalump commented Mar 31, 2015

@norman25: Thought I responded back then. But anyways, nothing really major. I'll take the suggestion and add the config for the error page, I think. =-)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Mar 31, 2015

@tinyema: Thanks, another mistake (cat /etc/block.build.list > /etc/block.hosts)... updated that, too. Thanks!

Owner

teffalump commented Mar 31, 2015

@tinyema: Thanks, another mistake (cat /etc/block.build.list > /etc/block.hosts)... updated that, too. Thanks!

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Mar 31, 2015

@lantis1008: I actually was thinking the same thing about the variable holding the redirection. I like what you did with that (and the pixel). Cool if I add it? =-)

Owner

teffalump commented Mar 31, 2015

@lantis1008: I actually was thinking the same thing about the variable holding the redirection. I like what you did with that (and the pixel). Cool if I add it? =-)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Mar 31, 2015

@tch3ung: Hmm... Hmm... I don't know. =-/ Maybe the interface isn't named that? Have you looked?

Owner

teffalump commented Mar 31, 2015

@tch3ung: Hmm... Hmm... I don't know. =-/ Maybe the interface isn't named that? Have you looked?

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Mar 31, 2015

Hopefully the ip6 endpoint choice works. Gotta test it. Regexes.... uggh.

Owner

teffalump commented Mar 31, 2015

Hopefully the ip6 endpoint choice works. Gotta test it. Regexes.... uggh.

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Apr 1, 2015

No worries man add away. :)

My script does actually work I've found, I was just being foolish in my testing.
You can make it universal by doing a similar thing to your firewall IF.
Test for the file httpd_gargoyle. If yes, do a uci set httpd_gargoyle.something.error_page "/1.gif"
Otherwise do a uci set uhttpd.uhttpd.error_page "/1.gif"

I'm currently at work so I don't have access to my router to get the actual file and command names sorry mate.

If you don't want to boost the script you can make people do it themselves :)

No worries man add away. :)

My script does actually work I've found, I was just being foolish in my testing.
You can make it universal by doing a similar thing to your firewall IF.
Test for the file httpd_gargoyle. If yes, do a uci set httpd_gargoyle.something.error_page "/1.gif"
Otherwise do a uci set uhttpd.uhttpd.error_page "/1.gif"

I'm currently at work so I don't have access to my router to get the actual file and command names sorry mate.

If you don't want to boost the script you can make people do it themselves :)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Apr 1, 2015

@lantis1008: Thanks! Hehe, the hands-off approach is tempting. Lol. I suppose I'll do it anyways though. =-)

Owner

teffalump commented Apr 1, 2015

@lantis1008: Thanks! Hehe, the hands-off approach is tempting. Lol. I suppose I'll do it anyways though. =-)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Apr 1, 2015

A first attempt at supporting both servers. Gotta test.

Owner

teffalump commented Apr 1, 2015

A first attempt at supporting both servers. Gotta test.

@OpenWrtFan

This comment has been minimized.

Show comment
Hide comment
@OpenWrtFan

OpenWrtFan Apr 1, 2015

Missing an extra 'fi' between line 107 and 108. Crashes otherwise.

Missing an extra 'fi' between line 107 and 108. Crashes otherwise.

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Apr 1, 2015

Code looks good apart from missing fi as @OpenWrtFan said.
Only issue I can think of is if a person does a sysupgrade with attempt to preserve files then they will end up with both files. May need a nicer way of testing, or could edit both.

If you want to test for those settings first you could do an AND test using uci get.

Code looks good apart from missing fi as @OpenWrtFan said.
Only issue I can think of is if a person does a sysupgrade with attempt to preserve files then they will end up with both files. May need a nicer way of testing, or could edit both.

If you want to test for those settings first you could do an AND test using uci get.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Apr 1, 2015

@OpenWrtFan: Thanks!

@lantis1008: You mean, the uhttpd or httpd_gargoyle thing? I could switch the order, so uhttpd is always first, then only ever updates uhttpd if it has it? Or, is it something else?

Owner

teffalump commented Apr 1, 2015

@OpenWrtFan: Thanks!

@lantis1008: You mean, the uhttpd or httpd_gargoyle thing? I could switch the order, so uhttpd is always first, then only ever updates uhttpd if it has it? Or, is it something else?

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Apr 1, 2015

Testing for uhttpd first would fix the issue :)

Testing for uhttpd first would fix the issue :)

@Schabernack

This comment has been minimized.

Show comment
Hide comment
@Schabernack

Schabernack Apr 1, 2015

blocking lists used with abp or ublock (e.g. easylist) have some nice entries to block youtube ads, social tracking and filters for non-english ad networks. any chance this could be included here?

thanks for putting this together.

blocking lists used with abp or ublock (e.g. easylist) have some nice entries to block youtube ads, social tracking and filters for non-english ad networks. any chance this could be included here?

thanks for putting this together.

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Apr 1, 2015

Easy list is designed in a format that works specifically with those ad blockers and would take a lot (too much) filtering to get into here IMO.

Easy list is designed in a format that works specifically with those ad blockers and would take a lot (too much) filtering to get into here IMO.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Apr 2, 2015

Moved order of server checks... hopefully should obviate any upgrade problems.

Owner

teffalump commented Apr 2, 2015

Moved order of server checks... hopefully should obviate any upgrade problems.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Apr 2, 2015

@Schabernack: Yeah, I thought about how to work with the lists, but I completely agree with @lantis1008. This method is leaky, but takes care of most unwanted domains. Those blocklists mostly work on 'objects' (e.g., /*.block.js) within the page. That kind of fine blocking should not happen on the router. Over TLS, it's not even possible (I think?!). This method of DNS blacklisting will, hopefully, become ineffective once dnscrypt/dnssec becomes more widely used. For non-browser addons, Polipo, or Privoxy, can do things like the fine blocking you are talking about. However, thanks for the interest! =-)

Owner

teffalump commented Apr 2, 2015

@Schabernack: Yeah, I thought about how to work with the lists, but I completely agree with @lantis1008. This method is leaky, but takes care of most unwanted domains. Those blocklists mostly work on 'objects' (e.g., /*.block.js) within the page. That kind of fine blocking should not happen on the router. Over TLS, it's not even possible (I think?!). This method of DNS blacklisting will, hopefully, become ineffective once dnscrypt/dnssec becomes more widely used. For non-browser addons, Polipo, or Privoxy, can do things like the fine blocking you are talking about. However, thanks for the interest! =-)

@keiler3000

This comment has been minimized.

Show comment
Hide comment
@keiler3000

keiler3000 Apr 4, 2015

I still believe it would be good to include an option to explicitly exclude a specific host or IP range from the filtering in the firewall rules.

I still believe it would be good to include an option to explicitly exclude a specific host or IP range from the filtering in the firewall rules.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Apr 5, 2015

@keiler3000: Yeah, I have no problems with adding another check thingy for that.

Owner

teffalump commented Apr 5, 2015

@keiler3000: Yeah, I have no problems with adding another check thingy for that.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Apr 5, 2015

@keiler3000: Added an ip range exemption. I don't know if it will work, lol.

Owner

teffalump commented Apr 5, 2015

@keiler3000: Added an ip range exemption. I don't know if it will work, lol.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Apr 12, 2015

Hello,

@teffalump: privoxy can't run as intercepting/transparent proxy for port 443 (HTTPS) though, so you can't block https sites, and you can't filter YouTube ads with it since YouTube is https (though dnsmasq can't either, of course). Anyway, I need a solution that works out of the box for every device connected, since I want to keep my net clean. dnsmasq is way simpler in this regard, and easy to script (I wrote my own implementation, resulting in more than 70000 entries -- works without any problems on a WDR4300). So we're left with null routing those malware and ad domains if dnssec/dnscrypt becomes a reality; iptables is too slow for several thousand sites, I'm afraid. Any other alternative?

Great work on your script, though, it was nice to see a different implementation.

ghost commented Apr 12, 2015

Hello,

@teffalump: privoxy can't run as intercepting/transparent proxy for port 443 (HTTPS) though, so you can't block https sites, and you can't filter YouTube ads with it since YouTube is https (though dnsmasq can't either, of course). Anyway, I need a solution that works out of the box for every device connected, since I want to keep my net clean. dnsmasq is way simpler in this regard, and easy to script (I wrote my own implementation, resulting in more than 70000 entries -- works without any problems on a WDR4300). So we're left with null routing those malware and ad domains if dnssec/dnscrypt becomes a reality; iptables is too slow for several thousand sites, I'm afraid. Any other alternative?

Great work on your script, though, it was nice to see a different implementation.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Apr 15, 2015

@1ems: Hiya! Yeah, what I meant was that over tls, the interception/filtering could not happen on the router. For example, I run dnscrypt + dnsmasq + polipo + adblock (in the browser), so I can still filter https sites but it must happen after the tls is stripped. I'm not completely understanding what you are saying about alternatives, the basic config for the script uses dnsmasq to null route. Or you mean that iptables intercepts the requests? Or? I hope that dnssec/crypt do become a reality! =-) Thanks!

Owner

teffalump commented Apr 15, 2015

@1ems: Hiya! Yeah, what I meant was that over tls, the interception/filtering could not happen on the router. For example, I run dnscrypt + dnsmasq + polipo + adblock (in the browser), so I can still filter https sites but it must happen after the tls is stripped. I'm not completely understanding what you are saying about alternatives, the basic config for the script uses dnsmasq to null route. Or you mean that iptables intercepts the requests? Or? I hope that dnssec/crypt do become a reality! =-) Thanks!

@aleblack

This comment has been minimized.

Show comment
Hide comment
@aleblack

aleblack Apr 18, 2015

@teffalump in order to iptables range to work you need to check for "xt_iprange" module, and if it's not loaded, install iptables-mod-iprange and modprobe xt_iprange, or iptables will simply ignore the extemption rules.

@teffalump in order to iptables range to work you need to check for "xt_iprange" module, and if it's not loaded, install iptables-mod-iprange and modprobe xt_iprange, or iptables will simply ignore the extemption rules.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Apr 21, 2015

@aleblack: Hahah, thanks! Man, checking for these packages....uggh. =-) Unless it's a non-standard install location, I don't believe the modprobe is necessary, no?

Owner

teffalump commented Apr 21, 2015

@aleblack: Hahah, thanks! Man, checking for these packages....uggh. =-) Unless it's a non-standard install location, I don't believe the modprobe is necessary, no?

@brainstromer

This comment has been minimized.

Show comment
Hide comment
@brainstromer

brainstromer Apr 23, 2015

hello dear,
i'm noob running gargoyle 1.7.1
i've tried to add all the way you suggested
now im confuse how i can know this script working or not
any way is there a log/report error when it's not run correct?
thanks alot

hello dear,
i'm noob running gargoyle 1.7.1
i've tried to add all the way you suggested
now im confuse how i can know this script working or not
any way is there a log/report error when it's not run correct?
thanks alot

@brainstromer

This comment has been minimized.

Show comment
Hide comment
@brainstromer

brainstromer Apr 23, 2015

*edit
root@Gargoyle:~# sh /etc/adblock.sh
iptables-mod-nat-extra is installed!
Updating package list...
Installing wget (with ssl)...
ERROR: Not enough space in destination root to install specified packages:
libpthread, zlib, libopenssl, wget, librt, libpcre

Updating config, if necessary...
Downloading hosts lists...
Adding blacklist...
Sorting lists...
Filtering white list...
Cleaning up...
Restarting dnsmasq...

which one do i need fix?
im using wr740

*edit
root@Gargoyle:~# sh /etc/adblock.sh
iptables-mod-nat-extra is installed!
Updating package list...
Installing wget (with ssl)...
ERROR: Not enough space in destination root to install specified packages:
libpthread, zlib, libopenssl, wget, librt, libpcre

Updating config, if necessary...
Downloading hosts lists...
Adding blacklist...
Sorting lists...
Filtering white list...
Cleaning up...
Restarting dnsmasq...

which one do i need fix?
im using wr740

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Apr 23, 2015

Hey teffalump. What's got you not recommending the pixel serving idea?
Just out of curiosity. Maybe you've thought of something I haven't :)

Hey teffalump. What's got you not recommending the pixel serving idea?
Just out of curiosity. Maybe you've thought of something I haven't :)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Apr 24, 2015

@lantis1008: Hahah, no. I just think many people won't understand what's happening behind the scenes, and it could lead to non-obvious errors. I'll remove the recommendation. =-)

Owner

teffalump commented Apr 24, 2015

@lantis1008: Hahah, no. I just think many people won't understand what's happening behind the scenes, and it could lead to non-obvious errors. I'll remove the recommendation. =-)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Apr 24, 2015

@brainstromer: Well, sounds like there is no space left to install the packages for wget w/ ssl. Odd. What's taking so much space? Use df and du. You don't actually need wget with ssl if you use non-ssl urls. Yeah, better error handling is on my to-do list.

Owner

teffalump commented Apr 24, 2015

@brainstromer: Well, sounds like there is no space left to install the packages for wget w/ ssl. Odd. What's taking so much space? Use df and du. You don't actually need wget with ssl if you use non-ssl urls. Yeah, better error handling is on my to-do list.

@NextBlaubeere

This comment has been minimized.

Show comment
Hide comment
@NextBlaubeere

NextBlaubeere Jun 2, 2015

how do i turn it off again?

how do i turn it off again?

@norman25

This comment has been minimized.

Show comment
Hide comment
@norman25

norman25 Jun 2, 2015

@teffalump: why you use (https://adaway.org/hosts.txt).?
I'm use (http://adaway.org/hosts.txt) and works exactly in the same way!
I'm think is no needed to install wget ssl package and waste the very limited space in the router!

norman25 commented Jun 2, 2015

@teffalump: why you use (https://adaway.org/hosts.txt).?
I'm use (http://adaway.org/hosts.txt) and works exactly in the same way!
I'm think is no needed to install wget ssl package and waste the very limited space in the router!

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jun 6, 2015

@norman25: Yeah, I thought that it redirects to https. But I think you're right. Weird. Maybe they changed it? Lol. I'll update it to be optional for ssl. Thanks.

Owner

teffalump commented Jun 6, 2015

@norman25: Yeah, I thought that it redirects to https. But I think you're right. Weird. Maybe they changed it? Lol. I'll update it to be optional for ssl. Thanks.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jun 6, 2015

@manaueruel: A few changes. 1) Remove the cron entries, 2) Remove the firewall additions, 3) Remove the dhcp change, and, if necessary, 4) Undo the pixel serving. Then restart the firewall and dnsmasq (and possibly the httpd server). You think I should add an option to revert the changes? Like, ./adblock.sh -r would undo all the changes?

Owner

teffalump commented Jun 6, 2015

@manaueruel: A few changes. 1) Remove the cron entries, 2) Remove the firewall additions, 3) Remove the dhcp change, and, if necessary, 4) Undo the pixel serving. Then restart the firewall and dnsmasq (and possibly the httpd server). You think I should add an option to revert the changes? Like, ./adblock.sh -r would undo all the changes?

@norman25

This comment has been minimized.

Show comment
Hide comment
@norman25

norman25 Jun 11, 2015

@manaueruel: how do i turn it off again?
@teffalump: would undo all the changes?
I think he wants something to lock on and off adblock in the router like ad-toggle.sh (on - off)

@manaueruel: how do i turn it off again?
@teffalump: would undo all the changes?
I think he wants something to lock on and off adblock in the router like ad-toggle.sh (on - off)

@lacek

This comment has been minimized.

Show comment
Hide comment
@lacek

lacek Jun 13, 2015

If TRANS is set to "Y", there's error about missing ]:

...
Updating config, if necessary...
sh: missing ]
httpd_gargoyle found...
...

lacek commented Jun 13, 2015

If TRANS is set to "Y", there's error about missing ]:

...
Updating config, if necessary...
sh: missing ]
httpd_gargoyle found...
...
@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Jun 14, 2015

^ looks like its line 125.

^ looks like its line 125.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jun 21, 2015

@lacek, @lantis1008: Thanks!

Owner

teffalump commented Jun 21, 2015

@lacek, @lantis1008: Thanks!

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jun 21, 2015

@norman25: Hmm... The toggle thing should be possible, but I'd have to add options (I think) because the cron entry must only update the block list. For example, ./adblock.sh = toggle on/off, ./adblock.sh -u = update block list (update dnsmasq), ./adblock.sh -f = first time use (adds config, too). Or something like that. Thoughts?

Owner

teffalump commented Jun 21, 2015

@norman25: Hmm... The toggle thing should be possible, but I'd have to add options (I think) because the cron entry must only update the block list. For example, ./adblock.sh = toggle on/off, ./adblock.sh -u = update block list (update dnsmasq), ./adblock.sh -f = first time use (adds config, too). Or something like that. Thoughts?

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jun 21, 2015

Just added some command line options. -f is for running it the first time, -t toggles it on/off, and default just updates the blocklist. Separated code into functions. Gotta test it, though. Lol.

Owner

teffalump commented Jun 21, 2015

Just added some command line options. -f is for running it the first time, -t toggles it on/off, and default just updates the blocklist. Separated code into functions. Gotta test it, though. Lol.

@norman25

This comment has been minimized.

Show comment
Hide comment
@norman25

norman25 Jun 28, 2015

@teffalump:
i think this is not more Simple OpenWRT adblock implementation is more a Advanced OpenWRT adblock implementation right now!! jaja!!
fanstastic work man!!

@teffalump:
i think this is not more Simple OpenWRT adblock implementation is more a Advanced OpenWRT adblock implementation right now!! jaja!!
fanstastic work man!!

@hunsaman

This comment has been minimized.

Show comment
Hide comment
@hunsaman

hunsaman Jun 29, 2015

Hi!
Sorry my bad english!

When i run first your script (on my gargoyle router), the ip in the host file is: 192.168.1.1. It's OK (TRANS=Y in my config).
When i update the blocklist manually, the ip in the hostfile is: 0.0.0.0
Why? is this ok?

Hi!
Sorry my bad english!

When i run first your script (on my gargoyle router), the ip in the host file is: 192.168.1.1. It's OK (TRANS=Y in my config).
When i update the blocklist manually, the ip in the hostfile is: 0.0.0.0
Why? is this ok?

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jun 29, 2015

@hunsaman: If TRANS=Y, then IP should be 192.168.1.1, and the update should keep the ip as 192.168.1.1. I think it is a bug. Let me look into it. Thanks.

UPDATE: I fixed the issue (I'll do more testing, too). Basically, the update_blocklist was not using the correct endpoint_ip6 and endpoint_ip4 values.

Owner

teffalump commented Jun 29, 2015

@hunsaman: If TRANS=Y, then IP should be 192.168.1.1, and the update should keep the ip as 192.168.1.1. I think it is a bug. Let me look into it. Thanks.

UPDATE: I fixed the issue (I'll do more testing, too). Basically, the update_blocklist was not using the correct endpoint_ip6 and endpoint_ip4 values.

@hunsaman

This comment has been minimized.

Show comment
Hide comment
@hunsaman

hunsaman Jun 30, 2015

Easier way:

# Redirect endpoint
if [ "$TRANS" == "Y" ] && [ -e "/www/1.gif" ] && ([ -s "/usr/sbin/uhttpd" ] || [ -s "/usr/sbin/httpd_gargoyle" ]); then
    ENDPOINT_IP4=$(uci get network.lan.ipaddr)
    if [ "$IPV6" == "Y" ]; then
        ENDPOINT_IP6=$(uci get network.lan6.ipaddr)
    fi
else
    ENDPOINT_IP4="0.0.0.0"
    ENDPOINT_IP6="::"
fi

Easier way:

# Redirect endpoint
if [ "$TRANS" == "Y" ] && [ -e "/www/1.gif" ] && ([ -s "/usr/sbin/uhttpd" ] || [ -s "/usr/sbin/httpd_gargoyle" ]); then
    ENDPOINT_IP4=$(uci get network.lan.ipaddr)
    if [ "$IPV6" == "Y" ]; then
        ENDPOINT_IP6=$(uci get network.lan6.ipaddr)
    fi
else
    ENDPOINT_IP4="0.0.0.0"
    ENDPOINT_IP6="::"
fi
@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jun 30, 2015

@norman25: Haha, that's probably true. Its getting a little complex, lol. But, the simplest use is pretty easy to use, so... hehe. Maybe I'll update description. =-)

Owner

teffalump commented Jun 30, 2015

@norman25: Haha, that's probably true. Its getting a little complex, lol. But, the simplest use is pretty easy to use, so... hehe. Maybe I'll update description. =-)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jul 3, 2015

@hunsaman: Yeah, could do it that way, more checks, etc. Although the 'else' clause is overwriting the config value which is a no-no =-)

UPDATE: I added your checks without the else clause. =-)

Owner

teffalump commented Jul 3, 2015

@hunsaman: Yeah, could do it that way, more checks, etc. Although the 'else' clause is overwriting the config value which is a no-no =-)

UPDATE: I added your checks without the else clause. =-)

@sleepy1

This comment has been minimized.

Show comment
Hide comment
@sleepy1

sleepy1 Jul 5, 2015

Hey, nice script. Just one question, how does the only use wifi blocking work. It seems to block ads on all interfaces even with this option turned on.

sleepy1 commented Jul 5, 2015

Hey, nice script. Just one question, how does the only use wifi blocking work. It seems to block ads on all interfaces even with this option turned on.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jul 5, 2015

@sleepy1: Hehe, it's suppose to work! Ideally, the firewall rule should only redirect DNS requests on wlan+ interface(s), that is, wireless interfaces. However, I have heard of issues. I don't really know why it doesn't work sometimes, maybe there is bridging, misleading names, etc. Any suggestions are welcome. Also, look in /etc/firewall.user for multiple conflicting rules - there should be only two DNS rules, one for udp, one for tcp. Thanks! =-)

Owner

teffalump commented Jul 5, 2015

@sleepy1: Hehe, it's suppose to work! Ideally, the firewall rule should only redirect DNS requests on wlan+ interface(s), that is, wireless interfaces. However, I have heard of issues. I don't really know why it doesn't work sometimes, maybe there is bridging, misleading names, etc. Any suggestions are welcome. Also, look in /etc/firewall.user for multiple conflicting rules - there should be only two DNS rules, one for udp, one for tcp. Thanks! =-)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jul 5, 2015

Updated to use '=' instead of '==' for portability reasons. Although, I'm unsure if it's necessary.

Owner

teffalump commented Jul 5, 2015

Updated to use '=' instead of '==' for portability reasons. Although, I'm unsure if it's necessary.

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Jul 6, 2015

I finished my plugin based on your work mate, sorry i forgot to tell you!
You can check it out here:
http://www.gargoyle-router.com/phpbb/viewtopic.php?f=14&t=7249

i have of course credited you in the post and the source code. Many thanks.

I finished my plugin based on your work mate, sorry i forgot to tell you!
You can check it out here:
http://www.gargoyle-router.com/phpbb/viewtopic.php?f=14&t=7249

i have of course credited you in the post and the source code. Many thanks.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jul 6, 2015

@lantis1008: AWESOME! SO AWESOME!!! =-) Great work! Sorry if I'm missing something, but is there a place for development? Like, push onto gh, bb, etc? =-)

Owner

teffalump commented Jul 6, 2015

@lantis1008: AWESOME! SO AWESOME!!! =-) Great work! Sorry if I'm missing something, but is there a place for development? Like, push onto gh, bb, etc? =-)

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Jul 6, 2015

I've got a private git setup for it. In the next week or so I'll probably push it to master to make it public. I keep a pretty close eye on this thread anyway so I haven't missed anything yet :)

I've got a private git setup for it. In the next week or so I'll probably push it to master to make it public. I keep a pretty close eye on this thread anyway so I haven't missed anything yet :)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jul 9, 2015

@lantis1008: Hehe, great to hear! I was more asking for some of the other people who have made suggestions and comments (seems like many run gargoyle) and may want to help with the plugin. =-)

Owner

teffalump commented Jul 9, 2015

@lantis1008: Hehe, great to hear! I was more asking for some of the other people who have made suggestions and comments (seems like many run gargoyle) and may want to help with the plugin. =-)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jul 9, 2015

Added option "-r" to fully reinstall since some config changes require it.

Owner

teffalump commented Jul 9, 2015

Added option "-r" to fully reinstall since some config changes require it.

@miniminime

This comment has been minimized.

Show comment
Hide comment
@miniminime

miniminime Jul 16, 2015

Hi teffalump, love your script, works very well and is quite elegant solution. What I miss is the possibility to import my adblock custom rules. Do you think it might be possible to integrate this syntax to your script so we can add adblock rules to the "black.list"? => https://adblockplus.org/en/filters

Hi teffalump, love your script, works very well and is quite elegant solution. What I miss is the possibility to import my adblock custom rules. Do you think it might be possible to integrate this syntax to your script so we can add adblock rules to the "black.list"? => https://adblockplus.org/en/filters

@roger-

This comment has been minimized.

Show comment
Hide comment
@roger-

roger- Jul 17, 2015

@teffalump

Have you considered submitting this as an OpenWrt package?

roger- commented Jul 17, 2015

@teffalump

Have you considered submitting this as an OpenWrt package?

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jul 20, 2015

@miniminime: Put simply, no. =-/ Those rules match against page objects, like '/ad-plugin.js'. This setup can't do that, it just isn't working at that level. You'd have to run something like privoxy on the router, MITMing SSL connections, etc. I think that more granular blocking should be done on the computer (e.g., privoxy) or in the browser (e.g., ublock/adb/etc).

@roger-: I had briefly looked at it, but was somewhat scared about my lack of knowledge. If you are game, feel free to do it. =-) I'll look at it more because there does seem to be a good amount of interest. And now with the gargoyle plugin... =-)

Owner

teffalump commented Jul 20, 2015

@miniminime: Put simply, no. =-/ Those rules match against page objects, like '/ad-plugin.js'. This setup can't do that, it just isn't working at that level. You'd have to run something like privoxy on the router, MITMing SSL connections, etc. I think that more granular blocking should be done on the computer (e.g., privoxy) or in the browser (e.g., ublock/adb/etc).

@roger-: I had briefly looked at it, but was somewhat scared about my lack of knowledge. If you are game, feel free to do it. =-) I'll look at it more because there does seem to be a good amount of interest. And now with the gargoyle plugin... =-)

@roger-

This comment has been minimized.

Show comment
Hide comment
@roger-

roger- Jul 20, 2015

@teffalump

I'll have a look and see how that might work. Until then I'd suggest moving this to GitHub to make collaboration easier.

roger- commented Jul 20, 2015

@teffalump

I'll have a look and see how that might work. Until then I'd suggest moving this to GitHub to make collaboration easier.

@hansvdmarel

This comment has been minimized.

Show comment
Hide comment
@hansvdmarel

hansvdmarel Jul 22, 2015

Hi Teffalump,

Awesome piece of software you created. Many thanks.

However, for some reason it is not working for me. I've installed the scripts and ran the first install part.
I ran into storage issues on the main flash, so I changed the scripts to work with my USB disk instead.

The firewall.user is changed
The block.list is added to dhcp config
The dnsmasq service was restarted
The firewall was restarted
I removed my dns server settings and put them in default state

The client pc has my router's IP as DNS server address.

When I open analytics.google.com (in Chrome), it just opens the site even though the block.list says that it should be routed to 0.0.0.0

It seems that dnsmasq is not processing the adblock stuff at all.

Any ideas what I can check?

Hi Teffalump,

Awesome piece of software you created. Many thanks.

However, for some reason it is not working for me. I've installed the scripts and ran the first install part.
I ran into storage issues on the main flash, so I changed the scripts to work with my USB disk instead.

The firewall.user is changed
The block.list is added to dhcp config
The dnsmasq service was restarted
The firewall was restarted
I removed my dns server settings and put them in default state

The client pc has my router's IP as DNS server address.

When I open analytics.google.com (in Chrome), it just opens the site even though the block.list says that it should be routed to 0.0.0.0

It seems that dnsmasq is not processing the adblock stuff at all.

Any ideas what I can check?

@hansvdmarel

This comment has been minimized.

Show comment
Hide comment
@hansvdmarel

hansvdmarel Jul 23, 2015

logread tells me this:
daemon.err dnsmasq[32666]: failed to load names from /mnt/usbdisk/adblock/block.hosts: Permission denied

Any thoughts on how to get around this?

logread tells me this:
daemon.err dnsmasq[32666]: failed to load names from /mnt/usbdisk/adblock/block.hosts: Permission denied

Any thoughts on how to get around this?

@hansvdmarel

This comment has been minimized.

Show comment
Hide comment
@hansvdmarel

hansvdmarel Jul 23, 2015

logread tells me this:
daemon.err dnsmasq[32666]: failed to load names from /mnt/usbdisk/adblock/block.hosts: Permission denied

Problem solved after moving the block.hosts file to the /etc folder

daemon.info dnsmasq[472]: read /etc/block.hosts - 56813 addresses

Now I need to figure out how to solve the permission issue with the USB drive.

Figured out that the file can be read by dnsmasq if the entire chain of folder names can be read by "nobody". Set file permissions to xx5.

logread tells me this:
daemon.err dnsmasq[32666]: failed to load names from /mnt/usbdisk/adblock/block.hosts: Permission denied

Problem solved after moving the block.hosts file to the /etc folder

daemon.info dnsmasq[472]: read /etc/block.hosts - 56813 addresses

Now I need to figure out how to solve the permission issue with the USB drive.

Figured out that the file can be read by dnsmasq if the entire chain of folder names can be read by "nobody". Set file permissions to xx5.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jul 23, 2015

@hansvdmarel: Sorry I didn't get back to you quickly. Also glad to hear you solved it. Folder/file permissions can be trick sometimes. =-)

Owner

teffalump commented Jul 23, 2015

@hansvdmarel: Sorry I didn't get back to you quickly. Also glad to hear you solved it. Folder/file permissions can be trick sometimes. =-)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jul 23, 2015

Owner

teffalump commented Jul 23, 2015

@hansvdmarel

This comment has been minimized.

Show comment
Hide comment
@hansvdmarel

hansvdmarel Jul 24, 2015

Another thing to add to the script is restarting cron. Otherwise it won't actually update via cron.

/etc/init.d/cron restart

Another thing to add to the script is restarting cron. Otherwise it won't actually update via cron.

/etc/init.d/cron restart

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Jul 27, 2015

^ that's not strictly true, and certainly not in this case in my experience.

^ that's not strictly true, and certainly not in this case in my experience.

@hansvdmarel

This comment has been minimized.

Show comment
Hide comment
@hansvdmarel

hansvdmarel Jul 27, 2015

Well, for OPENwrt it is really :)

Anyway, found another issue.

While updating the lists via cron, sometimes the router crashes. I think this has something to do with the device running out of memory.
I've also noticed a crash when running the script manually, so I'm quite confident that this is the issue.

I believe the issue manifests itself when the sorting is executed. How important is the sorting function for the operation? Can we do without it?

Well, for OPENwrt it is really :)

Anyway, found another issue.

While updating the lists via cron, sometimes the router crashes. I think this has something to do with the device running out of memory.
I've also noticed a crash when running the script manually, so I'm quite confident that this is the issue.

I believe the issue manifests itself when the sorting is executed. How important is the sorting function for the operation? Can we do without it?

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Jul 30, 2015

@hansvdmarel, @lantis1008: I can add the cron restart. I've had no issues, but nothing too onerous. The sorting is a unique sort... and since there is significant overlap between lists, I'm hesitant to undo that. Thoughts?

Owner

teffalump commented Jul 30, 2015

@hansvdmarel, @lantis1008: I can add the cron restart. I've had no issues, but nothing too onerous. The sorting is a unique sort... and since there is significant overlap between lists, I'm hesitant to undo that. Thoughts?

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Jul 31, 2015

Currently the sorting function also does the unique as well as far as i can figure out.
You can do a non sorting style of unique, but i don't think it is any less resource intensive.

We are talking about ~ 700kB in RAM (with ipv4 only) currently (unsorted list + sorted unique list). ipv6 is probably double this approximately (bit less because of only unique values).
~1.4MB of RAM... for me personally i have plenty of space. But smaller devices might run into this issue.

1 solution might be to remove "/tmp/block.build.list" as soon as you have done the sort, effectively halving the space we are using.
i.e. ~700kB in RAM, down to ~350kB after removing "/tmp/block.build.list", then potentially doubled due to ipv6 back to ~700kB.
It's not perfect, but it is a start. Achieved by either passing a value to the cleanup function, or just straight up rm -f'ing the file, and then the cleanup function only has to remove 1 file.

Currently the sorting function also does the unique as well as far as i can figure out.
You can do a non sorting style of unique, but i don't think it is any less resource intensive.

We are talking about ~ 700kB in RAM (with ipv4 only) currently (unsorted list + sorted unique list). ipv6 is probably double this approximately (bit less because of only unique values).
~1.4MB of RAM... for me personally i have plenty of space. But smaller devices might run into this issue.

1 solution might be to remove "/tmp/block.build.list" as soon as you have done the sort, effectively halving the space we are using.
i.e. ~700kB in RAM, down to ~350kB after removing "/tmp/block.build.list", then potentially doubled due to ipv6 back to ~700kB.
It's not perfect, but it is a start. Achieved by either passing a value to the cleanup function, or just straight up rm -f'ing the file, and then the cleanup function only has to remove 1 file.

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Aug 4, 2015

@teffalump I'm currently talking to a guy who is having a conflicting issue with your script + Tor.
They create conflicting iptables rules and .onion sites no longer get resolved.

The Tor rules are:
iptables -t nat -A tor_client -p tcp --dport 53 -m string --hex-string '|xxxxxxxxxxxx|' --algo bm -j REDIRECT --to-ports 9053
iptables -t nat -A tor_client -p udp --dport 53 -m string --hex-string '|xxxxxxxxxxxx|' --algo bm -j REDIRECT --to-ports 9053

Is there any way of making these rules play nicely or are they completely incompatible?

@teffalump I'm currently talking to a guy who is having a conflicting issue with your script + Tor.
They create conflicting iptables rules and .onion sites no longer get resolved.

The Tor rules are:
iptables -t nat -A tor_client -p tcp --dport 53 -m string --hex-string '|xxxxxxxxxxxx|' --algo bm -j REDIRECT --to-ports 9053
iptables -t nat -A tor_client -p udp --dport 53 -m string --hex-string '|xxxxxxxxxxxx|' --algo bm -j REDIRECT --to-ports 9053

Is there any way of making these rules play nicely or are they completely incompatible?

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Aug 11, 2015

@lantis1008: Sorry about inactivity. IRL stuff. The sorting thing is... I can remove the old list. On the iptables rule, hmm... that's an interesting question. Those rules are doing string matches against the content, then routing them, I wonder if it is possible to similarly do a narrower DNS match for the rules. Thoughts?

PS: I'm trying to think this through, basically, we want to pass through .onion sites. Could we do an inverse string match against '.onion' sites? Like, just subsume the tor rules into the default rule, but the inverse.

PPS: E.g., -m string ! --hex-string '|xxxxxxxx|' --algo bm

Owner

teffalump commented Aug 11, 2015

@lantis1008: Sorry about inactivity. IRL stuff. The sorting thing is... I can remove the old list. On the iptables rule, hmm... that's an interesting question. Those rules are doing string matches against the content, then routing them, I wonder if it is possible to similarly do a narrower DNS match for the rules. Thoughts?

PS: I'm trying to think this through, basically, we want to pass through .onion sites. Could we do an inverse string match against '.onion' sites? Like, just subsume the tor rules into the default rule, but the inverse.

PPS: E.g., -m string ! --hex-string '|xxxxxxxx|' --algo bm

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Aug 11, 2015

@lantis1008: I added your workaround for Tor, but I'm still thinking it should be possible to work around it in the rules themselves.

Owner

teffalump commented Aug 11, 2015

@lantis1008: I added your workaround for Tor, but I'm still thinking it should be possible to work around it in the rules themselves.

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Aug 12, 2015

Yeah I'd love to see a proper workaround for it but I can't figure it out. Currently what I implemented works. But it's ugly.

Yeah I'd love to see a proper workaround for it but I can't figure it out. Currently what I implemented works. But it's ugly.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Aug 14, 2015

@lantis1008: Why don't we re-route the non-found domains to the tor port? That is, clobber the Tor rule, but then have dnsmasq forward the "dns-miss" requests through the tor client.

Owner

teffalump commented Aug 14, 2015

@lantis1008: Why don't we re-route the non-found domains to the tor port? That is, clobber the Tor rule, but then have dnsmasq forward the "dns-miss" requests through the tor client.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Aug 14, 2015

Yeah, I think what I think can be done is set dnsmasq to noresolv and server to tor DNS port. This should sit dnsmasq in front of the tor DNS resolver, but keep the caching abilities.

Owner

teffalump commented Aug 14, 2015

Yeah, I think what I think can be done is set dnsmasq to noresolv and server to tor DNS port. This should sit dnsmasq in front of the tor DNS resolver, but keep the caching abilities.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Aug 14, 2015

First attempt at working around Tor.

Owner

teffalump commented Aug 14, 2015

First attempt at working around Tor.

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Sep 4, 2015

I've sent you a PM on the gargoyle-router website regards an issue.

I've sent you a PM on the gargoyle-router website regards an issue.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Sep 10, 2015

@lantis1008: Yeah, I'm going to open an issue on the github page. Thanks! =-)

Owner

teffalump commented Sep 10, 2015

@lantis1008: Yeah, I'm going to open an issue on the github page. Thanks! =-)

@dcavni

This comment has been minimized.

Show comment
Hide comment
@dcavni

dcavni Sep 29, 2015

First off all i must say that this is excellent script. It hunts ads very well :)

I have a question regarding how this works with OpenVPN. I set up a server running on
10.8.0.0 255.255.255.0. (http://wiki.openwrt.org/doc/howto/vpn.openvpn)
It works ok, until i run the script. Then traffic destinated to internet stops working on client. Local traffic still works.

When i use the option to exclude 10.8.0.1 - 10.8.0.254 from adblock.sh then traffic destinated to internet starts to work again (with ads).

Any idea what could be wrong?

dcavni commented Sep 29, 2015

First off all i must say that this is excellent script. It hunts ads very well :)

I have a question regarding how this works with OpenVPN. I set up a server running on
10.8.0.0 255.255.255.0. (http://wiki.openwrt.org/doc/howto/vpn.openvpn)
It works ok, until i run the script. Then traffic destinated to internet stops working on client. Local traffic still works.

When i use the option to exclude 10.8.0.1 - 10.8.0.254 from adblock.sh then traffic destinated to internet starts to work again (with ads).

Any idea what could be wrong?

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Oct 1, 2015

@dcavni: Thanks! Let me see if I understand. You setup a VPN, it works great. Then the script breaks it. =-( Are you connecting from the LAN side? Or WAN? Furthermore, can you ping past the router? That is, does...

# ping 8.8.8.8

work... or does it timeout? If it works, then can you see if you can resolve...

# dig google.com

I'm wondering if there is some firewall rule conflict going on.

Owner

teffalump commented Oct 1, 2015

@dcavni: Thanks! Let me see if I understand. You setup a VPN, it works great. Then the script breaks it. =-( Are you connecting from the LAN side? Or WAN? Furthermore, can you ping past the router? That is, does...

# ping 8.8.8.8

work... or does it timeout? If it works, then can you see if you can resolve...

# dig google.com

I'm wondering if there is some firewall rule conflict going on.

@dcavni

This comment has been minimized.

Show comment
Hide comment
@dcavni

dcavni Oct 3, 2015

Well it doesn't completly brake the tunnel, local traffic still works. I did some testing now and found out, that if i make a OpenVPN connection on a computer then everything works. If i use the same config file on a phone (Android), then only local traffic works.

If i exclude VPN ip range from script, then Internet and local traffic works on a phone and also on computer as before (with ads)

Now i'm not completly shure what is wrong. Ping works but very bad. Only 3 packets out of 4 returned, I cannot try dig command on Windows.

Any sugestions?

Edit: Ok, today i did some more testing. If i change Android OpenVPN app for OpenVPN client app (Two different apps). Then web browsing works on android. It look's like original OpenVPN app doesn't work well when adblock.sh works on router. Also i see ads on phone so it look's like the script isn't blocking ads on a 10.8.0.0 / 255.255.255.0 (default OpenVPN tun).

dcavni commented Oct 3, 2015

Well it doesn't completly brake the tunnel, local traffic still works. I did some testing now and found out, that if i make a OpenVPN connection on a computer then everything works. If i use the same config file on a phone (Android), then only local traffic works.

If i exclude VPN ip range from script, then Internet and local traffic works on a phone and also on computer as before (with ads)

Now i'm not completly shure what is wrong. Ping works but very bad. Only 3 packets out of 4 returned, I cannot try dig command on Windows.

Any sugestions?

Edit: Ok, today i did some more testing. If i change Android OpenVPN app for OpenVPN client app (Two different apps). Then web browsing works on android. It look's like original OpenVPN app doesn't work well when adblock.sh works on router. Also i see ads on phone so it look's like the script isn't blocking ads on a 10.8.0.0 / 255.255.255.0 (default OpenVPN tun).

@MichiMunich

This comment has been minimized.

Show comment
Hide comment
@MichiMunich

MichiMunich Oct 4, 2015

@dcavni and @teffalump
The Problem with the non working Internet connection for OpenVPN Clients is because of a dnsmasq config. By default DNS queries from a non local subnet are not allowed. This is be done by "option localservice '1'" in /etc/config/dhcp
If you switch this value to "0" and run 2 times "sh /etc/adblock.sh -t" all VPN clients should reach the internet. You also can extemp the VPN Range in the adblock.sh script, but then adblock wont work for VPN. I described this also in my Blog Post "http://blog.doenselmann.com/werbung-direkt-auf-openwrt-router-blocken/" (but its in german).
I hope this will help you.

@dcavni and @teffalump
The Problem with the non working Internet connection for OpenVPN Clients is because of a dnsmasq config. By default DNS queries from a non local subnet are not allowed. This is be done by "option localservice '1'" in /etc/config/dhcp
If you switch this value to "0" and run 2 times "sh /etc/adblock.sh -t" all VPN clients should reach the internet. You also can extemp the VPN Range in the adblock.sh script, but then adblock wont work for VPN. I described this also in my Blog Post "http://blog.doenselmann.com/werbung-direkt-auf-openwrt-router-blocken/" (but its in german).
I hope this will help you.

@dcavni

This comment has been minimized.

Show comment
Hide comment
@dcavni

dcavni Oct 4, 2015

It works :) Just perfect. Thanks to everyone for help.

dcavni commented Oct 4, 2015

It works :) Just perfect. Thanks to everyone for help.

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Oct 5, 2015

@MichiMunich
Is that the same reason for Tor not working do you think?

@MichiMunich
Is that the same reason for Tor not working do you think?

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Oct 6, 2015

I'm trying to get a handle on what is happening...

Client ---> Router ---> Internet

  • Router runs OpenVPN server
  • Client connects
  • Client uses router DNS, but the router drops the DNS request because it has no interface with that subnet

That about right?

I can add the config setting, but I'm wondering maybe if I should instead add a separate section for this special situation. I'm cautious about this setting because it was expressly added to prevent DNS reflection attacks, which can be very problematic. To me, turning it off is not inconsequential.

Owner

teffalump commented Oct 6, 2015

I'm trying to get a handle on what is happening...

Client ---> Router ---> Internet

  • Router runs OpenVPN server
  • Client connects
  • Client uses router DNS, but the router drops the DNS request because it has no interface with that subnet

That about right?

I can add the config setting, but I'm wondering maybe if I should instead add a separate section for this special situation. I'm cautious about this setting because it was expressly added to prevent DNS reflection attacks, which can be very problematic. To me, turning it off is not inconsequential.

@MichiMunich

This comment has been minimized.

Show comment
Hide comment
@MichiMunich

MichiMunich Oct 6, 2015

@teffalump
that´s right. the wiki says, that the localservice '0' is the default value.
But i would not change the setting within your script.You can not regard all constellations.
Maybe only a hint is helpfull.

@teffalump
that´s right. the wiki says, that the localservice '0' is the default value.
But i would not change the setting within your script.You can not regard all constellations.
Maybe only a hint is helpfull.

@MichiMunich

This comment has been minimized.

Show comment
Hide comment
@MichiMunich

MichiMunich Oct 6, 2015

@lantis1008
i am not using tor with OpenWRT. So i can´t give you an answer.
But you can just try it if it works for you.

@lantis1008
i am not using tor with OpenWRT. So i can´t give you an answer.
But you can just try it if it works for you.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Oct 9, 2015

@MichiMunich: Yeah, seems that the default is 0... Weird. =-) Anyways, I think, for now, I'll take your suggestion and put a hint about VPN issues if localservice is 1. =-)

Owner

teffalump commented Oct 9, 2015

@MichiMunich: Yeah, seems that the default is 0... Weird. =-) Anyways, I think, for now, I'll take your suggestion and put a hint about VPN issues if localservice is 1. =-)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Oct 9, 2015

Just added the hint, although maybe I should stress more caution. Lol.

Owner

teffalump commented Oct 9, 2015

Just added the hint, although maybe I should stress more caution. Lol.

@frontalot

This comment has been minimized.

Show comment
Hide comment
@frontalot

frontalot Oct 9, 2015

I run a persistent OpenVPN connection on my router and ad blocking wasn't working at first. Removing the localservice 1 entry from my dnsmasq config worked.

Is there any way to make the ad blocking work without removing the localservice entry? I don't want to open myself up to security issues...

I run a persistent OpenVPN connection on my router and ad blocking wasn't working at first. Removing the localservice 1 entry from my dnsmasq config worked.

Is there any way to make the ad blocking work without removing the localservice entry? I don't want to open myself up to security issues...

@frontalot

This comment has been minimized.

Show comment
Hide comment
@frontalot

frontalot Oct 9, 2015

Also, when turning on your script, I get a warning line:

Turning on!
Updating config...
uci: Entry not found
Restarting firewall...
Restarting dnsmasq...
Restarting uhttpd...
Restarting cron...

Is the uci warning normal or expected?

Also, when turning on your script, I get a warning line:

Turning on!
Updating config...
uci: Entry not found
Restarting firewall...
Restarting dnsmasq...
Restarting uhttpd...
Restarting cron...

Is the uci warning normal or expected?

@MichiMunich

This comment has been minimized.

Show comment
Hide comment
@MichiMunich

MichiMunich Oct 9, 2015

@frontalot
Probably it is because of the command "TOR=uci get tor.global.enabled"
If you don't use TOR you get "uci: Entry not found" as result. So it's not a problem.

@frontalot
Probably it is because of the command "TOR=uci get tor.global.enabled"
If you don't use TOR you get "uci: Entry not found" as result. So it's not a problem.

@bonanza123

This comment has been minimized.

Show comment
Hide comment
@bonanza123

bonanza123 Oct 10, 2015

when I enable the 'serving with transparent pixel' option then it actually redirects to my openWrt router right? But when it does so, doesn't it interfere with the LUCI login page also listening on port 80 and 443?

when I enable the 'serving with transparent pixel' option then it actually redirects to my openWrt router right? But when it does so, doesn't it interfere with the LUCI login page also listening on port 80 and 443?

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Oct 12, 2015

@frontalot: From what I understand, as long as you take normal precautions, like not listening on external DNS port, then it shouldn't be a problem. But, yeah, I'm not too knowledgeable about this stuff. =-/

Owner

teffalump commented Oct 12, 2015

@frontalot: From what I understand, as long as you take normal precautions, like not listening on external DNS port, then it shouldn't be a problem. But, yeah, I'm not too knowledgeable about this stuff. =-/

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Oct 12, 2015

Pushed change for that Tor check. Shouldn't have that error now. =-)

Owner

teffalump commented Oct 12, 2015

Pushed change for that Tor check. Shouldn't have that error now. =-)

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Oct 12, 2015

@bonanza123: You are right. =-) Although, what happens is that the redirection has the client request a page from the router, say /advertpage.jsp, and the router cannot find it, so it serves the page_error response (which we've overriden). Consequently, there shouldn't be any problems with the normal operation of the Openwrt web server functionality. However, I have seen some issues. Generally, it's somewhat inelegant and creates subtle issues, but works most of the time. =-)

Owner

teffalump commented Oct 12, 2015

@bonanza123: You are right. =-) Although, what happens is that the redirection has the client request a page from the router, say /advertpage.jsp, and the router cannot find it, so it serves the page_error response (which we've overriden). Consequently, there shouldn't be any problems with the normal operation of the Openwrt web server functionality. However, I have seen some issues. Generally, it's somewhat inelegant and creates subtle issues, but works most of the time. =-)

@dartraiden

This comment has been minimized.

Show comment
Hide comment
@dartraiden

dartraiden Oct 22, 2015

wget -qO- http://www.mvps.org/winhelp2002/hosts.txt| awk -v r="$ENDPOINT_IP4" '{sub(/^0.0.0.0/, r)} $0 ~ "^"r' > /tmp/block.build.list
wget -qO- "http://adaway.org/hosts.txt"|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list
#wget -qO- http://www.malwaredomainlist.com/hostslist/hosts.txt|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list
#wget -qO- "http://hosts-file.net/.\ad_servers.txt"|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list

Thanks for awesome script! Why links 2 and 4 have "", but 1 and 3 are not? It is important?

wget -qO- http://www.mvps.org/winhelp2002/hosts.txt| awk -v r="$ENDPOINT_IP4" '{sub(/^0.0.0.0/, r)} $0 ~ "^"r' > /tmp/block.build.list
wget -qO- "http://adaway.org/hosts.txt"|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list
#wget -qO- http://www.malwaredomainlist.com/hostslist/hosts.txt|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list
#wget -qO- "http://hosts-file.net/.\ad_servers.txt"|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list

Thanks for awesome script! Why links 2 and 4 have "", but 1 and 3 are not? It is important?

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Oct 28, 2015

@dartraiden: Link 2 could be unquoted (I believe), but link 4 has a weird, literal backslash that must be kept uninterpreted. Although, it's disabled by default. =-)

Owner

teffalump commented Oct 28, 2015

@dartraiden: Link 2 could be unquoted (I believe), but link 4 has a weird, literal backslash that must be kept uninterpreted. Although, it's disabled by default. =-)

@uneed

This comment has been minimized.

Show comment
Hide comment
@uneed

uneed Oct 29, 2015

very useful,any plan support adblock plus's filters

uneed commented Oct 29, 2015

very useful,any plan support adblock plus's filters

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Nov 2, 2015

@uneed: Nope. Wouldn't work for a variety of reasons: resource constraints, ssl security, etc. We've talked about this before... somewhere up in the comments. Hehe.

Owner

teffalump commented Nov 2, 2015

@uneed: Nope. Wouldn't work for a variety of reasons: resource constraints, ssl security, etc. We've talked about this before... somewhere up in the comments. Hehe.

@frontalot

This comment has been minimized.

Show comment
Hide comment
@frontalot

frontalot Nov 4, 2015

I've noticed that some hosts are added to block.hosts as "0.0.0.0domainnamehere.com", with the space missing. Maybe your awk code needs to add a leading space if the hosts lists contains just domains?

I've noticed that some hosts are added to block.hosts as "0.0.0.0domainnamehere.com", with the space missing. Maybe your awk code needs to add a leading space if the hosts lists contains just domains?

@lantis1008

This comment has been minimized.

Show comment
Hide comment
@lantis1008

lantis1008 Nov 5, 2015

But none of the included lists do not have leading ip's. And looking at my own list I don't observe this behaviour at all.

But none of the included lists do not have leading ip's. And looking at my own list I don't observe this behaviour at all.

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Nov 14, 2015

Yeah, like @lantis1008 said, I don't see that behavior @frontalot. There shouldn't be naked domains on the lists. Hmm...no matter, I'll keep an eye out for it. Thanks!

Owner

teffalump commented Nov 14, 2015

Yeah, like @lantis1008 said, I don't see that behavior @frontalot. There shouldn't be naked domains on the lists. Hmm...no matter, I'll keep an eye out for it. Thanks!

@n-land

This comment has been minimized.

Show comment
Hide comment
@n-land

n-land Nov 20, 2015

@teffalump: I'd like to try this great script, what is the license? MIT? GPL?

n-land commented Nov 20, 2015

@teffalump: I'd like to try this great script, what is the license? MIT? GPL?

@illitum

This comment has been minimized.

Show comment
Hide comment
@illitum

illitum Nov 22, 2015

do u create menu position on openwrt to manage this script?

illitum commented Nov 22, 2015

do u create menu position on openwrt to manage this script?

@teffalump

This comment has been minimized.

Show comment
Hide comment
@teffalump

teffalump Nov 29, 2015

@n-land: Say GPL, and we'll go from there. Thanks! =-)

@illitum: No, I need to create the package to have easier management and options, like the menu position. But I've been somewhat lazy in that regard. =-)

Owner

teffalump commented Nov 29, 2015

@n-land: Say GPL, and we'll go from there. Thanks! =-)

@illitum: No, I need to create the package to have easier management and options, like the menu position. But I've been somewhat lazy in that regard. =-)

@remlei

This comment has been minimized.

Show comment
Hide comment
@remlei

remlei Dec 12, 2015

it would be cool if I can define a path where to store the blacklist.host file instead of storing it on my router /etc folder and make my router's flash goes bad more quickly.

I use a adblock on tomato but I can define to store the host files on my RAM, which is cool.

Yeah yeah, I know, I can edit the source if I want, anyway I hope you can work on that.

remlei commented Dec 12, 2015

it would be cool if I can define a path where to store the blacklist.host file instead of storing it on my router /etc folder and make my router's flash goes bad more quickly.

I use a adblock on tomato but I can define to store the host files on my RAM, which is cool.

Yeah yeah, I know, I can edit the source if I want, anyway I hope you can work on that.

@b4hr4m

This comment has been minimized.

Show comment
Hide comment
@b4hr4m

b4hr4m Dec 17, 2015

Is this just work on incoming traffic from the WAN port? I'm using my phone to tethering via USB port and I still see the ads.

b4hr4m commented Dec 17, 2015

Is this just work on incoming traffic from the WAN port? I'm using my phone to tethering via USB port and I still see the ads.