Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
setup shorewall for docker networking beyond the default bridge network, e.g., for docker-compose

Docker(-compose) with shorewall

The shorewall documentation explains in http://shorewall.org/Docker.html how to configure shorewall for use with docker. The problem with the configuration is that it only allows connections from the host to the main bridge docker0. Connections to other networks on dynamically created bridges, with names starting by default with br-, is blocked. Instead of the recommended contents of /etc/shorewall/interfaces, use wild-card interface names as follows:

#ZONE	INTERFACE	OPTIONS
#dock	docker0		bridge     # disabled default recommendation
dock 	docker0		physical=docker+,routeback=1
dock 	br		physical=br-+,routeback=1

This declares interfaces with names starting with docker, including the default docker0, and starting with br- to be in the dock zone.

For the rest of the configuration, follow the shorewall documentation as is.

This setup fixes problems running composite apps set up manually or using docker-compose.

###############################################################################
?FORMAT 2
###############################################################################
#ZONE INTERFACE OPTIONS
net eth0 dhcp,tcpflags,nosmurfs,routefilter,logmartians,sourceroute=0
#dock docker0 bridge
dock docker0 physical=docker+,routeback=1
dock br physical=br-+,routeback=1
# and others ...
###############################################################################
#SOURCE DEST POLICY LOG LEVEL LIMIT:BURST
$FW net ACCEPT
dock $FW REJECT
dock all ACCEPT
# Some applications or use case require uncommenting the next line
$FW dock ACCEPT
# THE FOLLOWING POLICY MUST BE LAST
all all REJECT info
# apply the change below, as
# ...
# LN:
#DOCKER=No
DOCKER=Yes
# ...
###############################################################################
#ZONE TYPE OPTIONS IN OUT
# OPTIONS OPTIONS
fw firewall
dock ipv4
# and others ...
@ambis

This comment has been minimized.

Copy link

@ambis ambis commented Sep 15, 2019

Oh my god, thank you for this! Those wildcards... 👍

@tmihalik

This comment has been minimized.

Copy link

@tmihalik tmihalik commented Oct 4, 2019

Thank you, it was a great help for me too 👏👍

@ras-martin

This comment has been minimized.

Copy link

@ras-martin ras-martin commented Nov 18, 2019

Thank you 😀
I've added an additional rules (see below), because a MySQL server is running on the host machine and the containers have to connect to this MySQL instance.

File /etc/shorewall/rules:

ACCEPT          dock          $FW             tcp             mysql
@MichaelUray

This comment has been minimized.

Copy link

@MichaelUray MichaelUray commented Feb 19, 2020

I actually configured it how you described, but for some reason shorewall removes some of the docker rules on a restart.

root@dk1:~# iptables -L -v | grep DOCKER
5427 2371K DOCKER-USER all -- any any anywhere anywhere
5427 2371K DOCKER-ISOLATION-STAGE-1 all -- any any anywhere anywhere
0 0 DOCKER all -- any docker0 anywhere anywhere
177 10552 DOCKER all -- any br-61206706fa14 anywhere anywhere
1615 282K DOCKER all -- any any anywhere anywhere
Chain DOCKER (3 references)
Chain DOCKER-USER (1 references)
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 anywhere anywhere
432 126K DOCKER-ISOLATION-STAGE-2 all -- br-61206706fa14 !br-61206706fa14 anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (2 references)

systemctl restart shorewall.service

root@dk1:~# iptables -L -v | grep DOCKER
124 56242 DOCKER-USER all -- any any anywhere anywhere
124 56242 DOCKER-ISOLATION all -- any any anywhere anywhere
0 0 DOCKER all -- any docker0 anywhere anywhere
21 2308 DOCKER all -- any any anywhere anywhere
Chain DOCKER (2 references)
Chain DOCKER-ISOLATION (1 references)
Chain DOCKER-USER (1 references)

root@dk1:~# apt-show-versions shorewall docker-ce
docker-ce:amd64/buster 5:19.03.6~3-0~debian-buster uptodate
shorewall:all/buster 5.2.3.2-1 uptodate

So far as I understand should this issue already be fixed since 5.2.1.1, but I am still facing it.

Any idea what could be wrong?

@ambis

This comment has been minimized.

Copy link

@ambis ambis commented Feb 19, 2020

@MichaelUray Do you have DOCKER=Yes in /etc/shorewall/shorewall.conf?

@MichaelUray

This comment has been minimized.

Copy link

@MichaelUray MichaelUray commented Feb 19, 2020

Yes, I have.
root@dk1:~# cat /etc/shorewall/shorewall.conf | grep DOCKER
DOCKER=Yes

@MichaelUray

This comment has been minimized.

Copy link

@MichaelUray MichaelUray commented Feb 19, 2020

These are the other relevant parts of my configuration, maybe I miss something?

root@dk1:~# cat /etc/shorewall/interfaces
net ens3 - routefilter,tcpflags
vpn tun0
dock docker0 - physical=docker+,routeback=1
dock br - physical=br-+,routeback=1

root@dk1:~# cat /etc/shorewall/interfaces
net ens3 - routefilter,tcpflags
vpn tun0
dock docker0 - physical=docker+,routeback=1
dock br - physical=br-+,routeback=1

root@dk1:~# cat /etc/shorewall/policy
net fws DROP
fws all ACCEPT
vpn fws ACCEPT
dock fws REJECT
dock all ACCEPT
all all DROP

@ras-martin

This comment has been minimized.

Copy link

@ras-martin ras-martin commented Feb 19, 2020

I don't know if this helps you @MichaelUray, but I experienced that the order of starting Shorewall and Docker is important (also when restarting one of them). If I remember right, start Shorewall first, then Docker. Try to verify this.

@MichaelUray

This comment has been minimized.

Copy link

@MichaelUray MichaelUray commented Feb 19, 2020

You are right, it helps to restart docker after shorewall was restarted, but I actually don't want to restart Docker after every change on the firewall settings.

@lukasnellen

This comment has been minimized.

Copy link
Owner Author

@lukasnellen lukasnellen commented Feb 19, 2020

I don't know of a way to avoid restarting docker after restarting shorewall. I made it a habit to restart both together - but I agree it is a minor pain. There seems to be a way to tell systemd to restart docker automatically after restarting shorewall, got to look into the configuration changes required. This will still restart all running containers, though.

@MichaelUray

This comment has been minimized.

Copy link

@MichaelUray MichaelUray commented Feb 20, 2020

This will still restart all running containers, though.

Thank you, this is what I actually want to prevent.

@lukasnellen

This comment has been minimized.

Copy link
Owner Author

@lukasnellen lukasnellen commented Feb 20, 2020

If configured with the DYNAMIC_BLACKLIST=Yes option, you can use shorewall open, shorewall close and other commands for non-persistent changes (see https://shorewall.org/manpages/shorewall.html). You still have to update the config files to be sure changes are persistent. But the persistent rules won't get verified until you restart. I recommend to shorewall compile, which catches syntax errors but won't change the active rules.

In practice, I find firewall rules pretty stable and the restart issues don't arise often. Using the shorewall commands is OK for testing which ports are needed when setting up new apps. I would still do one restart with the persistent configuration at the end. Bugs that won't show up until the next reboot are worse than one container restart.

@zhaofeng-shu33

This comment has been minimized.

Copy link

@zhaofeng-shu33 zhaofeng-shu33 commented Feb 25, 2020

Thank you, it works.

@rogerniesten

This comment has been minimized.

Copy link

@rogerniesten rogerniesten commented Feb 28, 2020

(I think my issue is related to the ones mentioned above, so I'll add my issue here)

I'm using Docker 19.03.6 and shorewall 5.1.12.2 and encountered situations where (creating and) starting a docker container caused the following error:

Creating network "acc_default" with the default driver
ERROR: unable to insert jump to DOCKER-ISOLATION-STAGE-1 rule in FORWARD chain: (iptables failed: iptables --wait -I FORWARD -j DOCKER-ISOLATION-STAGE-1: iptables v1.6.1: Couldn't load target `DOCKER-ISOLATION-STAGE-1':No such file or directory

How to reproduce:
$ sudo service shorewall stop
$ suso service docker stop
$ sudo docker start
$ sudo shorewall start
<< when checking iptables -L, chain DOCKER is present, chains DOCKER-ISOLATION-STAGE-1 and DOCKER-ISOLATION-STAGE-2 are not! >>
$ docker-compose up -d
<< now the error as mentioned above occurs >>

I noticed that the chains DOCKER-ISOLATION-STAGE-1 and DOCKER-ISOLATION-STAGE-2 (including references in other chains) were removed after the command "shorewall restart". As they didn't contain any rules and everything kept working (also after stopping starting containers), I didn't see this as an issue. However, when starting a new container (e.g. after it has been removed) it will fail with the error mentioned above.
When executing shorewall with the trace option, I can see mentioned DOCKER-INGRESS and DOCKER-ISOLATION, but never DOCKER-ISOLATION-STAGE-1 or DOCKER-ISOLATION-STAGE-2.

With some further test I discovered that without the DOCKER-ISOLATION-STATE-x chains, I am able to start any docker container, but can NOT create a docker network!

So my questions are:

  1. Has docker changed their iptables (are the ...-STAGE-.. chains new) and is shorewall not ready for them yet?
  2. If so, when is a new version of shorewall to be expected?
  3. Most important: what do I need to do to allow shorewall to modify iptables without having to restart the docker daemon? (we are using mrlesmithjr/ansible-shorewall to configure shorewall via ansible)
@lukasnellen

This comment has been minimized.

Copy link
Owner Author

@lukasnellen lukasnellen commented Mar 3, 2020

In my experience, shorewall has to be started before other tools that manipulate the iptables. On Debian, I can verify that the dependencies in the systemd unit files are set up in a way that makes sure that shorewall starts before docker. If you change the order of startup to

$ sudo shorewall start
$ sudo docker start

things should work as expected.

BTW: If you have other services manipulating iptables, e.g., fail2ban, make sure they get restarted after shorewall gets restarted.

@rogerniesten

This comment has been minimized.

Copy link

@rogerniesten rogerniesten commented Mar 3, 2020

Thanks for your feedback, @lukasnellen.
I was under the impression that shorewall would preserve docker related chains and rules when being restarted, but apparently it doesn't :-(
That means a firewall modification will still require a restart of the docker daemon (which means an outage for our production services).

I was able to re-create the Docker Isolation chains, but the Docker chain itself is a bit more tricky...

@lukasnellen

This comment has been minimized.

Copy link
Owner Author

@lukasnellen lukasnellen commented Mar 3, 2020

In practice, the problem is that restarting docker restarts all running containers. This makes frequent changes (or trying changes) on a production machine difficult. This is not a problem for my setup here, where changes are limited (and I have a non-production machine for trying out things). This meant I never had to look into the use of shorewall open and shorewall close as means of run-time modification of shorewall's configuration (more in a comment higher up).

@aleqcz

This comment has been minimized.

Copy link

@aleqcz aleqcz commented Mar 22, 2020

Thank you very much

@schorschl

This comment has been minimized.

Copy link

@schorschl schorschl commented Mar 31, 2020

Thanks for your helpful hints.
I am using docker-compose and shorewall. When setting rules in /etc/shorewall/rules they are not triggered.
To be able to restrict access to the docker machines I need to insert rules into the "DOCKER-USER" chain. Any idea how I can manage that with shorewall?

@lukasnellen

This comment has been minimized.

Copy link
Owner Author

@lukasnellen lukasnellen commented Mar 31, 2020

Instead of using the DOCKER-USER chain, I set up rules in shorewall that filter connections to or from the dock zone. For more fine-grained control, you can use some of docker's advanced options:

  • create separate networks, maybe with a named bridge so you can create a special zone on that bridge
  • set the IP of the containers in docker run or in docker-compose.yml
  • user shorewall's dynamic zones (https://shorewall.org/Dynamic.html), put your container's IP into the correct, dynamic zone, and filter on those zones.

The point here is to control the interface or IP address used by a container. That way, you can write specific filter rules in shorewall w/o having to inspect the container's configuration, which might change when the container gets re-created. This is very similar to how I set up filtering for full-blown virtual machines.

The alternative is to use iptables to manipulate the DOCKER-USER table, which is not managed directly by shorewall.

@phlax

This comment has been minimized.

Copy link

@phlax phlax commented Apr 7, 2020

related to above comments about iptables rules on debian, it seems that the versions of shorewall and docker in debian stable and above do not work (at least not without always restarting docker after shorewall is restarted)

i rebuilt the debian shorewall packages (shorewall + shorewall-core) with the 5.2.4 version and this fixed the problem

i have raised an issue on the shorewall gitlab issue tracker

https://gitlab.com/shorewall/debian/-/issues/1

@interso

This comment has been minimized.

Copy link

@interso interso commented Aug 4, 2020

Thank you very much.

@grharry

This comment has been minimized.

Copy link

@grharry grharry commented Oct 31, 2020

Great post!
However in my case it's missing a vital detail !
I disabled docker iptables in /etc/docker/daemon.json
{
"iptables":false
}
And after that all works Great! Docker containers are accessible from net if enabled in shorewall ( rules, zones, policy ) etc !!
Regards,
Harry

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment