I don't expect moby/moby#24170 to be solved by Docker team on near future so this gist is about looking for least ugly workarounds which are available today and on near future.
It is possible to create overlay network with use user specified subnet. On this example I create network which is size is two C -class networks ( IP range 10.0.0.0 - 10.0.1.255 ) and force Docker to use IPs from second part of it ( 10.0.1.0 - 10.0.1.255 ).
That way I can make sure that IPs 10.0.0.2 - 10.0.0.254 can be specified to containers and they do not collide with IPs which Docker engine assign for service(s)/container(s).
docker network create --driver overlay --subnet 10.0.0.0/23 --ip-range 10.0.1.0/24 --gateway 10.0.0.1 --attachable test
By default containers IP addresses are controlled by Docker engine and those cannot be changed inside of container.
Luckily that behavior can be overridden by adding NET_ADMIN
capability for container.
So first we need create custom docker image which is able set IP for container when it starts. That we can do example with Dockerfile like this:
FROM ubuntu
RUN apt-get update \
&& apt-get install -y net-tools iputils-ping
COPY /start.sh /
ENTRYPOINT /start.sh
and start script like this:
#!/bin/bash
if [[ -f "${STATIC_IP}" ]]; then
echo "Using default IP from Docker"
else
echo "Found static IP: ${STATIC_IP} using it"
ifconfig eth0 ${STATIC_IP} netmask 255.255.255.0 up
fi
sleep infinity
Then just build that with command:
docker build . -t static-ip
Then we can start two containers with commands like these which connects then to overlay network and changes IP addresses inside of the to static ones.
docker run --name test1 --detach=true --rm --cap-add=NET_ADMIN --network=test -e STATIC_IP=10.0.0.11 static-ip
docker run --name test2 --detach=true --rm --cap-add=NET_ADMIN --network=test -e STATIC_IP=10.0.0.12 static-ip
Above solution solves two issues:
- It sets static IPs for containers.
- It allow containers communicate between Docker nodes using overlay network.
However those containers are not swarm services which means that you lose some control to them. Luckily I have already implemented docker/cli#1940 and I'm expecting that it will get merged and released as part of next Docker version.
That makes possible to use swarm stack like this:
version: "3.7"
services:
test1:
image: static-ip
environment:
STATIC_IP: 10.0.0.11
networks:
- test
cap_add:
- NET_ADMIN
test2:
image: static-ip
environment:
STATIC_IP: 10.0.0.12
networks:
- test
cap_add:
- NET_ADMIN
networks:
test:
external: true
name: test
and deploy it with command:
docker stack deploy -c static-ip.yml static-ip
I was trying to use this but it does not work. Docker creates an overlay network and one is able to see the assigned IP addresses once the stack got deployed with "docker network inspect NAME". If you use the proposed method from above, the IP is changed inside the container, without updating the mapping assigned by the docker engine. Therefore, networking works fine in the overlay network as long as you remain on one node, but if you want to establish inter-node communication via the overlay network, packets are not forwarded.
I was doing the following inside my docker-compose.yml before:
command:
- /bin/sh
- -c
- |
ip addr flush dev eth0
ip addr add 172.30.0.101/15 dev eth0
cap_add:
- NET_ADMIN
It is essentially the same that you are doing via the environment variable. Therefore, also the result is the same. The IP is changed but then communication only works for containers within the node, not inbetween nodes (which is exactly what the overlay is supposed to be for).
I could still not find a working solution - how to assign a static IP to a docker swarm container in the docker-compose.yml while maintaining inter-node communication?
I am only spawning one instance per container and believe there should be such a feature. That it does not make sense for multiple replicas is obvious, but for just one instance one should be able to assign a static IP.