Skip to content

Instantly share code, notes, and snippets.

@nerdalert
Last active September 26, 2024 05:40
Show Gist options
  • Save nerdalert/3d2b891d41e0fa8d688c to your computer and use it in GitHub Desktop.
Save nerdalert/3d2b891d41e0fa8d688c to your computer and use it in GitHub Desktop.

Experimental Docker Libnetwork DHCP Driver

The DHCP driver is intended for users to be able to integrate Docker IP address management with their existing IPAM strategies that use DHCP for dynamic address assignment. DHCP enables users to allocate addresses in an organized fashion that will prevent overlapping IP address assignment by associating a unique MAC address from the container eth0 Ethernet interface to an IP address as determined by the DHCP pools defined in the DHCP configuration.

This driver only provides the DHCP client functionality. It does not include a DHCP server. The default driver offers single-host IPAM or for distributed multi-host orchestrated IPAM see the libnetwork overlay driver.

Getting Started

Example DHCP Driver Client Usages

By default, the DHCP client driver will automatically probe for a network that eth0 is attached to by using a DHCP DISCOVER broadcast. If there is a DHCP server on the broadcast domain or handled by a relay/helper agent the existing DHCP server will reply with details about the available network that exists. The driver will then create the Docker IPAM pool and network for the user using the discovered information. From there, a user needs to simply start containers and receive IP addresses from the DHCP server.

The DHCP driver creates the DHCP requests using the MAC address of the container. The DHCP server will keep track of available addresses in the pool to ensure there are no overlapping addresses for all of the different Docker hosts running on the same Ethernet segment. Users can leverage their DHCP server of choice and monitor address allocations of all containers from the DHCP server's interface. When containers are removed docker rm <container_id>, the DHCP client driver sends a DHCP release out the specified --ipam-opt dhcp_interface=<host interface> and returns the IP address back to the DHCP server's address pool.

Note: If running Docker nested in a hypervisor, the VM's NIC needs to be in promiscous mode just as it would if you were running a nested hypervisor. This means the host Ethernet device can look at traffic with destination MAC addresseses that are different from its own MAC address. Since the DHCP client driver will be sending DHCP requests on behalf of the MAC address of a container, the host needs to be able to process those packets with differing MAC addresses. Along with various DHCP troubleshooting tools for Linux that are generally available, tcpdump can be used to troubleshoot filtering on the relevant UDP ports sudo tcpdump -i eth0 -n udp -v portrange 67-68. Tcpdump on a production network can be extremely chatty so use with care if not experienced with troubleshooting network issues.

  • The following example discovers both the subnet and gateway of the host. It does not discover any other DHCP options such as DNS or any other fields that are plumbed into the container. Any other options discovered are used by the driver in order to provide an easy network environment to manage.
# Create a network using eth0 as the parent and interface to discover what subnet to use for the network eth0 is attached
docker network create -d macvlan \
  --ipam-driver=dhcp \
  -o parent=eth0 \
  --ipam-opt dhcp_interface=eth0 mcv0

# When containers are created, DHCP requests will assign an IP addresses to each container
docker run --net=mcv0 -itd alpine /bin/sh
docker run --net=mcv0 -itd alpine /bin/sh

# When the containers are destroyed, the DHCP client driver sends a DHCP release and returns the IP address back to the DHCP servers address pool
docker rm -f `docker ps -qa`
docker network rm mcv0

Users can also explicitly specify the network's address pool --subnet= and --gateway= rather then the DHCP client driver discovering the subnet for them. Either way, when the containers are started, DHCP requests are sent by the driver when containers are started. The specified --subnet must match the subnet the DHCP server is handing out or else the driver will prevent a container creation (docker run) since the network and containers on the network need to coincide.

docker network create -d macvlan \
  --ipam-driver=dhcp \
  --subnet=172.16.86.0/24 \
  --gateway=172.16.86.2 \
  -o parent=eth0 \
  --ipam-opt dhcp_interface=eth0 mcv0

docker run --net=mcv0 -itd alpine /bin/sh
docker run --net=mcv0 -itd alpine /bin/sh

docker rm -f `docker ps -qa`
docker network rm mcv0
  • If a gateway is not specified, the driver will infer the gateway by using the first usable address on the user specified --subnet.
docker network create -d macvlan \
  --ipam-driver=dhcp \
  --subnet=172.16.86.0/24 \
  -o parent=eth0 \
  --ipam-opt dhcp_interface=eth1 mcv0

docker run --net=mcv0 -itd alpine /bin/sh
docker run --net=mcv0 -itd alpine /bin/sh

VLAN Tagged Networks

  • Just as sub-interfaces tagged with VLAN IDs get dynamically created with the macvlan driver, this will also work in conjunction with the DHCP IPAM driver. Since Spanning-Tree will block links during STP convergence for up to 50 seconds, the --network and --gateway need to be specified by the user when creating the network using the standard docker network create command.

  • Once the new sub-interface is forwarding, containers started on that network will receive their IPv4 addresses from the DHCP server.

  • As with the macvlan driver, an existing interface can be passed and used, rather then the driver creating the sub-interface. If libnetwork drivers create a sub-interface, it will also delete them on docker network delete and rebuild them upon server reboot or Docker engine reboot from persistent storage offered by libnetwork. If the driver did not create the link but is passed an existing link, it will never delete that link, deletes will only occur if the driver created the link.

The following example will create a sub-interface tagged with VLAN 10 and create a network pool of usable addresses on the 172.16.86.0/24 network. The corresponding address pool on the DHCP server that is providing address on VLAN 10 will need to offer addresses on the 172.16.86.0/24 network.

docker network create -d macvlan \
  --ipam-driver=dhcp \
  --subnet=172.16.86.0/24 \
  --gateway=172.16.86.2 \
  -o parent=eth0.10 \
  --ipam-opt dhcp_interface=eth1.10 mcv0

docker network rm mcv0

If a network create is attempted with a sub-inerface, it will be rejected like the following example. It is merely because VLAN tagged links going to a Linux server will almost always block while Spanning-Tree converges since RSTP (Rapid Spanning-Tree) is primarily reserved for network switches with more advanced Layer2 implementations then Linux.

$ docker network create -d macvlan --ipam-driver=dhcp -o parent=eth0.10 --ipam-opt dhcp_interface=eth0.10 mcv0
Error response from daemon: Spanning-Tree convergence can block forwarding and thus DHCP for up to 50 seconds. If creating VLAN subinterfaces, --gateway and --subnet are required in 'docker network create'.

Same as if a sub-interface is not used, if a gateway is not passed, the driver will infer the gateway by using the first usable address from the --subnet passed in the network create. The following will get a gateway address of 172.16.86.1

docker network create -d macvlan \
  --ipam-driver=dhcp \
  --subnet=172.16.86.0/24 \
  -o parent=eth0.10 \
  --ipam-opt dhcp_interface=eth1.10 mcv0

docker rm -f `docker ps -qa`
docker network rm mcv0

Driver TODOs

  • What to do with DNS and Domain options from DHCP server?
  • Verify or add functionality with Bridge driver.
  • Opt code 12 (hostname) needs to be included in the DHCP REQUEST for added management value to see meaningful hostname/MAC/IP address mappings in the users DHCP server.
@enterit
Copy link

enterit commented Jul 8, 2016

Brent, your DHCP support implementation is awesome, thank you!
Question about opt code 12 (hostname) - how difficult would it be to add this?

I've checked the code of d2g/dhcp4client, it does not seem to allow for adding new options easily. I can see a couple of hooks for existing settings (HardwareAddr, IgnoreServers etc), but these only allow for modifying an existing set of options defined by Client structure. I assume what we need instead is the ability to extend (override) RequestPacket and RenewalRequestPacket functions in Client.

Is this even possible? If not, does it make sense to create a fork of that library with OptionHostName from d2g/dhcp4/constants.go supported?

Please forgive me if these questions are lame, I have not written any line of Go code (yet).

@segator
Copy link

segator commented Aug 29, 2016

are you going to pull request?
How Can I Compile libnetwork and try to my docker local env.

@amit213
Copy link

amit213 commented Jan 5, 2017

Thanks for this gist. Super useful information. Could you please also share a link to getting "dockerd" (docker service) with DHCP/IPAM support static compiled into them? I was able to use your docker binary, however, the docker daemon doesn't have the IPAM DHCP support.

@mterron
Copy link

mterron commented Feb 23, 2017

Isn't this https://github.com/d2g/dhcp4/blob/master/packet.go#L92 exactly what you are after? AddOption(12,"$HOSTNAME")

@enterit
Copy link

enterit commented Mar 2, 2017

@mterron, I understand that the option is supported by dhcp4, but I did not find an easy way to get it passed through the dhcp4client.
I meant forking dhcp4client and adding support for hostname.

@killcity
Copy link

Just wondering if this project has gained any traction? Also, would it be fairly simple to compile into 17.06? I've been using macvlan and swarm. I've had to chunk out the network ranges across hosts with no IPAM. This would be really handy...

@RonaldSwaine
Copy link

DHCP would be lovely, but sometimes need a good workaround. Here's what worked for us:

Office gateway setup (RV042) with DHCP on its primary (office-wide) subnet. ie: 10.0.0.0/24
A secondary subnet (ie: 10.0.1.0/24) is also registered with the gateway (and has no DHCP).
Macvlan docker network can then be created for the full range of the gateway's secondary subnet.

Notes on this setup for others:

  • remember to exclude the docker host machine's IP unless it resides within the main subnet
  • some gateways (or if using multiple physical ones) will require that you create specific routes for inter-subnet requests between the gateways (the RV042 automatically allowed traffic between the 2 subnets)

@DKalenscher
Copy link

DKalenscher commented Apr 24, 2018

Forgive what might be a very obvious question. I am new to docker but I need my containers to be automatically assigned IP addresses via DHCP so I would like to test this solution. Question is, how on earth do I actually use this? I downloaded the driver and it looks like I can run it directly and replace the regular /usr/bin/docker with it, is that all there is to it?

When I do this I get the following error trying to create the network:
Error response from daemon: plugin "dhcp" not found

Seems as though I am missing some dependencies.

@fletom
Copy link

fletom commented Jul 10, 2018

I can't wait for this to be released. in my opinion it will be the biggest new feature for Docker in years. thanks so much for your excellent work.

@arnolix
Copy link

arnolix commented Jul 23, 2018

hi, [root@dcos-slave01 ~]# docker network create -d macvlan --ipam-driver=dhcp --subnet=10.0.200.0/24 --gateway=10.0.200.1 -o parent=enp2s0.200 --ipam-opt dhcp_interface=enp2s0.200 net200
Error response from daemon: legacy plugin: plugin not found
How to resolv it ?

[root@dcos-slave01 ~]# docker info
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 3
Server Version: 17.05.0-ce
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.17.6-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.857GiB
Name: dcos-slave01
ID: WXIK:KYMY:XJTO:QA2F:D24A:UILA:VAFF:WPN3:4MXY:GYNZ:5LKX:PTZT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

@seawang54
Copy link

Excellent work! Does this driver support DHCPv6?

@AnonDEmaYan
Copy link

Forgive what might be a very obvious question. I am new to docker but I need my containers to be automatically assigned IP addresses via DHCP so I would like to test this solution. Question is, how on earth do I actually use this? I downloaded the driver and it looks like I can run it directly and replace the regular /usr/bin/docker with it, is that all there is to it?

When I do this I get the following error trying to create the network:
Error response from daemon: plugin "dhcp" not found

Seems as though I am missing some dependencies.

I'm new to Docker as well. @nerdalert Can you help us out? I'm not sure how to replace the docker driver and even I got the same error response.

@codefaux
Copy link

codefaux commented Jul 6, 2019

FYI - finding this a bit late, but for any future/current people trying to do this sort of thing, the "legacy plugin not found" message can indicate that the driver wasn't found internally, so it tried to fallback to legacy plugin checking and couldn't find it there either.

Launch your Docker daemon with experimental support enabled. Google it, there are different ways based on your system and I won't pretend to know them all - mine was adding --experimental=true to the launch command line.

@zh99998
Copy link

zh99998 commented Jul 12, 2019

FYI - finding this a bit late, but for any future/current people trying to do this sort of thing, the "legacy plugin not found" message can indicate that the driver wasn't found internally, so it tried to fallback to legacy plugin checking and couldn't find it there either.

Launch your Docker daemon with experimental support enabled. Google it, there are different ways based on your system and I won't pretend to know them all - mine was adding --experimental=true to the launch command line.

I still can't find how to make it work.
I have already installed official docker-ce 18.09.7.
then unzip this downloaded docker engine with driver. replace /usr/bin/docker. I confirm the docker command is just downloaded one.

➜  test1 docker -v
Docker version 1.11.0-dev, build 6d43dc9-unsupported, experimental

I added experimental flag to /etc/docker/daemon.json, and systemctl restart docker

➜  test1 cat /etc/docker/daemon.json
{
  "experimental": true
}

confirm experimental enabled

➜  test1 docker info
Containers: 3
 Running: 3
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 18.09.7
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Execution Driver: <not supported>
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Kernel Version: 4.19.0-5-amd64
Operating System: Debian GNU/Linux 10 (buster)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.948 GiB
Name: lancache
ID: VUXJ:QZPT:RNIL:7JUB:TOXP:5X76:5XHH:D7TY:RYJC:6T72:QXZF:X7GA
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Labels:
Experimental: true

then
docker cli is 1.11.0-dev with experimental dhcp driver . dockerd services is official 18.09.7. and plugin "dhcp" not found

➜  test1 docker network create -d macvlan \
           --ipam-driver=dhcp \
           -o parent=ens192 \
           --ipam-opt dhcp_interface=ens192 mcv0
Error response from daemon: plugin "dhcp" not found

how should I do, is there another dockerd program that should replace?

@edgararroyo
Copy link

This looks promising! I need this. How can I use this? I am getting the same result:

Error response from daemon: plugin "dhcp" not found

@JBFUK
Copy link

JBFUK commented Jan 11, 2020

Sorry if this is a silly question but how does this differ from using the 'docker-net-dhcp' plugin? I'm trying to work out the best way to move from using VMs on my network (and a big power hungry server) to using services running in docker containers instead.

@abhikeny
Copy link

The DHCP driver is really interesting and seems like a necessary option, since people are looking out for this. Can someone please post an update on whether this will be added to docker in the near future? If not, what are the alternatives?

I have tried the following (by making the uploaded driver-compiled-into-docker-engine binary an executable), but the command still fails:

$ ./docker-1.11.0-dev network create -d macvlan --ipam-driver=dhcp -o parent=eth1 --ipam-opt dhcp_interface=eth1 mcv1
Error response from daemon: plugin "dhcp" not found

Any help or update would be highly appreciated.

Thanks.

@sohojmanush
Copy link

How do I install "docker binary with libnetwork test dhcp client spam driver"?

@zhujintao
Copy link

How do I install "docker binary with libnetwork test dhcp client spam driver"??

@sunnyghat
Copy link

Has anyone been able to figure out how to install the plugin ?

@kcossabo
Copy link

kcossabo commented Jan 9, 2024

ugh, this looks so good but same as others

Error response from daemon: plugin "dhcp" not found

@Kitof
Copy link

Kitof commented Mar 25, 2024

I can't believe this feature has been waiting here for 8 years! Yet it seems essential for anyone who has to manipulate dozens of static IP addresses through macvlan interfaces. I really hope it will be available natively in the near future...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment