Instantly share code, notes, and snippets.

Embed
What would you like to do?

Experimental Docker Libnetwork DHCP Driver

The DHCP driver is intended for users to be able to integrate Docker IP address management with their existing IPAM strategies that use DHCP for dynamic address assignment. DHCP enables users to allocate addresses in an organized fashion that will prevent overlapping IP address assignment by associating a unique MAC address from the container eth0 Ethernet interface to an IP address as determined by the DHCP pools defined in the DHCP configuration.

This driver only provides the DHCP client functionality. It does not include a DHCP server. The default driver offers single-host IPAM or for distributed multi-host orchestrated IPAM see the libnetwork overlay driver.

Getting Started

Example DHCP Driver Client Usages

By default, the DHCP client driver will automatically probe for a network that eth0 is attached to by using a DHCP DISCOVER broadcast. If there is a DHCP server on the broadcast domain or handled by a relay/helper agent the existing DHCP server will reply with details about the available network that exists. The driver will then create the Docker IPAM pool and network for the user using the discovered information. From there, a user needs to simply start containers and receive IP addresses from the DHCP server.

The DHCP driver creates the DHCP requests using the MAC address of the container. The DHCP server will keep track of available addresses in the pool to ensure there are no overlapping addresses for all of the different Docker hosts running on the same Ethernet segment. Users can leverage their DHCP server of choice and monitor address allocations of all containers from the DHCP server's interface. When containers are removed docker rm <container_id>, the DHCP client driver sends a DHCP release out the specified --ipam-opt dhcp_interface=<host interface> and returns the IP address back to the DHCP server's address pool.

Note: If running Docker nested in a hypervisor, the VM's NIC needs to be in promiscous mode just as it would if you were running a nested hypervisor. This means the host Ethernet device can look at traffic with destination MAC addresseses that are different from its own MAC address. Since the DHCP client driver will be sending DHCP requests on behalf of the MAC address of a container, the host needs to be able to process those packets with differing MAC addresses. Along with various DHCP troubleshooting tools for Linux that are generally available, tcpdump can be used to troubleshoot filtering on the relevant UDP ports sudo tcpdump -i eth0 -n udp -v portrange 67-68. Tcpdump on a production network can be extremely chatty so use with care if not experienced with troubleshooting network issues.

  • The following example discovers both the subnet and gateway of the host. It does not discover any other DHCP options such as DNS or any other fields that are plumbed into the container. Any other options discovered are used by the driver in order to provide an easy network environment to manage.
# Create a network using eth0 as the parent and interface to discover what subnet to use for the network eth0 is attached
docker network create -d macvlan \
  --ipam-driver=dhcp \
  -o parent=eth0 \
  --ipam-opt dhcp_interface=eth0 mcv0

# When containers are created, DHCP requests will assign an IP addresses to each container
docker run --net=mcv0 -itd alpine /bin/sh
docker run --net=mcv0 -itd alpine /bin/sh

# When the containers are destroyed, the DHCP client driver sends a DHCP release and returns the IP address back to the DHCP servers address pool
docker rm -f `docker ps -qa`
docker network rm mcv0

Users can also explicitly specify the network's address pool --subnet= and --gateway= rather then the DHCP client driver discovering the subnet for them. Either way, when the containers are started, DHCP requests are sent by the driver when containers are started. The specified --subnet must match the subnet the DHCP server is handing out or else the driver will prevent a container creation (docker run) since the network and containers on the network need to coincide.

docker network create -d macvlan \
  --ipam-driver=dhcp \
  --subnet=172.16.86.0/24 \
  --gateway=172.16.86.2 \
  -o parent=eth0 \
  --ipam-opt dhcp_interface=eth0 mcv0

docker run --net=mcv0 -itd alpine /bin/sh
docker run --net=mcv0 -itd alpine /bin/sh

docker rm -f `docker ps -qa`
docker network rm mcv0
  • If a gateway is not specified, the driver will infer the gateway by using the first usable address on the user specified --subnet.
docker network create -d macvlan \
  --ipam-driver=dhcp \
  --subnet=172.16.86.0/24 \
  -o parent=eth0 \
  --ipam-opt dhcp_interface=eth1 mcv0

docker run --net=mcv0 -itd alpine /bin/sh
docker run --net=mcv0 -itd alpine /bin/sh

VLAN Tagged Networks

  • Just as sub-interfaces tagged with VLAN IDs get dynamically created with the macvlan driver, this will also work in conjunction with the DHCP IPAM driver. Since Spanning-Tree will block links during STP convergence for up to 50 seconds, the --network and --gateway need to be specified by the user when creating the network using the standard docker network create command.

  • Once the new sub-interface is forwarding, containers started on that network will receive their IPv4 addresses from the DHCP server.

  • As with the macvlan driver, an existing interface can be passed and used, rather then the driver creating the sub-interface. If libnetwork drivers create a sub-interface, it will also delete them on docker network delete and rebuild them upon server reboot or Docker engine reboot from persistent storage offered by libnetwork. If the driver did not create the link but is passed an existing link, it will never delete that link, deletes will only occur if the driver created the link.

The following example will create a sub-interface tagged with VLAN 10 and create a network pool of usable addresses on the 172.16.86.0/24 network. The corresponding address pool on the DHCP server that is providing address on VLAN 10 will need to offer addresses on the 172.16.86.0/24 network.

docker network create -d macvlan \
  --ipam-driver=dhcp \
  --subnet=172.16.86.0/24 \
  --gateway=172.16.86.2 \
  -o parent=eth0.10 \
  --ipam-opt dhcp_interface=eth1.10 mcv0

docker network rm mcv0

If a network create is attempted with a sub-inerface, it will be rejected like the following example. It is merely because VLAN tagged links going to a Linux server will almost always block while Spanning-Tree converges since RSTP (Rapid Spanning-Tree) is primarily reserved for network switches with more advanced Layer2 implementations then Linux.

$ docker network create -d macvlan --ipam-driver=dhcp -o parent=eth0.10 --ipam-opt dhcp_interface=eth0.10 mcv0
Error response from daemon: Spanning-Tree convergence can block forwarding and thus DHCP for up to 50 seconds. If creating VLAN subinterfaces, --gateway and --subnet are required in 'docker network create'.

Same as if a sub-interface is not used, if a gateway is not passed, the driver will infer the gateway by using the first usable address from the --subnet passed in the network create. The following will get a gateway address of 172.16.86.1

docker network create -d macvlan \
  --ipam-driver=dhcp \
  --subnet=172.16.86.0/24 \
  -o parent=eth0.10 \
  --ipam-opt dhcp_interface=eth1.10 mcv0

docker rm -f `docker ps -qa`
docker network rm mcv0

Driver TODOs

  • What to do with DNS and Domain options from DHCP server?
  • Verify or add functionality with Bridge driver.
  • Opt code 12 (hostname) needs to be included in the DHCP REQUEST for added management value to see meaningful hostname/MAC/IP address mappings in the users DHCP server.
@enterit

This comment has been minimized.

Show comment
Hide comment
@enterit

enterit Jul 8, 2016

Brent, your DHCP support implementation is awesome, thank you!
Question about opt code 12 (hostname) - how difficult would it be to add this?

I've checked the code of d2g/dhcp4client, it does not seem to allow for adding new options easily. I can see a couple of hooks for existing settings (HardwareAddr, IgnoreServers etc), but these only allow for modifying an existing set of options defined by Client structure. I assume what we need instead is the ability to extend (override) RequestPacket and RenewalRequestPacket functions in Client.

Is this even possible? If not, does it make sense to create a fork of that library with OptionHostName from d2g/dhcp4/constants.go supported?

Please forgive me if these questions are lame, I have not written any line of Go code (yet).

enterit commented Jul 8, 2016

Brent, your DHCP support implementation is awesome, thank you!
Question about opt code 12 (hostname) - how difficult would it be to add this?

I've checked the code of d2g/dhcp4client, it does not seem to allow for adding new options easily. I can see a couple of hooks for existing settings (HardwareAddr, IgnoreServers etc), but these only allow for modifying an existing set of options defined by Client structure. I assume what we need instead is the ability to extend (override) RequestPacket and RenewalRequestPacket functions in Client.

Is this even possible? If not, does it make sense to create a fork of that library with OptionHostName from d2g/dhcp4/constants.go supported?

Please forgive me if these questions are lame, I have not written any line of Go code (yet).

@segator

This comment has been minimized.

Show comment
Hide comment
@segator

segator Aug 29, 2016

are you going to pull request?
How Can I Compile libnetwork and try to my docker local env.

segator commented Aug 29, 2016

are you going to pull request?
How Can I Compile libnetwork and try to my docker local env.

@amit213

This comment has been minimized.

Show comment
Hide comment
@amit213

amit213 Jan 5, 2017

Thanks for this gist. Super useful information. Could you please also share a link to getting "dockerd" (docker service) with DHCP/IPAM support static compiled into them? I was able to use your docker binary, however, the docker daemon doesn't have the IPAM DHCP support.

amit213 commented Jan 5, 2017

Thanks for this gist. Super useful information. Could you please also share a link to getting "dockerd" (docker service) with DHCP/IPAM support static compiled into them? I was able to use your docker binary, however, the docker daemon doesn't have the IPAM DHCP support.

@mterron

This comment has been minimized.

Show comment
Hide comment
@mterron

mterron Feb 23, 2017

Isn't this https://github.com/d2g/dhcp4/blob/master/packet.go#L92 exactly what you are after? AddOption(12,"$HOSTNAME")

mterron commented Feb 23, 2017

Isn't this https://github.com/d2g/dhcp4/blob/master/packet.go#L92 exactly what you are after? AddOption(12,"$HOSTNAME")

@enterit

This comment has been minimized.

Show comment
Hide comment
@enterit

enterit Mar 2, 2017

@mterron, I understand that the option is supported by dhcp4, but I did not find an easy way to get it passed through the dhcp4client.
I meant forking dhcp4client and adding support for hostname.

enterit commented Mar 2, 2017

@mterron, I understand that the option is supported by dhcp4, but I did not find an easy way to get it passed through the dhcp4client.
I meant forking dhcp4client and adding support for hostname.

@killcity

This comment has been minimized.

Show comment
Hide comment
@killcity

killcity Aug 22, 2017

Just wondering if this project has gained any traction? Also, would it be fairly simple to compile into 17.06? I've been using macvlan and swarm. I've had to chunk out the network ranges across hosts with no IPAM. This would be really handy...

killcity commented Aug 22, 2017

Just wondering if this project has gained any traction? Also, would it be fairly simple to compile into 17.06? I've been using macvlan and swarm. I've had to chunk out the network ranges across hosts with no IPAM. This would be really handy...

@RonaldSwaine

This comment has been minimized.

Show comment
Hide comment
@RonaldSwaine

RonaldSwaine Nov 2, 2017

DHCP would be lovely, but sometimes need a good workaround. Here's what worked for us:

Office gateway setup (RV042) with DHCP on its primary (office-wide) subnet. ie: 10.0.0.0/24
A secondary subnet (ie: 10.0.1.0/24) is also registered with the gateway (and has no DHCP).
Macvlan docker network can then be created for the full range of the gateway's secondary subnet.

Notes on this setup for others:

  • remember to exclude the docker host machine's IP unless it resides within the main subnet
  • some gateways (or if using multiple physical ones) will require that you create specific routes for inter-subnet requests between the gateways (the RV042 automatically allowed traffic between the 2 subnets)

RonaldSwaine commented Nov 2, 2017

DHCP would be lovely, but sometimes need a good workaround. Here's what worked for us:

Office gateway setup (RV042) with DHCP on its primary (office-wide) subnet. ie: 10.0.0.0/24
A secondary subnet (ie: 10.0.1.0/24) is also registered with the gateway (and has no DHCP).
Macvlan docker network can then be created for the full range of the gateway's secondary subnet.

Notes on this setup for others:

  • remember to exclude the docker host machine's IP unless it resides within the main subnet
  • some gateways (or if using multiple physical ones) will require that you create specific routes for inter-subnet requests between the gateways (the RV042 automatically allowed traffic between the 2 subnets)
@DKalenscher

This comment has been minimized.

Show comment
Hide comment
@DKalenscher

DKalenscher Apr 24, 2018

Forgive what might be a very obvious question. I am new to docker but I need my containers to be automatically assigned IP addresses via DHCP so I would like to test this solution. Question is, how on earth do I actually use this? I downloaded the driver and it looks like I can run it directly and replace the regular /usr/bin/docker with it, is that all there is to it?

When I do this I get the following error trying to create the network:
Error response from daemon: plugin "dhcp" not found

Seems as though I am missing some dependencies.

DKalenscher commented Apr 24, 2018

Forgive what might be a very obvious question. I am new to docker but I need my containers to be automatically assigned IP addresses via DHCP so I would like to test this solution. Question is, how on earth do I actually use this? I downloaded the driver and it looks like I can run it directly and replace the regular /usr/bin/docker with it, is that all there is to it?

When I do this I get the following error trying to create the network:
Error response from daemon: plugin "dhcp" not found

Seems as though I am missing some dependencies.

@fletom

This comment has been minimized.

Show comment
Hide comment
@fletom

fletom Jul 10, 2018

I can't wait for this to be released. in my opinion it will be the biggest new feature for Docker in years. thanks so much for your excellent work.

fletom commented Jul 10, 2018

I can't wait for this to be released. in my opinion it will be the biggest new feature for Docker in years. thanks so much for your excellent work.

@arnolix

This comment has been minimized.

Show comment
Hide comment
@arnolix

arnolix Jul 23, 2018

hi, [root@dcos-slave01 ~]# docker network create -d macvlan --ipam-driver=dhcp --subnet=10.0.200.0/24 --gateway=10.0.200.1 -o parent=enp2s0.200 --ipam-opt dhcp_interface=enp2s0.200 net200
Error response from daemon: legacy plugin: plugin not found
How to resolv it ?

[root@dcos-slave01 ~]# docker info
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 3
Server Version: 17.05.0-ce
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.17.6-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.857GiB
Name: dcos-slave01
ID: WXIK:KYMY:XJTO:QA2F:D24A:UILA:VAFF:WPN3:4MXY:GYNZ:5LKX:PTZT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

arnolix commented Jul 23, 2018

hi, [root@dcos-slave01 ~]# docker network create -d macvlan --ipam-driver=dhcp --subnet=10.0.200.0/24 --gateway=10.0.200.1 -o parent=enp2s0.200 --ipam-opt dhcp_interface=enp2s0.200 net200
Error response from daemon: legacy plugin: plugin not found
How to resolv it ?

[root@dcos-slave01 ~]# docker info
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 3
Server Version: 17.05.0-ce
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.17.6-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.857GiB
Name: dcos-slave01
ID: WXIK:KYMY:XJTO:QA2F:D24A:UILA:VAFF:WPN3:4MXY:GYNZ:5LKX:PTZT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

@seawang54

This comment has been minimized.

Show comment
Hide comment
@seawang54

seawang54 Jul 27, 2018

Excellent work! Does this driver support DHCPv6?

seawang54 commented Jul 27, 2018

Excellent work! Does this driver support DHCPv6?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment