droplet discovery over digitalocean private networks

gistfile1.md

Digital Ocean recently released private networking support in their NYC2 Data center.

They also published a blog post on how to setup a new droplet with private networking. But one thing the post doesn't do is tell you how to scale your private network for many boxes. One approach is obviously to edit /etc/hosts (but this gets annoying when you add a new box). A better way is to create an internal DNS zone (via the digital ocean web interface) and have your droplets use it:

Steps

setup the internal DNS zone file

1. Login to digital ocean
2. Click "DNS" on the right hand menu
3. Click "Add Domain"
4. Name it "in.example.com" (obviously use whatever TLD you want).
5. Click "Create Domain"
6. For each droplet you want resolved by name, add an A record with the droplet hostname and private IP.

Using doctl & jq:

1. Get a list of each droplet (assuming their name contains the domain) and it's private ip:

   doctl d l | grep example.com | awk '{print $2 }' | xargs -L 1 -P 5 doctl -f json d f | jq -r '.name as $host_name | .networks.v4[] | select(.type == "private")  | "\($host_name) \(.ip_address)"'
   

   Explanation: this uses xargs with the -L flag to parse each incoming line and -P to send parallel requests, jq is used to store the hostname in a variable and filter the ipv4 networks down to private types and only return the raw (-r) ip_address

2. create a domain

   doctl dns c in.example.com <droplet-name>
   

3. Add records for each droplet with their private ip (from above listing)

   doctl dns add --name <droplet-name> --data <private-ip> in.example.com
   

configure droplets (ref: private dns tutorial)

On each Ubuntu/Debian Droplet you create the head file, which is prepended to resolv.conf on boot to use the IP addresses of digital ocean nameserves (ns1.digitalocean.com, ns2.digitalocean.com, ns3.digitalocean.com (note these may change)) as well as google nameservers

sudo vi /etc/resolvconf/resolv.conf.d/head

search in.example.com
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 173.245.58.51
nameserver 173.245.59.41
nameserver 198.41.222.173

More info on search option, you may overwrite the search option from resolv conf using LOCALDOMAIN=test.com ping host1 this will try to resolve host1.test.com with the nameservers provided.

Now run resolvconf to generate a new resolv.conf file:

sudo resolvconf -u

Done!

Now all of your droplets can ping one another by hostname over the private network. And when you add a new droplet, just add it to the internal DNS zone and it will be visible by other droplets.

If you own the domain name example.com and have configured the digital ocean nameservers for it, you do not need to adjust the resolv configuration

my droplet could no longer resolve normal hostnames... resolv configuration needs to be revised

EDIT: had to put google dns first

Thanks for this I was looking into it as we unfortunately have some legacy stuff in DO, a couple pieces of feedback:

• resolv.conf only supports using up to 3 nameservers even if you specify more. The last two will never get used
• the way DNS fails over is that it will keep trying the first one until it fails and then will move on to the next so with this setup you have two dns resolvers for google and then one for DO (per previous comment) that means that there is a lot of failover and in some cases you are doubling your attempt when failing over to the internal names vs public ones. Depending on which one is more important/frequent you might want the internal ones to be the majority or even just have two and accept that you are not really N+1 for DNS durability.

You might also consider just making your internal DNS server a forwarder for public domains so to avoid some of these problems.

I have just created a shell script that works with latest doctl version 1.62.0-release and does not use jq

DO_upd_domain.sh

Thank you @so0k for the idea.