Change default DNS nameserver used by Kubernetes pods

defaultdns.md

Change default DNS nameserver used by Kubernetes pods

This can be applied generically but usually applies to Linux nodes that have a local caching nameserver running, which means pointing to an IP in the loopback range (127.0.0.0/8). Ubuntu 18.04 Bionic Beaver does this by default.

Option 1: Change host configuration

sudo systemctl mask systemd-resolved
rm -f /etc/resolv.conf
sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

Option 2: Change default resolv.conf by adding kubelet parameter

The parameter will make sure that the kubelet will use a different file as /etc/resolv.conf.

From https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/:

--resolv-conf string
Resolver configuration file used as the basis for the container DNS resolution configuration. (default "/etc/resolv.conf")

You can create the cluster using the following snippet in the Edit as YAML under Cluster Options.

services:
  kubelet:
    extra_args:
      resolv-conf: /host/etc/mycustomresolv.conf

The referenced file must be present on the host filesystem (/etc is mounted in the kubelet under /host/etc):

echo "nameserver 8.8.8.8" > /etc/mycustomresolv.conf

Option 3: Configure kube-dns to use a different upstream using ConfigMap

Configure kube-dns to use an upstream nameserver instead of the one in /etc/resolv.conf:

Save in configmap.yml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
data:
  upstreamNameservers: |
    ["8.8.8.8"]

kubectl create -f configmap.yml

DNS troubleshooting

Retrieve nameserver kube-dns is using:

kubectl exec -ti -n kube-system $(kubectl get --no-headers=true pods -l k8s-app=kube-dns -o custom-columns=:metadata.name -n kube-system) -c kubedns -- cat /etc/resolv.conf

Host should have net.ipv4.ip_forward set to 1:

sysctl -w net.ipv4.ip_forward=1

I found this page very helpful, thank you! My specific issue though was here: kubernetes/kubernetes#64924 and gliderlabs/docker-alpine#255. The discussion seems to have narrowed external DNS resolution to alpine images (may be related to musl libc). The easy fix is to change pod options to

  dnsConfig:
    options:
      - name: ndots
        value: "1"

or use your described method to replace /etc/resolv.conf with no ndots (defaults to 1).

Thank a lot, still useful