Traefik 2.5 with Consul Connect on Nomad

traefik.nomad

# Simple example to deploy traefik with consul connect enabled.
# For simplicity the job includes traefik as well as the backend service.
# Please note that traefik currently only supports connect for HTTP.
job "traefik-consul-connect-demo" {
  datacenters = ["dc1"]

  group "edge" {

    network {
      mode = "bridge"
      port "http" {
        to     = 8080
      }
    }

    service {
      name = "traefik-ingress"
      port = "http"

      connect {
        native = true
      }
    }

    task "traefik" {
      driver = "docker"
      config {
        image = "traefik:v2.5.2"
        args = [
          # Enables connect support, otherwise only http connections would be tried
          "--providers.consulcatalog.connectaware=true",
          # Make the communication secure by default
          "--providers.consulcatalog.connectbydefault=true",
          "--providers.consulcatalog.exposedbydefault=false",
          "--entrypoints.http=true",
          "--entrypoints.http.address=:8080",
          # The service name below should match the nomad/consul service above
          # and is used for intentions in consul
          "--providers.consulcatalog.servicename=traefik-ingress",
          "--providers.consulcatalog.prefix=traefik",

          # Automatically configured by Nomad through CONSUL_* environment variables
          # as long as client consul.share_ssl is enabled
          # "--providers.consulcatalog.endpoint.address=<socket|address>"
          # "--providers.consulcatalog.endpoint.tls.ca=<path>"
          # "--providers.consulcatalog.endpoint.tls.cert=<path>"
          # "--providers.consulcatalog.endpoint.tls.key=<path>"
          # "--providers.consulcatalog.endpoint.token=<token>"
        ]
      }

      env {
        # Enable this if nomad is older than 1.1.3
        # CONSUL_TLS_SERVER_NAME = "localhost"
      }
    }
  }

  group "backend" {
    network {
      mode = "bridge"
    }

    service {
      name = "whoami"
      port = 80
      tags = [
        "traefik.enable=true",
        "traefik.http.routers.whoami.rule=Host(`whoami.example.com`)"
      ]

      connect {
        sidecar_service {}
      }
    }

    # Note: For increased security the service should only listen on localhost
    # Otherwise it could be reachable from the outside world without going through connect
    task "whoami" {
      driver = "docker"
      config {
        image = "containous/whoami"
      }
    }
  }
}

Enables connect support, otherwise only http connections would be tried
"--providers.consulcatalog.connectaware=true",

I suppose this is what needs to be done differently?

Enables connect support, otherwise only http connections would be tried
"--providers.consulcatalog.connectaware=true",

I suppose this is what needs to be done differently?

Differently as opposed to what? Setting this flag allows traefik to talk to the connect sidecars and secure the connection. By itself it doesn't change behavior yet. You still need to use the global connectbydefault config or traefik.consulcatalog.connect=true per service.

As for hashicorp/consul#9643 (comment) I maybe should have been clearer. Adding the tags is enough if you are securing your connections ot the backend servers. If you simply wish to use traefik like before and do not encrypt the communication you still have to have the dummy tag, otherwise the sidecar will be considered a valid service. Technically I think the later might be (relatively easy) fixable, as traefik could just filter services that have a kind set to connect-proxy -- but I haven't thought about the implications of that.

Oh, just a side note, I felt like I recognized your username from somewhere, and while I'm reading the Traefik blog post, I saw your name come up, and I remembered that you did a lot of the heavy lifting on this functionality. Thank you so much for your work! 😃

Also, I think I figured it out. I was adding the "traefik.consulcatalog.connect=true" tag on a normal service that had a sidecar hooked up with connect

service {
  name = "shynet-django"
  port = "shynet"

  tags = [
    "traefik.enable=true",
    "traefik.http.routers.shynet.rule=Host(`shynet.discretemath.ca`)",
    "traefik.http.routers.shynet.entrypoints=https",
    "traefik.http.routers.shynet.tls.certresolver=letsencrypt",
    "traefik.consulcatalog.connect=true" # <-- The problem
  ]

  connect {
    sidecar_service {
      proxy {
        upstreams {
          destination_name = "shynet-postgres"
          local_bind_port  = 5432
        }
      }
      tags = [
        "dummy"
      ]
    }
  }
}

Thank you so much for your work!

Glad you like it!

<-- The problem

Oh yes, if you are using the sidecar only for the purpose of using encrypted connections to upstreams and not to encrypt inbound connections you should not set traefik.consulcatalog.connect=true.

Thank you guys. Without the traefik.consulcatalog.connect=true I still need to set the dummy tag to the sidecar service?

Hi !
Thanks for that example, works like a charm!

I've one interrogation about the providers.consulcatalog.servicename that should match the nomad service name in the service stanza.
It seems that this requirement makes it impossible to route multiple service via that traefik instance, am I right ?

The goal would be to use traefik deployed as a system job on each nomad client, and being able to serve as ingress-gateway for multiple services inside the service mesh.

That might be a limitation of the Traefik integration with consul-connect but I could not find proper explanation/documentation for this.

rgd,

It seems that this requirement makes it impossible to route multiple service via that traefik instance, am I right ?

What makes you say that?

It seems that this requirement makes it impossible to route multiple service via that traefik instance, am I right ?

What makes you say that?

I want to use multiple services using different entrypoints, which means multiple service stanza, I don't understand how to make that possible using the service-mesh integration.
ie I can configure multiple service stanza, but only one can be exposed to consul-catalog via service-name.

The different entrypoint is probably what is missing from my first comment

Here is a job spec for reference: basically adding one more entrypoint and duplicating the backend section:
I could not make both the http1-ingress-demo and http2-ingress-demo working at the same time.

job "ingress-connect-demo" {
  datacenters = ["meow"]

  group "edge" {

    network {
      mode = "bridge"

      port "http1" {
        static = 8901
        to     = 8901
      }

      port "http2" {
        static = 8902
        to     = 8902
      }
    }

    service {
      name = "http1-ingress-demo"
      port = "http1"

      connect {
        native = true
      }
    }

    service {
      name = "http2-ingress-demo"
      port = "http2"

      connect {
        native = true
      }
    }

    task "traefik" {
      driver = "docker"
      config {
        image = "traefik:2.6"
        args = [
          # Enables connect support, otherwise only http connections would be tried
          "--providers.consulcatalog.connectaware=true",
          # Make the communication secure by default
          "--providers.consulcatalog.connectbydefault=true",
          "--providers.consulcatalog.exposedbydefault=false",
          "--entrypoints.http1=true",
          "--entrypoints.http1.address=:8901",
          "--entrypoints.http2=true",
          "--entrypoints.http2.address=:8902",
          # The service name below should match the nomad/consul service above
          # and is used for intentions in consul
          "--providers.consulcatalog.servicename=http1-ingress-demo",
#          "--providers.consulcatalog.servicename=http2-ingress-demo",
          "--providers.consulcatalog.prefix=ingress-demo",

          # Automatically configured by Nomad through CONSUL_* environment variables
          # as long as client consul.share_ssl is enabled
          # "--providers.consulcatalog.endpoint.address=<socket|address>"
          # "--providers.consulcatalog.endpoint.tls.ca=<path>"
          # "--providers.consulcatalog.endpoint.tls.cert=<path>"
          # "--providers.consulcatalog.endpoint.tls.key=<path>"
          # "--providers.consulcatalog.endpoint.token=<token>"
        ]
      }

      env {
        # Enable this if nomad is older than 1.1.3
        # CONSUL_TLS_SERVER_NAME = "localhost"
      }
    }
  }

  group "backend1" {
    network {
      mode = "bridge"
    }

    service {
      name = "whoami1"
      port = 80
      tags = [
        "ingress-demo.enable=true",
        "ingress-demo.http.routers.whoami1.rule=Host(`whoami1.example.com`)",
        "ingress-demo.http.routers.whoami1.entrypoints=http1"
      ]

      connect {
        sidecar_service {}
      }
    }

    # Note: For increased security the service should only listen on localhost
    # Otherwise it could be reachable from the outside world without going through connect
    task "whoami" {
      driver = "docker"
      config {
        image = "containous/whoami:v1.5.0"
      }
    }
  }

  group "backend2" {
    network {
      mode = "bridge"
    }

    service {
      name = "whoami2"
      port = 80
      tags = [
        "ingress-demo.enable=true",
        "ingress-demo.http.routers.whoami2.rule=Host(`whoami2.example.com`)",
        "ingress-demo.http.routers.whoami2.entrypoints=http2"
      ]

      connect {
        sidecar_service {}
      }
    }

    # Note: For increased security the service should only listen on localhost
    # Otherwise it could be reachable from the outside world without going through connect
    task "whoami" {
      driver = "docker"
      config {
        image = "containous/whoami:v1.5.0"
      }
    }
  }
}

If you have a way to make it work, I take it :)

That being said, if you use the same entrypoint for both backends, and route differently based on smth (like HOST header) it works as expected.

job "ingress-connect-demo" {
  datacenters = ["meow"]

  group "edge" {

    network {
      mode = "bridge"

      port "http1" {
        static = 8901
        to     = 8901
      }

    }

    service {
      name = "http1-ingress-demo"
      port = "http1"

      connect {
        native = true
      }
    }

    task "traefik" {
      driver = "docker"
      config {
        image = "traefik:2.6"
        args = [
          # Enables connect support, otherwise only http connections would be tried
          "--providers.consulcatalog.connectaware=true",
          # Make the communication secure by default
          "--providers.consulcatalog.connectbydefault=true",
          "--providers.consulcatalog.exposedbydefault=false",
          "--entrypoints.http1=true",
          "--entrypoints.http1.address=:8901",
          # The service name below should match the nomad/consul service above
          # and is used for intentions in consul
          "--providers.consulcatalog.servicename=http1-ingress-demo",
          "--providers.consulcatalog.prefix=ingress-demo",

          # Automatically configured by Nomad through CONSUL_* environment variables
          # as long as client consul.share_ssl is enabled
          # "--providers.consulcatalog.endpoint.address=<socket|address>"
          # "--providers.consulcatalog.endpoint.tls.ca=<path>"
          # "--providers.consulcatalog.endpoint.tls.cert=<path>"
          # "--providers.consulcatalog.endpoint.tls.key=<path>"
          # "--providers.consulcatalog.endpoint.token=<token>"
        ]
      }

      env {
        # Enable this if nomad is older than 1.1.3
        # CONSUL_TLS_SERVER_NAME = "localhost"
      }
    }
  }

  group "backend1" {
    network {
      mode = "bridge"
    }

    service {
      name = "whoami1"
      port = 80
      tags = [
        "ingress-demo.enable=true",
        "ingress-demo.http.routers.whoami1.rule=Host(`whoami1.example.com`)",
        "ingress-demo.http.routers.whoami1.entrypoints=http1"
      ]

      connect {
        sidecar_service {}
      }
    }

    # Note: For increased security the service should only listen on localhost
    # Otherwise it could be reachable from the outside world without going through connect
    task "whoami" {
      driver = "docker"
      config {
        image = "containous/whoami:v1.5.0"
      }
    }
  }

  group "backend2" {
    network {
      mode = "bridge"
    }

    service {
      name = "whoami2"
      port = 80
      tags = [
        "ingress-demo.enable=true",
        "ingress-demo.http.routers.whoami2.rule=Host(`whoami2.example.com`)",
        "ingress-demo.http.routers.whoami2.entrypoints=http1"
      ]

      connect {
        sidecar_service {}
      }
    }

    # Note: For increased security the service should only listen on localhost
    # Otherwise it could be reachable from the outside world without going through connect
    task "whoami" {
      driver = "docker"
      config {
        image = "containous/whoami:v1.5.0"
      }
    }
  }
}

I hope that it makes thing clearer

You can use as many entrypoints and services as you want. But traefik is just one application, it will only use one certificate (which is the one from providers.consulcatalog.servicename and also the one you will have to use in intentions).

I use my traefik service like this (for two entrypoints):

    service {
      name = "traefik"
      port = "port1"
      tags = ["port1"]

      connect {
        native = true
      }
    }

    service {
      name = "traefik"
      port = "port2"
      tags = ["port2"]
    }

Note that one service can expose multiple ports (via tags) and one (!) connect {} stanza is enough.

You can use as many entrypoints and services as you want. But traefik is just one application, it will only use one certificate (which is the one from providers.consulcatalog.servicename and also the one you will have to use in intentions).
Note that one service can expose multiple ports (via tags) and one (!) connect {} stanza is enough.

Thanks a lot ! That's something I didn't know about the service exposure.

Here is a working example thanks to your input:
With proper intentions to allow ingress-demo to speak to whoami1 and whoami2, it works like a charm.

job "ingress-connect-demo" {
  datacenters = ["meow"]

  group "edge" {

    network {
      mode = "bridge"

      port "http1" {
        static = 8901
        to     = 8901
      }

      port "http2" {
        static = 8902
        to     = 8902
      }

      # Treafik dashboard
      port "dash" {
        static = 8909
        to     = 8909
      }
    }

    service {
      name = "ingress-demo"
      port = "http1"
      tags = ["http1"]

      connect {
        native = true
      }
    }

    service {
      name = "ingress-demo"
      port = "http2"
      tags = ["http2"]

#      connect {
#        native = true
#      }
    }

    task "traefik" {
      driver = "docker"
      config {
        image = "traefik:2.6"
        args = [
          # Enables connect support, otherwise only http connections would be tried
          "--providers.consulcatalog.connectaware=true",
          # Make the communication secure by default
          "--providers.consulcatalog.connectbydefault=true",
          "--providers.consulcatalog.exposedbydefault=false",
          "--entrypoints.http1=true",
          "--entrypoints.http1.address=:8901",
          "--entrypoints.http2=true",
          "--entrypoints.http2.address=:8902",
          "--entrypoints.traefik=true",
          "--entrypoints.traefik.address=:8909",
          "--api.dashboard=true",
          "--api.insecure=true",
          # The service name below should match the nomad/consul service above
          # and is used for intentions in consul
          "--providers.consulcatalog.servicename=ingress-demo",
          "--providers.consulcatalog.prefix=ingress-demo",

          # Automatically configured by Nomad through CONSUL_* environment variables
          # as long as client consul.share_ssl is enabled
          # "--providers.consulcatalog.endpoint.address=<socket|address>"
          # "--providers.consulcatalog.endpoint.tls.ca=<path>"
          # "--providers.consulcatalog.endpoint.tls.cert=<path>"
          # "--providers.consulcatalog.endpoint.tls.key=<path>"
          # "--providers.consulcatalog.endpoint.token=<token>"
        ]
      }

      env {
        # Enable this if nomad is older than 1.1.3
        # CONSUL_TLS_SERVER_NAME = "localhost"
      }
    }
  }

  group "backend1" {
    network {
      mode = "bridge"
    }

    service {
      name = "whoami1"
      port = 80
      tags = [
        "ingress-demo.enable=true",
        "ingress-demo.http.routers.whoami1.rule=Host(`whoami1.example.com`)",
        "ingress-demo.http.routers.whoami1.entrypoints=http1"
      ]

      connect {
        sidecar_service {}
      }
    }

    # Note: For increased security the service should only listen on localhost
    # Otherwise it could be reachable from the outside world without going through connect
    task "whoami" {
      driver = "docker"
      config {
        image = "containous/whoami:v1.5.0"
      }
    }
  }

  group "backend2" {
    network {
      mode = "bridge"
    }

    service {
      name = "whoami2"
      port = 80
      tags = [
        "ingress-demo.enable=true",
        "ingress-demo.http.routers.whoami2.rule=Host(`whoami2.example.com`)",
        "ingress-demo.http.routers.whoami2.entrypoints=http2"
      ]

      connect {
        sidecar_service {}
      }
    }

    # Note: For increased security the service should only listen on localhost
    # Otherwise it could be reachable from the outside world without going through connect
    task "whoami" {
      driver = "docker"
      config {
        image = "containous/whoami:v1.5.0"
      }
    }
  }
}

And curl is happy:

$ curl -H "Host: whoami2.example.com" http://whoami2.example.com:8902/health/ --resolve whoami2.example.com:8902:10.xxx.xxx.12
Hostname: 0aa882d6ab30
IP: 127.0.0.1
IP: 172.26.74.189
RemoteAddr: 127.0.0.1:59626
GET /health/ HTTP/1.1
Host: whoami2.example.com
User-Agent: curl/7.81.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: xxx.xxx.xxx.xxx
X-Forwarded-Host: whoami2.example.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: 27c5f7da9048
X-Real-Ip: xxx.xxx.xxx.xxx

$ curl -H "Host: whoami1.example.com" http://whoami1.example.com:8901/health/ --resolve whoami1.example.com:8901:10.xxx.xxx.12
Hostname: ab3616414ad0
IP: 127.0.0.1
IP: 172.26.74.188
RemoteAddr: 127.0.0.1:59644
GET /health/ HTTP/1.1
Host: whoami1.example.com
User-Agent: curl/7.81.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: xxx.xxx.xxx.xxx
X-Forwarded-Host: whoami1.example.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: 27c5f7da9048
X-Real-Ip: xxx.xxx.xxx.xxx

Thanks for your help !

The sidecar is necessary for traefik?