Skip to content

Instantly share code, notes, and snippets.

@apollo13

apollo13/traefik.nomad

Last active April 15, 2024 14:46
Show Gist options
  • Star 21 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save apollo13/857ae4c5e18de619815c2628212449e1 to your computer and use it in GitHub Desktop.
Save apollo13/857ae4c5e18de619815c2628212449e1 to your computer and use it in GitHub Desktop.
Download ZIP
Traefik 2.5 with Consul Connect on Nomad
Raw
traefik.nomad
# Simple example to deploy traefik with consul connect enabled.
# For simplicity the job includes traefik as well as the backend service.
# Please note that traefik currently only supports connect for HTTP.
job "traefik-consul-connect-demo" {
datacenters = ["dc1"]
group "edge" {
network {
mode = "bridge"
port "http" {
to = 8080
}
}
service {
name = "traefik-ingress"
port = "http"
connect {
native = true
}
}
task "traefik" {
driver = "docker"
config {
image = "traefik:v2.5.2"
args = [
# Enables connect support, otherwise only http connections would be tried
"--providers.consulcatalog.connectaware=true",
# Make the communication secure by default
"--providers.consulcatalog.connectbydefault=true",
"--providers.consulcatalog.exposedbydefault=false",
"--entrypoints.http=true",
"--entrypoints.http.address=:8080",
# The service name below should match the nomad/consul service above
# and is used for intentions in consul
"--providers.consulcatalog.servicename=traefik-ingress",
"--providers.consulcatalog.prefix=traefik",
# Automatically configured by Nomad through CONSUL_* environment variables
# as long as client consul.share_ssl is enabled
# "--providers.consulcatalog.endpoint.address=<socket|address>"
# "--providers.consulcatalog.endpoint.tls.ca=<path>"
# "--providers.consulcatalog.endpoint.tls.cert=<path>"
# "--providers.consulcatalog.endpoint.tls.key=<path>"
# "--providers.consulcatalog.endpoint.token=<token>"
]
}
env {
# Enable this if nomad is older than 1.1.3
# CONSUL_TLS_SERVER_NAME = "localhost"
}
}
}
group "backend" {
network {
mode = "bridge"
}
service {
name = "whoami"
port = 80
tags = [
"traefik.enable=true",
"traefik.http.routers.whoami.rule=Host(`whoami.example.com`)"
]
connect {
sidecar_service {}
}
}
# Note: For increased security the service should only listen on localhost
# Otherwise it could be reachable from the outside world without going through connect
task "whoami" {
driver = "docker"
config {
image = "containous/whoami"
}
}
}
}
@apollo13
Copy link
Author

apollo13 commented Mar 8, 2022

It seems that this requirement makes it impossible to route multiple service via that traefik instance, am I right ?

What makes you say that?

@lodpp
Copy link

lodpp commented Mar 8, 2022

It seems that this requirement makes it impossible to route multiple service via that traefik instance, am I right ?

What makes you say that?

I want to use multiple services using different entrypoints, which means multiple service stanza, I don't understand how to make that possible using the service-mesh integration.
ie I can configure multiple service stanza, but only one can be exposed to consul-catalog via service-name.

The different entrypoint is probably what is missing from my first comment

Here is a job spec for reference: basically adding one more entrypoint and duplicating the backend section:
I could not make both the http1-ingress-demo and http2-ingress-demo working at the same time.

job "ingress-connect-demo" {
  datacenters = ["meow"]

  group "edge" {

    network {
      mode = "bridge"

      port "http1" {
        static = 8901
        to     = 8901
      }

      port "http2" {
        static = 8902
        to     = 8902
      }
    }

    service {
      name = "http1-ingress-demo"
      port = "http1"

      connect {
        native = true
      }
    }

    service {
      name = "http2-ingress-demo"
      port = "http2"

      connect {
        native = true
      }
    }

    task "traefik" {
      driver = "docker"
      config {
        image = "traefik:2.6"
        args = [
          # Enables connect support, otherwise only http connections would be tried
          "--providers.consulcatalog.connectaware=true",
          # Make the communication secure by default
          "--providers.consulcatalog.connectbydefault=true",
          "--providers.consulcatalog.exposedbydefault=false",
          "--entrypoints.http1=true",
          "--entrypoints.http1.address=:8901",
          "--entrypoints.http2=true",
          "--entrypoints.http2.address=:8902",
          # The service name below should match the nomad/consul service above
          # and is used for intentions in consul
          "--providers.consulcatalog.servicename=http1-ingress-demo",
#          "--providers.consulcatalog.servicename=http2-ingress-demo",
          "--providers.consulcatalog.prefix=ingress-demo",

          # Automatically configured by Nomad through CONSUL_* environment variables
          # as long as client consul.share_ssl is enabled
          # "--providers.consulcatalog.endpoint.address=<socket|address>"
          # "--providers.consulcatalog.endpoint.tls.ca=<path>"
          # "--providers.consulcatalog.endpoint.tls.cert=<path>"
          # "--providers.consulcatalog.endpoint.tls.key=<path>"
          # "--providers.consulcatalog.endpoint.token=<token>"
        ]
      }

      env {
        # Enable this if nomad is older than 1.1.3
        # CONSUL_TLS_SERVER_NAME = "localhost"
      }
    }
  }

  group "backend1" {
    network {
      mode = "bridge"
    }

    service {
      name = "whoami1"
      port = 80
      tags = [
        "ingress-demo.enable=true",
        "ingress-demo.http.routers.whoami1.rule=Host(`whoami1.example.com`)",
        "ingress-demo.http.routers.whoami1.entrypoints=http1"
      ]

      connect {
        sidecar_service {}
      }
    }

    # Note: For increased security the service should only listen on localhost
    # Otherwise it could be reachable from the outside world without going through connect
    task "whoami" {
      driver = "docker"
      config {
        image = "containous/whoami:v1.5.0"
      }
    }
  }

  group "backend2" {
    network {
      mode = "bridge"
    }

    service {
      name = "whoami2"
      port = 80
      tags = [
        "ingress-demo.enable=true",
        "ingress-demo.http.routers.whoami2.rule=Host(`whoami2.example.com`)",
        "ingress-demo.http.routers.whoami2.entrypoints=http2"
      ]

      connect {
        sidecar_service {}
      }
    }

    # Note: For increased security the service should only listen on localhost
    # Otherwise it could be reachable from the outside world without going through connect
    task "whoami" {
      driver = "docker"
      config {
        image = "containous/whoami:v1.5.0"
      }
    }
  }
}

If you have a way to make it work, I take it :)

That being said, if you use the same entrypoint for both backends, and route differently based on smth (like HOST header) it works as expected.

job "ingress-connect-demo" {
  datacenters = ["meow"]

  group "edge" {

    network {
      mode = "bridge"

      port "http1" {
        static = 8901
        to     = 8901
      }

    }

    service {
      name = "http1-ingress-demo"
      port = "http1"

      connect {
        native = true
      }
    }

    task "traefik" {
      driver = "docker"
      config {
        image = "traefik:2.6"
        args = [
          # Enables connect support, otherwise only http connections would be tried
          "--providers.consulcatalog.connectaware=true",
          # Make the communication secure by default
          "--providers.consulcatalog.connectbydefault=true",
          "--providers.consulcatalog.exposedbydefault=false",
          "--entrypoints.http1=true",
          "--entrypoints.http1.address=:8901",
          # The service name below should match the nomad/consul service above
          # and is used for intentions in consul
          "--providers.consulcatalog.servicename=http1-ingress-demo",
          "--providers.consulcatalog.prefix=ingress-demo",

          # Automatically configured by Nomad through CONSUL_* environment variables
          # as long as client consul.share_ssl is enabled
          # "--providers.consulcatalog.endpoint.address=<socket|address>"
          # "--providers.consulcatalog.endpoint.tls.ca=<path>"
          # "--providers.consulcatalog.endpoint.tls.cert=<path>"
          # "--providers.consulcatalog.endpoint.tls.key=<path>"
          # "--providers.consulcatalog.endpoint.token=<token>"
        ]
      }

      env {
        # Enable this if nomad is older than 1.1.3
        # CONSUL_TLS_SERVER_NAME = "localhost"
      }
    }
  }

  group "backend1" {
    network {
      mode = "bridge"
    }

    service {
      name = "whoami1"
      port = 80
      tags = [
        "ingress-demo.enable=true",
        "ingress-demo.http.routers.whoami1.rule=Host(`whoami1.example.com`)",
        "ingress-demo.http.routers.whoami1.entrypoints=http1"
      ]

      connect {
        sidecar_service {}
      }
    }

    # Note: For increased security the service should only listen on localhost
    # Otherwise it could be reachable from the outside world without going through connect
    task "whoami" {
      driver = "docker"
      config {
        image = "containous/whoami:v1.5.0"
      }
    }
  }

  group "backend2" {
    network {
      mode = "bridge"
    }

    service {
      name = "whoami2"
      port = 80
      tags = [
        "ingress-demo.enable=true",
        "ingress-demo.http.routers.whoami2.rule=Host(`whoami2.example.com`)",
        "ingress-demo.http.routers.whoami2.entrypoints=http1"
      ]

      connect {
        sidecar_service {}
      }
    }

    # Note: For increased security the service should only listen on localhost
    # Otherwise it could be reachable from the outside world without going through connect
    task "whoami" {
      driver = "docker"
      config {
        image = "containous/whoami:v1.5.0"
      }
    }
  }
}

I hope that it makes thing clearer

@apollo13
Copy link
Author

apollo13 commented Mar 8, 2022

You can use as many entrypoints and services as you want. But traefik is just one application, it will only use one certificate (which is the one from providers.consulcatalog.servicename and also the one you will have to use in intentions).

I use my traefik service like this (for two entrypoints):

    service {
      name = "traefik"
      port = "port1"
      tags = ["port1"]

      connect {
        native = true
      }
    }

    service {
      name = "traefik"
      port = "port2"
      tags = ["port2"]
    }

Note that one service can expose multiple ports (via tags) and one (!) connect {} stanza is enough.

@lodpp
Copy link

lodpp commented Mar 9, 2022

You can use as many entrypoints and services as you want. But traefik is just one application, it will only use one certificate (which is the one from providers.consulcatalog.servicename and also the one you will have to use in intentions).
Note that one service can expose multiple ports (via tags) and one (!) connect {} stanza is enough.

Thanks a lot ! That's something I didn't know about the service exposure.

Here is a working example thanks to your input:
With proper intentions to allow ingress-demo to speak to whoami1 and whoami2, it works like a charm.

job "ingress-connect-demo" {
  datacenters = ["meow"]

  group "edge" {

    network {
      mode = "bridge"

      port "http1" {
        static = 8901
        to     = 8901
      }

      port "http2" {
        static = 8902
        to     = 8902
      }

      # Treafik dashboard
      port "dash" {
        static = 8909
        to     = 8909
      }
    }

    service {
      name = "ingress-demo"
      port = "http1"
      tags = ["http1"]

      connect {
        native = true
      }
    }

    service {
      name = "ingress-demo"
      port = "http2"
      tags = ["http2"]

#      connect {
#        native = true
#      }
    }

    task "traefik" {
      driver = "docker"
      config {
        image = "traefik:2.6"
        args = [
          # Enables connect support, otherwise only http connections would be tried
          "--providers.consulcatalog.connectaware=true",
          # Make the communication secure by default
          "--providers.consulcatalog.connectbydefault=true",
          "--providers.consulcatalog.exposedbydefault=false",
          "--entrypoints.http1=true",
          "--entrypoints.http1.address=:8901",
          "--entrypoints.http2=true",
          "--entrypoints.http2.address=:8902",
          "--entrypoints.traefik=true",
          "--entrypoints.traefik.address=:8909",
          "--api.dashboard=true",
          "--api.insecure=true",
          # The service name below should match the nomad/consul service above
          # and is used for intentions in consul
          "--providers.consulcatalog.servicename=ingress-demo",
          "--providers.consulcatalog.prefix=ingress-demo",

          # Automatically configured by Nomad through CONSUL_* environment variables
          # as long as client consul.share_ssl is enabled
          # "--providers.consulcatalog.endpoint.address=<socket|address>"
          # "--providers.consulcatalog.endpoint.tls.ca=<path>"
          # "--providers.consulcatalog.endpoint.tls.cert=<path>"
          # "--providers.consulcatalog.endpoint.tls.key=<path>"
          # "--providers.consulcatalog.endpoint.token=<token>"
        ]
      }

      env {
        # Enable this if nomad is older than 1.1.3
        # CONSUL_TLS_SERVER_NAME = "localhost"
      }
    }
  }

  group "backend1" {
    network {
      mode = "bridge"
    }

    service {
      name = "whoami1"
      port = 80
      tags = [
        "ingress-demo.enable=true",
        "ingress-demo.http.routers.whoami1.rule=Host(`whoami1.example.com`)",
        "ingress-demo.http.routers.whoami1.entrypoints=http1"
      ]

      connect {
        sidecar_service {}
      }
    }

    # Note: For increased security the service should only listen on localhost
    # Otherwise it could be reachable from the outside world without going through connect
    task "whoami" {
      driver = "docker"
      config {
        image = "containous/whoami:v1.5.0"
      }
    }
  }

  group "backend2" {
    network {
      mode = "bridge"
    }

    service {
      name = "whoami2"
      port = 80
      tags = [
        "ingress-demo.enable=true",
        "ingress-demo.http.routers.whoami2.rule=Host(`whoami2.example.com`)",
        "ingress-demo.http.routers.whoami2.entrypoints=http2"
      ]

      connect {
        sidecar_service {}
      }
    }

    # Note: For increased security the service should only listen on localhost
    # Otherwise it could be reachable from the outside world without going through connect
    task "whoami" {
      driver = "docker"
      config {
        image = "containous/whoami:v1.5.0"
      }
    }
  }
}

And curl is happy:

$ curl -H "Host: whoami2.example.com" http://whoami2.example.com:8902/health/ --resolve whoami2.example.com:8902:10.xxx.xxx.12
Hostname: 0aa882d6ab30
IP: 127.0.0.1
IP: 172.26.74.189
RemoteAddr: 127.0.0.1:59626
GET /health/ HTTP/1.1
Host: whoami2.example.com
User-Agent: curl/7.81.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: xxx.xxx.xxx.xxx
X-Forwarded-Host: whoami2.example.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: 27c5f7da9048
X-Real-Ip: xxx.xxx.xxx.xxx

$ curl -H "Host: whoami1.example.com" http://whoami1.example.com:8901/health/ --resolve whoami1.example.com:8901:10.xxx.xxx.12
Hostname: ab3616414ad0
IP: 127.0.0.1
IP: 172.26.74.188
RemoteAddr: 127.0.0.1:59644
GET /health/ HTTP/1.1
Host: whoami1.example.com
User-Agent: curl/7.81.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: xxx.xxx.xxx.xxx
X-Forwarded-Host: whoami1.example.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: 27c5f7da9048
X-Real-Ip: xxx.xxx.xxx.xxx

Thanks for your help !

@Allan-Nava
Copy link

The sidecar is necessary for traefik?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment