-
-
Save apollo13/857ae4c5e18de619815c2628212449e1 to your computer and use it in GitHub Desktop.
# Simple example to deploy traefik with consul connect enabled. | |
# For simplicity the job includes traefik as well as the backend service. | |
# Please note that traefik currently only supports connect for HTTP. | |
job "traefik-consul-connect-demo" { | |
datacenters = ["dc1"] | |
group "edge" { | |
network { | |
mode = "bridge" | |
port "http" { | |
to = 8080 | |
} | |
} | |
service { | |
name = "traefik-ingress" | |
port = "http" | |
connect { | |
native = true | |
} | |
} | |
task "traefik" { | |
driver = "docker" | |
config { | |
image = "traefik:v2.5.2" | |
args = [ | |
# Enables connect support, otherwise only http connections would be tried | |
"--providers.consulcatalog.connectaware=true", | |
# Make the communication secure by default | |
"--providers.consulcatalog.connectbydefault=true", | |
"--providers.consulcatalog.exposedbydefault=false", | |
"--entrypoints.http=true", | |
"--entrypoints.http.address=:8080", | |
# The service name below should match the nomad/consul service above | |
# and is used for intentions in consul | |
"--providers.consulcatalog.servicename=traefik-ingress", | |
"--providers.consulcatalog.prefix=traefik", | |
# Automatically configured by Nomad through CONSUL_* environment variables | |
# as long as client consul.share_ssl is enabled | |
# "--providers.consulcatalog.endpoint.address=<socket|address>" | |
# "--providers.consulcatalog.endpoint.tls.ca=<path>" | |
# "--providers.consulcatalog.endpoint.tls.cert=<path>" | |
# "--providers.consulcatalog.endpoint.tls.key=<path>" | |
# "--providers.consulcatalog.endpoint.token=<token>" | |
] | |
} | |
env { | |
# Enable this if nomad is older than 1.1.3 | |
# CONSUL_TLS_SERVER_NAME = "localhost" | |
} | |
} | |
} | |
group "backend" { | |
network { | |
mode = "bridge" | |
} | |
service { | |
name = "whoami" | |
port = 80 | |
tags = [ | |
"traefik.enable=true", | |
"traefik.http.routers.whoami.rule=Host(`whoami.example.com`)" | |
] | |
connect { | |
sidecar_service {} | |
} | |
} | |
# Note: For increased security the service should only listen on localhost | |
# Otherwise it could be reachable from the outside world without going through connect | |
task "whoami" { | |
driver = "docker" | |
config { | |
image = "containous/whoami" | |
} | |
} | |
} | |
} |
Enables connect support, otherwise only http connections would be tried
"--providers.consulcatalog.connectaware=true",I suppose this is what needs to be done differently?
Differently as opposed to what? Setting this flag allows traefik to talk to the connect sidecars and secure the connection. By itself it doesn't change behavior yet. You still need to use the global connectbydefault
config or traefik.consulcatalog.connect=true
per service.
As for hashicorp/consul#9643 (comment) I maybe should have been clearer. Adding the tags is enough if you are securing your connections ot the backend servers. If you simply wish to use traefik like before and do not encrypt the communication you still have to have the dummy
tag, otherwise the sidecar will be considered a valid service. Technically I think the later might be (relatively easy) fixable, as traefik could just filter services that have a kind set to connect-proxy
-- but I haven't thought about the implications of that.
Oh, just a side note, I felt like I recognized your username from somewhere, and while I'm reading the Traefik blog post, I saw your name come up, and I remembered that you did a lot of the heavy lifting on this functionality. Thank you so much for your work! 😃
Also, I think I figured it out. I was adding the "traefik.consulcatalog.connect=true"
tag on a normal service that had a sidecar hooked up with connect
service {
name = "shynet-django"
port = "shynet"
tags = [
"traefik.enable=true",
"traefik.http.routers.shynet.rule=Host(`shynet.discretemath.ca`)",
"traefik.http.routers.shynet.entrypoints=https",
"traefik.http.routers.shynet.tls.certresolver=letsencrypt",
"traefik.consulcatalog.connect=true" # <-- The problem
]
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "shynet-postgres"
local_bind_port = 5432
}
}
tags = [
"dummy"
]
}
}
}
Thank you so much for your work!
Glad you like it!
<-- The problem
Oh yes, if you are using the sidecar only for the purpose of using encrypted connections to upstreams and not to encrypt inbound connections you should not set traefik.consulcatalog.connect=true
.
Thank you guys. Without the traefik.consulcatalog.connect=true
I still need to set the dummy tag to the sidecar service?
Hi !
Thanks for that example, works like a charm!
I've one interrogation about the providers.consulcatalog.servicename
that should match the nomad service name in the service
stanza.
It seems that this requirement makes it impossible to route multiple service via that traefik instance, am I right ?
The goal would be to use traefik deployed as a system job on each nomad client, and being able to serve as ingress-gateway for multiple services inside the service mesh.
That might be a limitation of the Traefik integration with consul-connect but I could not find proper explanation/documentation for this.
rgd,
It seems that this requirement makes it impossible to route multiple service via that traefik instance, am I right ?
What makes you say that?
It seems that this requirement makes it impossible to route multiple service via that traefik instance, am I right ?
What makes you say that?
I want to use multiple services using different entrypoints, which means multiple service
stanza, I don't understand how to make that possible using the service-mesh integration.
ie I can configure multiple service stanza, but only one can be exposed to consul-catalog via service-name.
The different entrypoint
is probably what is missing from my first comment
Here is a job spec for reference: basically adding one more entrypoint and duplicating the backend section:
I could not make both the http1-ingress-demo
and http2-ingress-demo
working at the same time.
job "ingress-connect-demo" {
datacenters = ["meow"]
group "edge" {
network {
mode = "bridge"
port "http1" {
static = 8901
to = 8901
}
port "http2" {
static = 8902
to = 8902
}
}
service {
name = "http1-ingress-demo"
port = "http1"
connect {
native = true
}
}
service {
name = "http2-ingress-demo"
port = "http2"
connect {
native = true
}
}
task "traefik" {
driver = "docker"
config {
image = "traefik:2.6"
args = [
# Enables connect support, otherwise only http connections would be tried
"--providers.consulcatalog.connectaware=true",
# Make the communication secure by default
"--providers.consulcatalog.connectbydefault=true",
"--providers.consulcatalog.exposedbydefault=false",
"--entrypoints.http1=true",
"--entrypoints.http1.address=:8901",
"--entrypoints.http2=true",
"--entrypoints.http2.address=:8902",
# The service name below should match the nomad/consul service above
# and is used for intentions in consul
"--providers.consulcatalog.servicename=http1-ingress-demo",
# "--providers.consulcatalog.servicename=http2-ingress-demo",
"--providers.consulcatalog.prefix=ingress-demo",
# Automatically configured by Nomad through CONSUL_* environment variables
# as long as client consul.share_ssl is enabled
# "--providers.consulcatalog.endpoint.address=<socket|address>"
# "--providers.consulcatalog.endpoint.tls.ca=<path>"
# "--providers.consulcatalog.endpoint.tls.cert=<path>"
# "--providers.consulcatalog.endpoint.tls.key=<path>"
# "--providers.consulcatalog.endpoint.token=<token>"
]
}
env {
# Enable this if nomad is older than 1.1.3
# CONSUL_TLS_SERVER_NAME = "localhost"
}
}
}
group "backend1" {
network {
mode = "bridge"
}
service {
name = "whoami1"
port = 80
tags = [
"ingress-demo.enable=true",
"ingress-demo.http.routers.whoami1.rule=Host(`whoami1.example.com`)",
"ingress-demo.http.routers.whoami1.entrypoints=http1"
]
connect {
sidecar_service {}
}
}
# Note: For increased security the service should only listen on localhost
# Otherwise it could be reachable from the outside world without going through connect
task "whoami" {
driver = "docker"
config {
image = "containous/whoami:v1.5.0"
}
}
}
group "backend2" {
network {
mode = "bridge"
}
service {
name = "whoami2"
port = 80
tags = [
"ingress-demo.enable=true",
"ingress-demo.http.routers.whoami2.rule=Host(`whoami2.example.com`)",
"ingress-demo.http.routers.whoami2.entrypoints=http2"
]
connect {
sidecar_service {}
}
}
# Note: For increased security the service should only listen on localhost
# Otherwise it could be reachable from the outside world without going through connect
task "whoami" {
driver = "docker"
config {
image = "containous/whoami:v1.5.0"
}
}
}
}
If you have a way to make it work, I take it :)
That being said, if you use the same entrypoint for both backends, and route differently based on smth (like HOST header) it works as expected.
job "ingress-connect-demo" {
datacenters = ["meow"]
group "edge" {
network {
mode = "bridge"
port "http1" {
static = 8901
to = 8901
}
}
service {
name = "http1-ingress-demo"
port = "http1"
connect {
native = true
}
}
task "traefik" {
driver = "docker"
config {
image = "traefik:2.6"
args = [
# Enables connect support, otherwise only http connections would be tried
"--providers.consulcatalog.connectaware=true",
# Make the communication secure by default
"--providers.consulcatalog.connectbydefault=true",
"--providers.consulcatalog.exposedbydefault=false",
"--entrypoints.http1=true",
"--entrypoints.http1.address=:8901",
# The service name below should match the nomad/consul service above
# and is used for intentions in consul
"--providers.consulcatalog.servicename=http1-ingress-demo",
"--providers.consulcatalog.prefix=ingress-demo",
# Automatically configured by Nomad through CONSUL_* environment variables
# as long as client consul.share_ssl is enabled
# "--providers.consulcatalog.endpoint.address=<socket|address>"
# "--providers.consulcatalog.endpoint.tls.ca=<path>"
# "--providers.consulcatalog.endpoint.tls.cert=<path>"
# "--providers.consulcatalog.endpoint.tls.key=<path>"
# "--providers.consulcatalog.endpoint.token=<token>"
]
}
env {
# Enable this if nomad is older than 1.1.3
# CONSUL_TLS_SERVER_NAME = "localhost"
}
}
}
group "backend1" {
network {
mode = "bridge"
}
service {
name = "whoami1"
port = 80
tags = [
"ingress-demo.enable=true",
"ingress-demo.http.routers.whoami1.rule=Host(`whoami1.example.com`)",
"ingress-demo.http.routers.whoami1.entrypoints=http1"
]
connect {
sidecar_service {}
}
}
# Note: For increased security the service should only listen on localhost
# Otherwise it could be reachable from the outside world without going through connect
task "whoami" {
driver = "docker"
config {
image = "containous/whoami:v1.5.0"
}
}
}
group "backend2" {
network {
mode = "bridge"
}
service {
name = "whoami2"
port = 80
tags = [
"ingress-demo.enable=true",
"ingress-demo.http.routers.whoami2.rule=Host(`whoami2.example.com`)",
"ingress-demo.http.routers.whoami2.entrypoints=http1"
]
connect {
sidecar_service {}
}
}
# Note: For increased security the service should only listen on localhost
# Otherwise it could be reachable from the outside world without going through connect
task "whoami" {
driver = "docker"
config {
image = "containous/whoami:v1.5.0"
}
}
}
}
I hope that it makes thing clearer
You can use as many entrypoints and services as you want. But traefik is just one application, it will only use one certificate (which is the one from providers.consulcatalog.servicename
and also the one you will have to use in intentions).
I use my traefik service like this (for two entrypoints):
service {
name = "traefik"
port = "port1"
tags = ["port1"]
connect {
native = true
}
}
service {
name = "traefik"
port = "port2"
tags = ["port2"]
}
Note that one service can expose multiple ports (via tags) and one (!) connect {}
stanza is enough.
You can use as many entrypoints and services as you want. But traefik is just one application, it will only use one certificate (which is the one from
providers.consulcatalog.servicename
and also the one you will have to use in intentions).
Note that one service can expose multiple ports (via tags) and one (!)connect {}
stanza is enough.
Thanks a lot ! That's something I didn't know about the service exposure.
Here is a working example thanks to your input:
With proper intentions to allow ingress-demo
to speak to whoami1
and whoami2
, it works like a charm.
job "ingress-connect-demo" {
datacenters = ["meow"]
group "edge" {
network {
mode = "bridge"
port "http1" {
static = 8901
to = 8901
}
port "http2" {
static = 8902
to = 8902
}
# Treafik dashboard
port "dash" {
static = 8909
to = 8909
}
}
service {
name = "ingress-demo"
port = "http1"
tags = ["http1"]
connect {
native = true
}
}
service {
name = "ingress-demo"
port = "http2"
tags = ["http2"]
# connect {
# native = true
# }
}
task "traefik" {
driver = "docker"
config {
image = "traefik:2.6"
args = [
# Enables connect support, otherwise only http connections would be tried
"--providers.consulcatalog.connectaware=true",
# Make the communication secure by default
"--providers.consulcatalog.connectbydefault=true",
"--providers.consulcatalog.exposedbydefault=false",
"--entrypoints.http1=true",
"--entrypoints.http1.address=:8901",
"--entrypoints.http2=true",
"--entrypoints.http2.address=:8902",
"--entrypoints.traefik=true",
"--entrypoints.traefik.address=:8909",
"--api.dashboard=true",
"--api.insecure=true",
# The service name below should match the nomad/consul service above
# and is used for intentions in consul
"--providers.consulcatalog.servicename=ingress-demo",
"--providers.consulcatalog.prefix=ingress-demo",
# Automatically configured by Nomad through CONSUL_* environment variables
# as long as client consul.share_ssl is enabled
# "--providers.consulcatalog.endpoint.address=<socket|address>"
# "--providers.consulcatalog.endpoint.tls.ca=<path>"
# "--providers.consulcatalog.endpoint.tls.cert=<path>"
# "--providers.consulcatalog.endpoint.tls.key=<path>"
# "--providers.consulcatalog.endpoint.token=<token>"
]
}
env {
# Enable this if nomad is older than 1.1.3
# CONSUL_TLS_SERVER_NAME = "localhost"
}
}
}
group "backend1" {
network {
mode = "bridge"
}
service {
name = "whoami1"
port = 80
tags = [
"ingress-demo.enable=true",
"ingress-demo.http.routers.whoami1.rule=Host(`whoami1.example.com`)",
"ingress-demo.http.routers.whoami1.entrypoints=http1"
]
connect {
sidecar_service {}
}
}
# Note: For increased security the service should only listen on localhost
# Otherwise it could be reachable from the outside world without going through connect
task "whoami" {
driver = "docker"
config {
image = "containous/whoami:v1.5.0"
}
}
}
group "backend2" {
network {
mode = "bridge"
}
service {
name = "whoami2"
port = 80
tags = [
"ingress-demo.enable=true",
"ingress-demo.http.routers.whoami2.rule=Host(`whoami2.example.com`)",
"ingress-demo.http.routers.whoami2.entrypoints=http2"
]
connect {
sidecar_service {}
}
}
# Note: For increased security the service should only listen on localhost
# Otherwise it could be reachable from the outside world without going through connect
task "whoami" {
driver = "docker"
config {
image = "containous/whoami:v1.5.0"
}
}
}
}
And curl is happy:
$ curl -H "Host: whoami2.example.com" http://whoami2.example.com:8902/health/ --resolve whoami2.example.com:8902:10.xxx.xxx.12
Hostname: 0aa882d6ab30
IP: 127.0.0.1
IP: 172.26.74.189
RemoteAddr: 127.0.0.1:59626
GET /health/ HTTP/1.1
Host: whoami2.example.com
User-Agent: curl/7.81.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: xxx.xxx.xxx.xxx
X-Forwarded-Host: whoami2.example.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: 27c5f7da9048
X-Real-Ip: xxx.xxx.xxx.xxx
$ curl -H "Host: whoami1.example.com" http://whoami1.example.com:8901/health/ --resolve whoami1.example.com:8901:10.xxx.xxx.12
Hostname: ab3616414ad0
IP: 127.0.0.1
IP: 172.26.74.188
RemoteAddr: 127.0.0.1:59644
GET /health/ HTTP/1.1
Host: whoami1.example.com
User-Agent: curl/7.81.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: xxx.xxx.xxx.xxx
X-Forwarded-Host: whoami1.example.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: 27c5f7da9048
X-Real-Ip: xxx.xxx.xxx.xxx
Thanks for your help !
The sidecar is necessary for traefik?
I suppose this is what needs to be done differently?