traefik: node-red + mosquitto using letsencrypt

docker-compose.yaml

version: "3.3"

services:

  traefik:
    image: "traefik:v2.2"
    container_name: "traefik"
    command:
      - "--api=true"
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
        
        # Entrypoints 
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.mqtt.address=:8883"
      
        # Redirect http to https
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      
        # Let's encrypt configuration
      - "--certificatesresolvers.myresolver.acme.tlschallenge=true"
      - "--certificatesresolvers.myresolver.acme.email=email@host.com"
      - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
    ports:
      - "80:80"
      - "443:443"
      - "8883:8883"
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    labels:
      - "traefik.enable=true"
      
      - "traefik.http.routers.dashboard.rule=Host(`traefik.zoo.ocean.mofa.studio`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      - "traefik.http.routers.dashboard.entrypoints=websecure"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.tls.certresolver=myresolver"
      - "traefik.http.routers.dashboard.middlewares=auth"
      - "traefik.http.middlewares.auth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/,test2:$$apr1$$d9hr9HBB$$4HxwgUir3HP4EsggP/QNo0"

  whoami:
    image: "containous/whoami"
    container_name: "simple-service"
    labels:
      - "traefik.enable=true"

      - "traefik.http.routers.whoami.rule=Host(`whoami.zoo.ocean.mofa.studio`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls.certresolver=myresolver"
  
  mqtt:
    image: "eclipse-mosquitto"
    container_name: "mosquitto"
    expose:
      - "8883"
      - "9001"
    volumes:
      - "./mosquitto.conf:/mosquitto/config/mosquitto.conf"
    labels:
      - "traefik.enable=true"

      - "traefik.http.routers.mqtt.rule=Host(`mqtt.zoo.ocean.mofa.studio`)"
      - "traefik.http.routers.mqtt.entrypoints=websecure"
      - "traefik.http.routers.mqtt.tls.certresolver=myresolver"

      - "traefik.tcp.routers.mqtt.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.mqtt.tls.certresolver=myresolver"
      - "traefik.tcp.services.mqtt.loadbalancer.server.port=8883"
      
      - "traefik.tcp.routers.mqtt.entrypoints=mqtt"

      - "traefik.http.services.mqtt.loadbalancer.server.port=9001"

  nodered:
    image: "nodered/node-red"
    container_name: "nodered"
    labels:
      - "traefik.enable=true"

      - "traefik.http.routers.nodered.rule=Host(`nodered.zoo.ocean.mofa.studio`)"
      - "traefik.http.routers.nodered.entrypoints=websecure"
      - "traefik.http.routers.nodered.tls.certresolver=myresolver"

      - "traefik.http.services.nodered.loadbalancer.server.port=1880"

mosquitto.conf

port 8883

listener 9001
protocol websockets

That’s normal behaviour.
You’ll have to check the if the mqtt library you’ll be using uses the OS certificate store or wether you need to include it.
If the goal is to secure your broker so that only your devices can connect you will also need to setup password authentication. Make sure the password is different for ssl vs unencrypted because it’ll pass as clear text.

Is your mqtt websocket connection working using this docker-compose.yaml ? If I understand the traefik info correctly, you don't capture traffic on the outside port 9001, but use the 443 port and redirect to the 9001 port of the mqtt docker ? I can't get this to work .... Thanks for your help

Yes that’s correct, mqtt over websocket is on port 443 in this example so you client needs to use port 443 in secured web socket mode.

After hours of debugging I finally got it ... Your script works perfectly, The error was with the websocket client (hivemq) which for some reason only wants to use the default 9001 ws port (even if a different port is specified). Thanks for sharing !

Nice ! glad to know it worked ! Thanks for the update !

Hi @natcl,

I am trying to use your instructions to get mosquitto working with traefik and for whatever reason I get a connection refused error through MQTT explorer. I've looked at traefik and see the following error with my mqtt tcp service:

I can telnet into that IP/port from my docker server, so the port is working. I have setup things to use slightly different naming, but everything else should be the same. Here is the code that I am using:

mosquitto traefik labels:

  mosquitto:
    image: eclipse-mosquitto
    container_name: mosquitto
    restart: unless-stopped
    ports:
      - 1883:1883
      - 8884:8883
      - 9001:9001
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ${DOCKER}/mosquitto/data:/mosquitto/data
      - ${DOCKER}/mosquitto/config:/mosquitto/config
      - ${DOCKER}/mosquitto/log:/mosquitto/log
    environment:
      - TZ=${TZ}
    labels:
      - "traefik.enable=true"

      - "traefik.http.routers.mqtt.rule=Host(`mqtt.${DOMAINNAME}`)"
      - "traefik.http.routers.mqtt.entrypoints=https"
      - "traefik.http.routers.mqtt.tls.certresolver=http"

      - "traefik.tcp.routers.mqtt.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.mqtt.tls.certresolver=http"
      - "traefik.tcp.services.mqtt.loadbalancer.server.port=8883"

      - "traefik.tcp.routers.mqtt.entrypoints=mqtt"

      - "traefik.http.services.mqtt.loadbalancer.server.port=9001"
      - "traefik.docker.network=public_facing"

mosquitto conf:

persistence true
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log
allow_anonymous true
listener 1883
port 8883
listener 9001
protocol websockets

output from my mosquitto.log:

1619451746: mosquitto version 2.0.10 starting
1619451746: Config loaded from /mosquitto/config/mosquitto.conf.
1619451746: Opening ipv4 listen socket on port 1883.
1619451746: Opening ipv6 listen socket on port 1883.
1619451746: Opening websockets listen socket on port 9001.
1619451746: Opening ipv4 listen socket on port 8883.
1619451746: Opening ipv6 listen socket on port 8883.
1619451746: mosquitto version 2.0.10 running

Any ideas on why this would happen?

What port are you trying to connect to from MQTT explorer ?

I am using the following:

I don't have a username/password setup yet, at this point I am just trying to connect anonymously.

Can you show the main traefik config ?

Sure:

docker compose piece:

  traefik:
    image: traefik:v2.4
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - public_facing
    ports:
      - "80:80"
      - "443:443"
      - "8883:8883"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ${DOCKER}/traefik/acme.json:/acme.json
      - ${DOCKER}/traefik/traefik.yml:/traefik.yml:ro
      - ${DOCKER}/traefik/users:/users:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik.${DOMAINNAME}`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.usersfile=/users"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.${DOMAINNAME}`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=http"
      - "traefik.http.routers.traefik-secure.service=api@internal"

traefik.yml

api:
  dashboard: true

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"
  mqtt:
    address: ":8883"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false

certificatesResolvers:
  http:
    acme:
      email: fake@email.com
      storage: acme.json
      httpChallenge:
        entryPoint: http

Can you try using my examples to see if it works ? Yours are quite modified so I'm not sure where the issue could be....

Nice docker script. But for using TCP TLS with 8883 port you should change protocol to mqtt instead websocket into mosquitto.conf .

The configuration works for mosquitto 1.x but the ´port’ configuration is deprecated so I will update it.

The goal is to have both TCP and websockets, that’s why both configurations are there.

In this way, I'm guessing some malicious client can also pub/sub to your mqtt broker if somebody knows your server URI.
Am I correct??

I'm trying to setup client certificate type of authentication but with no luck. mqtt broker is behind Traefik as you do.
a reference of client certificate type of authentication is here:
https://primalcortex.wordpress.com/2016/11/08/mqtt-mosquitto-broker-client-authentication-and-client-certificates/
Do you have any experience on this??

I use mosquitto authentication to secure the broker, could this be enough for your use case ?

That may work, I'll give it a try

…

On Thu, May 19, 2022, 6:40 AM Nathanaël Lécaudé ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ I use mosquitto authentication to secure the broker, could this be enough for your use case ? — Reply to this email directly, view it on GitHub <https://gist.github.com/ed8253a34e7b87d879baabeba82cb846#gistcomment-4172110>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMO6L75PZQAOA7NACKWG3TTVKYSDZANCNFSM4T7ORYGQ> . You are receiving this because you commented.Message ID: ***@***.***>

Thank you for your help!
Very short answer is No.
Why I am doing this because I'd like to run eclipse-mosquitto broker (mqqts) on 443/tcp to make some firewalls happy.
My broker is a part of Netmaker (https://github.com/gravitl/netmaker) system that needs client certificate authentication.
I tried tls.passthrough with hostSNI("*") but no luck,,

I believe this could work but I doubt you'll be able to make it work using let's encrypt, you'll probably need you own certificates.

Yes, without reverse proxy, client cert authentication works perfect with own ca. but behind traefik it doesn’t work,,
Thank you

Hi, a doubt..in your docker-compose file why are you using the entrypoint for 8883 if you are using the websecure entrypoint in the mqtt section config?

@iboluda The 8883 entry point is for TCP connections while the 443 is for web sockets, does that answer your question ?

Thanks for clarify me that point. Yes that answer my question