Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Tutorial: Debugging MQTT traffic flows in IOTstack

Tutorial: Debugging MQTT traffic flows in IOTstack

You have built an ESP32, ESP8266 or similar project. You are sure it is sending payloads via MQTT but the data doesn't seem to be arriving in Node-Red. You're at a bit of a loss as to what to do next.

This tutorial is specific to IOTstack where Mosquitto and Node-Red are running as Docker containers on a Raspberry Pi. Much of it is probably applicable to other environments but "your mileage may vary".

Related resources

Assumptions

  1. SensorsIot/IOTstack is installed (new- or old-menu does not matter).
  2. The Mosquitto configuration is "out of the box".
  3. Mosquitto and Node-Red containers are running and stable (not restarting).

Checking assumption #2

This assumption is very important. The reference version of the Mosquitto configuration is at:

~/IOTstack/.templates/mosquitto/mosquitto.conf

The reference version is updated whenever you run the menu. It can also be updated manually via:

$ cd ~/IOTstack
$ git pull

Typically, your active Mosquitto configuration will be at the path:

~/IOTstack/services/mosquitto/mosquitto.conf

The active Mosquitto configuration is not usually updated by the menu (to avoid overwriting your customisations) so the two versions can easily be out-of-sync, particularly when it comes to essential changes. Run the following:

$ cd ~/IOTstack
$ diff -y ./.templates/mosquitto/mosquitto.conf ./services/mosquitto/mosquitto.conf

You should be able to explain any differences. In particular, your active configuration must contain:

listener 1883

and should not activate a password scheme. The relevant lines should be:

#password_file /mosquitto/pwfile/pwfile
allow_anonymous true

If you have implemented a password scheme, you can re-activate it later.

If you make any changes to your active configuration, apply them like this:

$ cd ~/IOTstack
$ docker-compose restart mosquitto

Checking assumption #3

Run the following command, several times, with a short delay between each run:

$ docker ps --format "table {{.Names}}\t{{.RunningFor}}\t{{.Status}}"

Make sure the "Status" column never mentions "restart". If it does, the most likely cause is a problem in mosquitto.conf. You may get some hints from:

$ docker logs mosquitto

As a second-to-last resort, you may find that some brute force is helpful:

$ cd ~/IOTstack
$ docker-compose stop mosquitto
$ docker-compose rm -rf mosquitto
$ sudo rm -rf ./volumes/mosquitto
$ ./.templates/mosquitto/directoryfix.sh
$ docker-compose up -d mosquitto

Key points:

  • Double-check the sudo command before you hit return. You can do a lot of damage if you get it wrong.
  • Do not make the mistake of changing the working directory before you run the directoryfix.sh script. The script must be run from ~/IOTstack.

The "last resort" is a clean install of IOTstack but that is rarely necessary.

Install MQTT debugging tools

Install the Mosquitto clients.

$ sudo apt install -y mosquitto-clients
$ docker exec nodered apk add --no-cache mosquitto-clients

The first line installs the clients on your Raspberry Pi. They will remain installed until you explicitly uninstall them or rebuild your Raspberry Pi. I install these tools as part of my standard Raspberry Pi build sequence so they are always available.

The second line installs the clients in the running Node-Red container. This installation is temporary and you will have to repeat it each time you recreate the Node-Red container (eg "down" then "up" your stack). See "Hints: Running Node-Red in IOTstack" if you want to learn how to bake the clients into your Node-Red installation.

Make sure Mosquitto is working

Try publishing a message to Mosquitto:

$ mosquitto_pub -h 127.0.0.1 -p 1883 -t "/test" -m "hello world on $(date)"

If you see:

Error: Connection refused

it means Docker and/or Mosquitto are not working. This is a problem you will have to solve before you can continue. Best case is carefully re-checking assumptions #2 and #3. Worst case may involve reinstalling IOTstack.

Mosquitto's service definition includes:

ports:
  - "1883:1883"

If mosquitto_pub returns without error, it means that Docker is listening on the Raspberry Pi's port 1883 and is forwarding any traffic to the Mosquitto container where the Mosquitto broker process is listening on the container's port 1883.

In short, silence from mosquitto_pub means that that message forwarding process is working.

Note:

  • If you have changed the left-hand-side of the ports definition, you will have to substitute the new port number throughout these instructions. There is no good reason for changing the right-hand-side of the ports definition.

Tests

Test 1: subscriber outside container-space

Set up a subscriber process running in the background (the & on the end of the command):

$ mosquitto_sub -v -h 127.0.0.1 -p 1883 -t "/test" -F "%I %t %p" &
[1] xxxxx

The absence of any error means is that mosquitto_sub is "connected" to the Mosquitto broker process running in the Mosquitto container via the 1883:1883 port-forwarding mechanism. It will sit there waiting for messages published to the "/test" topic to be relayed by the broker and will display whatever it receives.

Test 2: publisher outside container-space

Re-send the test message:

$ mosquitto_pub -h 127.0.0.1 -p 1883 -t "/test" -m "hello world on $(date)"
2021-02-27T12:24:21+1100 /test hello world on Sat 27 Feb 2021 12:24:21 PM AEDT

The second line is coming from the mosquitto_sub process running in the background.

Test 3: publisher inside container-space

$ docker exec nodered mosquitto_pub -h mosquitto -p 1883 -t "/test" -m "hello world on $(date)"
2021-02-27T12:31:00+1100 /test hello world on Sat 27 Feb 2021 12:31:00 PM AEDT

Note the subtle difference between -h 127.0.0.1 of the earlier command and -h mosquitto on this command. Outside container-space, "127.0.0.1" means "this Raspberry Pi". Inside the Node-Red container, "127.0.0.1" means "this container". Mosquitto is running in a different container so it can't be reached using "127.0.0.1". You use the destination container name instead. Docker provides the mapping between container names and containers.

Key point:

  • This has implications for your Node-Red flows. If you are following an example you found on the Internet and it specifies "127.0.0.1", you must change to "mosquitto" so that it will work in IOTstack. This is explained in Recipe: from MQTT to InfluxDB via Node-Red but it's subtle and easy to miss.

Kill the background subscriber process by running the following command and pressing RETURN a second time:

$ kill %1
$
[1]+  Terminated mosquitto_sub -v -h 127.0.0.1 -p 1883 -t "/test" -F "%I %t %p"

Test 4: subscriber inside container-space

Set up a background listener:

$ docker exec nodered mosquitto_sub -v -h mosquitto -p 1883 -t "/test" -F "%I %t %p" &
[1] xxxx

Test 5: publisher outside container-space

Re-send the test message:

$ mosquitto_pub -h 127.0.0.1 -p 1883 -t "/test" -m "hello world on $(date)"
2021-02-27T12:40:13+1100 /test hello world on Sat 27 Feb 2021 12:40:13 PM AEDT

Test 6: publisher inside container-space

Re-send the test message:

$ docker exec nodered mosquitto_pub -h mosquitto -p 1883 -t "/test" -m "hello world on $(date)"
2021-02-27T12:47:56+1100 /test hello world on Sat 27 Feb 2021 12:47:55 PM AEDT

Kill the background process:

$ kill %1
$
[1]+  Terminated docker exec nodered mosquitto_sub -v -h mosquitto -p 1883 -t "/test" -F "%I %t %p"

What does this all prove?

With respect to inside vs outside container-space, the tests show:

Tests Publisher Subscriber Message flow
1 + 2 outside outside client to client
1 + 3 inside outside Node-Red to client
4 + 5 outside inside client to Node-Red
4 + 6 inside inside Node-Red to Node-Red

In a typical IoT scenario, a client device (ESP32 or ESP8266) will be transmitting telemetry via MQTT which it expects Node-Red to process. This is the "client to Node-Red" flow of which "weather sensor logs data" is the classic example.

Node-Red may utilise the "Node-Red to client" flow (probably in conjunction with the retain flag) to tell a client to take an action like "turn the light off or on".

The "client to client" flow could also be used without involving Node-Red to send a message from, say, an iDevice to "turn the light off or on". It is also useful for general debugging, as in this script:

#!/usr/bin/env sh

# set defaults (pass "#" including the quotes to listen to everything)
TOPIC=${1:-"#"}
BROKER=${2:-"127.0.0.1"}

mosquitto_sub -v -h "$BROKER" -t "$TOPIC" -F "%I %t %p"

That can be launched in either the foreground or background as needed. I call this script "mossieMonitor" and it is in my search path on all of my computers.

If you replace the "127.0.0.1" default with the domain name or IP address of your Raspberry Pi running IOTstack then just typing:

$ mossieMonitor

will immediately start listening and relaying all of your MQTT traffic as it arrives at your IOTstack device.

The next steps

Test from another computer

The tests described above prove that MQTT traffic can get into and out of container-space, and can move between containers. But this is all on the same machine.

The next step is to install the Mosquitto clients on another computer and repeat the relevant tests there. What you will actually be testing is your local communications infrastructure.

If that works then there is no reason why an ESP32 or ESP8266 client will not also be able to send and receive messages via MQTT.

Test from a Node-Red flow

Drag an "mqtt-in" node plus a "debug" node onto the canvas and wire them together.

Configure the "mqtt-in" to listen to the "/test" topic. Make sure that the "Server" popup leads to a server configuration of "mosquitto" on port "1883".

Keep in mind that MQTT Server configurations (the entries you see in the "Server" popup menu) are global to all your flows. Don't just change an MQTT Server configuration that is working because that will affect the flows that already depend on it. Don't assume you can copy an "mqtt-in" node from one flow, paste it into another flow, and then tailor its MQTT Server configuration. It doesn't work that way. Laziness is your enemy in Node-Red. When in doubt, choose "Add new mqtt broker…" from the popup menu and start from scratch.

Deploy the flow, make sure you can see the "Debug" screen, and send the test message as above. You should see the "hello world" payload in the debug window.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment