You have built an ESP32, ESP8266 or similar project. You are sure it is sending payloads via MQTT but the data doesn't seem to be arriving in Node-Red. You're at a bit of a loss as to what to do next.
This tutorial is specific to IOTstack where Mosquitto and Node-Red are running as Docker containers on a Raspberry Pi. Much of it is probably applicable to other environments but "your mileage may vary".
- IOTstack Mosquitto documentation
- Recipe: from MQTT to InfluxDB via Node-Red
- Hints: Running Node-Red in IOTstack
- Useful aliases for working with IOTstack
- Tutorial: MQTT and the retain flag
- SensorsIot/IOTstack is installed (new- or old-menu does not matter).
- The Mosquitto configuration is "out of the box".
- Mosquitto and Node-Red containers are running and stable (not restarting).
This assumption is very important. The reference version of the Mosquitto configuration is at:
~/IOTstack/.templates/mosquitto/iotstack_defaults/config/mosquitto.confNote:
- If you can't find that file on your instance of IOTstack, it means you have not updated against GitHub in a while. See the
git pullbelow.
The reference version is updated whenever you run the menu. It can also be updated manually via:
$ cd ~/IOTstack
$ git pullYour active Mosquitto configuration will be at the path:
~/IOTstack/volumes/mosquitto/config/mosquitto.confNote:
- If you can't find that file on your instance of IOTstack, it means you are running an older deployment of Mosquitto. See the IOTstack Mosquitto documentation and, in particular, the section on migration.
Your active active Mosquitto configuration is not updated by either the menu or git. This is to avoid overwriting your customisations.
Occasionally, however, the reference version of the Mosquitto configuration has to be changed to cope with decisions made by Mosquitto's maintainers. Issue 265 is a good example of this kind of situation.
Because of this possibility, if you are having trouble with Mosquitto, it is a good idea to make sure that your active active Mosquitto configuration is the same as the reference version, or that you can explain any differences.
To compare the reference and active versions:
$ cd ~/IOTstack
$ diff ./.templates/mosquitto/iotstack_defaults/config/mosquitto.conf ./volumes/mosquitto/config/mosquitto.confIf you get "silence" then it means the two files are the same. If you get differences, you can also repeat the comparison in side-by-side format by adding the -y option after diff.
In particular, your active configuration must contain:
listener 1883
and should not activate a password scheme. The relevant lines should be:
#password_file /mosquitto/pwfile/pwfile
allow_anonymous true
If you have implemented a password scheme, you can re-activate it later.
If you make any changes to your active configuration, apply them like this:
$ cd ~/IOTstack
$ docker-compose restart mosquittoRun the following command, several times, with a short delay between each run:
$ docker ps --format "table {{.Names}}\t{{.RunningFor}}\t{{.Status}}"Make sure the "Status" column never mentions "restart". If it does, the most likely cause is a problem in mosquitto.conf. You may get some hints from:
$ docker logs mosquittoAs a second-to-last resort, you may find that some brute force is helpful:
$ cd ~/IOTstack
$ docker-compose stop mosquitto
$ docker-compose rm -rf mosquitto
$ sudo rm -rf ./volumes/mosquitto
$ docker-compose up -d mosquittoNote:
- Double-check the
sudocommand before you hit return. You can do a lot of damage if you get it wrong.
The "very last resort" is a clean install of IOTstack but that is rarely necessary.
Install the Mosquitto clients.
$ sudo apt install -y mosquitto-clients
$ docker exec nodered apk add --no-cache mosquitto-clientsThe first line installs the clients on your Raspberry Pi. They will remain installed until you explicitly uninstall them or rebuild your Raspberry Pi. I install these tools as part of my standard Raspberry Pi build sequence so they are always available.
The second line installs the clients in the running Node-Red container. This installation is temporary and you will have to repeat it each time you recreate the Node-Red container (eg "down" then "up" your stack). See "Hints: Running Node-Red in IOTstack" if you want to learn how to bake the clients into your Node-Red installation.
Assume two devices:
-
A Raspberry Pi running IOTstack where
docker-compose.ymlincludes service definitions for bothmosquittoandnodered. The service definition formosquittoincludes this port mapping:ports: - "1883:1883"
The port on the left hand side (ie "1883:") is called the external port. The port on the right hand side (ie ":1883") is called the internal port.
-
A client device like an ESP32.
The model shows the devices connected by Ethernet to help with visualising the traffic flows but it makes no difference if WiFi is the data-link media.
The diagram contains the reference command:
$ mosquitto_pub -h 192.168.203.60 -p 1883 -t "/topic" -m "hello"In words, "publish the MQTT message hello against topic /topic to the host with the IP address 192.168.203.60 using port 1883 ."
A client like an ESP32 is more likely to be making a call to an MQTT API but the same principles apply.
When a client wishes to publish an MQTT message, it addresses the packet to the Raspberry Pi using the external port. The IP address can be replaced with the Raspberry Pi's fully-qualified domain name or multicast DNS name (eg raspberrypi.local).
On the Raspberry Pi, Docker is listening to the external port. Docker is doing that because of the left hand side of the port mapping in docker-compose.yml.
Docker receives the MQTT packet and uses Network Address Translation rules to re-address the packet:
- The destination port is set to the internal port 1883, which it gets from the right hand side of the port mapping in
docker-compose.yml. - The destination IP address is set to 172.18.0.6 which Docker derives from the fact that it knows which container is associated with external port 1883 (there can only be one external port 1883).
Docker then routes (Layer Three) the packet across the internal bridged network.
Inside the Mosquitto container, the Mosquitto broker is listening to the container port 1883. Mosquitto gets the port number from listener 1883 in its configuration file. Mosquitto is not aware of docker-compose.yml because that is outside of container-space.
Each container is like an independent computer so listener 1883 implies listener 0.0.0.0:1883, where the all-zeroes address means "all interfaces on this [virtual] computer".
This is not shown on the diagram but a reference command would be:
$ mosquitto_sub -h 192.168.203.60 -p 1883 -t "/topic"and the traffic flows are identical to Client publishes to broker.
Two reference commands are shown in the diagram, assumed to be running in a Terminal window on the same Raspberry Pi that is running Docker:
$ mosquitto_pub -h 127.0.0.1 -p 1883 -t "/topic" -m "hello"
$ mosquitto_sub -h 127.0.0.1 -p 1883 -t "/topic"The only difference is using the loopback address of 127.0.0.1. There is, in fact, no reason not to use either the IP address of the Raspberry Pi, or its fully-qualified or multicast domain name. The traffic flows are identical to Client publishes to broker and Client subscribes to broker.
The diagram contains the reference command:
$ mosquitto_sub -h mosquitto -p 1883 -t "/topic"This is typical of what happens in a Node-RED flow when you use the MQTT-in node. In this situation:
-
the "mosquitto" in
-h mosquittorefers to the name of the container. Given the following snippet fromdocker-compose.yml:mosquitto: container_name: mosquitto build: ./.templates/mosquitto/. restart: unless-stopped …
The right hand side of the
container_name:field provides a mapping between the name "mosquitto" and the dynamic IP address assigned to the container by Docker when the container is instantiated. -
the "1883" in
-p 1883is the port the Mosquitto broker is listening on, which comes from thelistener 1883in its configuration file.
However, the above only applies where both Mosquitto and Node-RED are running as non-host-mode containers in the same Docker instance. This is the most common situation.
If you are trying to use a different configuration, you may find that studying the following diagram will help you work out the correct addressing:
This is not shown on the diagram but a reference command would be:
$ mosquitto_pub -h mosquitto -p 1883 -t "/topic" -m "hello"The traffic flows are identical to Node-RED flow subscribes to broker.
From the Raspberry Pi command line, try publishing a message to Mosquitto:
$ mosquitto_pub -h 127.0.0.1 -p 1883 -t "/test" -m "hello world on $(date)"If you see:
Error: Connection refusedit means Docker and/or Mosquitto are not working. This is a problem you will have to solve before you can continue. Best case is carefully re-checking assumptions #2 and #3. Worst case may involve reinstalling IOTstack.
Mosquitto's service definition includes:
ports:
- "1883:1883"If mosquitto_pub returns without error, it means that Docker is listening on the Raspberry Pi's port 1883 and is forwarding any traffic to the Mosquitto container where the Mosquitto broker process is listening on the container's port 1883.
In short, silence from mosquitto_pub means that that message forwarding process is working.
Note:
- If you have changed the left-hand-side of the ports definition, you will have to substitute the new port number throughout these instructions. There really is no good reason for changing the right-hand-side of the ports definition.
Set up a subscriber process running in the background (the & on the end of the command):
$ mosquitto_sub -v -h 127.0.0.1 -p 1883 -t "/test" -F "%I %t %p" &
[1] xxxxxThe absence of any error means is that mosquitto_sub is "connected" to the Mosquitto broker process running in the Mosquitto container via the 1883:1883 port-forwarding mechanism. It will sit there waiting for messages published to the "/test" topic to be relayed by the broker and will display whatever it receives.
Re-send the test message:
$ mosquitto_pub -h 127.0.0.1 -p 1883 -t "/test" -m "hello world on $(date)"
2021-02-27T12:24:21+1100 /test hello world on Sat 27 Feb 2021 12:24:21 PM AEDTThe second line is coming from the mosquitto_sub process running in the background.
$ docker exec nodered mosquitto_pub -h mosquitto -p 1883 -t "/test" -m "hello world on $(date)"
2021-02-27T12:31:00+1100 /test hello world on Sat 27 Feb 2021 12:31:00 PM AEDTNote the subtle difference between -h 127.0.0.1 of the earlier command and -h mosquitto on this command. Outside container-space, "127.0.0.1" means "this Raspberry Pi". Inside the Node-Red container, "127.0.0.1" means "this container". Mosquitto is running in a different container so it can't be reached using "127.0.0.1". You use the destination container name instead. Docker provides the mapping between container names and containers.
Key point:
- If you are following an example you found on the Internet and it specifies "127.0.0.1", you must change to "mosquitto" so that it will work in IOTstack. This is explained in Node-RED flow subscribes to broker and Recipe: from MQTT to InfluxDB via Node-Red but experience shows it is a subtle point and quite easy to miss.
Kill the background subscriber process by running the following command and pressing RETURN a second time:
$ kill %1
$
[1]+ Terminated mosquitto_sub -v -h 127.0.0.1 -p 1883 -t "/test" -F "%I %t %p"Set up a background listener:
$ docker exec nodered mosquitto_sub -v -h mosquitto -p 1883 -t "/test" -F "%I %t %p" &
[1] xxxxRe-send the test message:
$ mosquitto_pub -h 127.0.0.1 -p 1883 -t "/test" -m "hello world on $(date)"
2021-02-27T12:40:13+1100 /test hello world on Sat 27 Feb 2021 12:40:13 PM AEDTRe-send the test message:
$ docker exec nodered mosquitto_pub -h mosquitto -p 1883 -t "/test" -m "hello world on $(date)"
2021-02-27T12:47:56+1100 /test hello world on Sat 27 Feb 2021 12:47:55 PM AEDTKill the background process:
$ kill %1
$
[1]+ Terminated docker exec nodered mosquitto_sub -v -h mosquitto -p 1883 -t "/test" -F "%I %t %p"With respect to inside vs outside container-space, the tests show:
| Tests | Publisher | Subscriber | Message flow |
|---|---|---|---|
| 1 + 2 | outside | outside | client to client |
| 1 + 3 | inside | outside | Node-Red to client |
| 4 + 5 | outside | inside | client to Node-Red |
| 4 + 6 | inside | inside | Node-Red to Node-Red |
In a typical IoT scenario, a client device (ESP32 or ESP8266) will be transmitting telemetry via MQTT which it expects Node-Red to process. This is the "client to Node-Red" flow of which "weather sensor logs data" is the classic example.
Node-Red may utilise the "Node-Red to client" flow (probably in conjunction with the retain flag) to tell a client to take an action like "turn the light off or on".
The "client to client" flow could also be used without involving Node-Red to send a message from, say, an iDevice to "turn the light off or on". It is also useful for general debugging, as in this script:
#!/usr/bin/env bash
# set up defaults for optional parameters
TOPIC=${1:-"#"}
BROKER=${2:-"127.0.0.1"}
if [ "$1" = "-h" -o "$1" = "--help" -o "$1" = "-?" ] ; then
cat <<-EOM
Usage: mossieMonitor {«topic» {«broker»}}
«topic» defaults to "#" which is a wildcard meaning "all topics".
Examples:
mossieMonitor "/home/#"
mossieMonitor "/home/+/battery"
mossieMonitor "/home/garageDoor/battery"
«broker» is the IP address, host name, domain name or multicast
domain name of the host where your Mosquitto broker is running.
The default is 127.0.0.1 but it is best to change this script
before adding it to your ~/.local/bin so it defaults to your
broker. If you do that, the only time you will need to pass
the second argument is in special situations. Examples:
Running a test broker on a different host:
mossieMonitor "#" otherhost.local
Running this script from inside a Node-RED container where both
Node-RED and Mosquitto are running in non-host-mode on the same
Docker instance (the usual situation with IOTstack):
mossieMonitor "#" mosquitto
EOM
exit 0
fi
# The command below subscribes to the requested topic(s) and emits each
# MQTT message as it arrives along with a timestamp. It assumes Mosquitto
# is reachable on port 1883. You can use the "-p «port»" option to select
# a different port. Similarly, if your Mosquitto instance requires user
# credentials, you can pass "-u «username»" and "-P «password»".
mosquitto_sub -v -h "$BROKER" -t "$TOPIC" -F "%I %t %p"That can be launched in either the foreground or background as needed. I call this script "mossieMonitor" and it is in my search path on all of my computers.
If you replace the "127.0.0.1" default with the domain name or IP address of your Raspberry Pi running IOTstack then just typing:
$ mossieMonitorwill immediately start listening and relaying all of your MQTT traffic as it arrives at your IOTstack device.
The tests described above prove that MQTT traffic can get into and out of container-space, and can move between containers. But this is all on the same machine.
The next step is to install the Mosquitto clients on another computer and repeat the relevant tests there. What you will actually be testing is your local communications infrastructure.
If that works then there is no reason why an ESP32 or ESP8266 client will not also be able to send and receive messages via MQTT.
Drag an "mqtt-in" node plus a "debug" node onto the canvas and wire them together.
Configure the "mqtt-in" to listen to the "/test" topic. Make sure that the "Server" popup leads to a server configuration of "mosquitto" on port "1883".
Keep in mind that MQTT Server configurations (the entries you see in the "Server" popup menu) are global to all your flows. Don't just change an MQTT Server configuration that is working because that will affect the flows that already depend on it. Don't assume you can copy an "mqtt-in" node from one flow, paste it into another flow, and then tailor its MQTT Server configuration. It doesn't work that way. Laziness is your enemy in Node-Red. When in doubt, choose "Add new mqtt broker…" from the popup menu and start from scratch.
Deploy the flow, make sure you can see the "Debug" screen, and send the test message as above. You should see the "hello world" payload in the debug window.
I have tried several MQTT libraries for the ESP8266 and ESP32. The one I have found to be the simplest and most reliable is arduino-mqtt by Joel Gaehwiler. It shows up in the Arduino IDE "Manage Libraries" as just "MQTT".