Skip to content

Instantly share code, notes, and snippets.

@woshizilong
Forked from mattkasun/mq_troubleshooting.md
Created June 2, 2022 12:29
Show Gist options
  • Save woshizilong/401db307b07aa0a6653f2f1792864e80 to your computer and use it in GitHub Desktop.
Save woshizilong/401db307b07aa0a6653f2f1792864e80 to your computer and use it in GitHub Desktop.
Netmaker v0.13.0 MQ Troubleshooting
Basic Troubleshooting/Cert Regeneration
1. Check docker-compose.yml & mosquitto.conf
- mq
- image: eclipse-mosquitto:2.0.14-openssl
- ports:
- 127.0.0.1:1883:1883
- 8883:8883
- volumes:
- /root/mosquitto.conf:/mosquitto/config/mosquitto.conf
- /root/certs/:/mosquitto/certs/
- mosquitto_data:/mosquitto/data
- mosquitto_logs:/mosquitto/log
- netmaker
- volumes
- dnsconfig:/root/config/dnsconfig
- sqldata:/root/data
- /root/certs/:/etc/netmaker/
- mosquitto.conf
per_listener_settings true
listener 8883
allow_anonymous false
require_certificate true
use_identity_as_username true
cafile /mosquitto/certs/root.pem
certfile /mosquitto/certs/server.pem
keyfile /mosquitto/certs/server.key
listener 1883
allow_anonymous true
2. make sure the broker is reachable: 8883 must be open on the container and broker.domain must point to the host.
- nslookup broker.NETMAKER_BASE_DOMAIN must resolve to the netmaker server
2.a port 8883 must be open on server and reachable from internet
3. make sure certs are generated properly. run "docker logs mq" and check for following startup messages
1651234045: mosquitto version 2.0.14 starting
1651234045: Config loaded from /mosquitto/config/mosquitto.conf.
1651234045: Opening ipv4 listen socket on port 8883.
1651234045: Opening ipv6 listen socket on port 8883.
1651234045: Opening ipv4 listen socket on port 1883.
1651234045: Opening ipv6 listen socket on port 1883.
1651234045: mosquitto version 2.0.14 running
4. If there is a certificate issue, eg.
1651234143: mosquitto version 2.0.14 starting
1651234143: Config loaded from /mosquitto/config/mosquitto.conf.
1651234143: Opening ipv4 listen socket on port 8883.
1651234143: Opening ipv6 listen socket on port 8883.
1651234143: Error: Unable to load server certificate "/mosquitto/certs/server.pem". Check certfile.
OpenSSL Error[0]: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
4.a first check if certs are in /root/certs. If so, delete them
4.b Restart netmaker: "docker restart netmaker"
4.c Restart mq: "docker restart mq"
4.d check mq logs again, make sure it has started appropriately
4.e run "netclient pull -n <network>" on all clients
Detailed troubleshooting for valid certs
1. Using Openssl
on server
openssl verify -verbose -CAfile /root/certs/root.pem /root/certs/server.pem
on client
openssl verify -verbose -CAfile /etc/netclient/<broker.domain>/root.pem /etc/netclient/<broker.domain>/client.pem
2. Using mosquitto_pub
on server -
mosquitto_pub -d -t test -m "hello world" -h broker.<domainname> -p 8883 --cafile /root/certs/root.pem --cert /root/certs/server.pem --key /root/certs/server.key
on client
mosquitto_pub -d -t test -m "hello world" -h broker.domainname -p 8883 --cafile /etc/netclient/broker.<domainname>/root.pem --cert /etc/netclient/broker.<domainname>/client.pem --key /etc/netclient/client.key
Good Result
Client mosq-KOrMCTPqn1rejoTFSl sending CONNECT
Client mosq-KOrMCTPqn1rejoTFSl received CONNACK (0)
Client mosq-KOrMCTPqn1rejoTFSl sending PUBLISH (d0, q0, r0, m1, 'test', ... (11 bytes))
Client mosq-KOrMCTPqn1rejoTFSl sending DISCONNECT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment