mattkasun/mq_troubleshooting.md

## mq_troubleshooting.md

      
    Raw
  

              mq_troubleshooting.md
            
          
    not applicable post v0.16.1

Basic Troubleshooting/Cert Regeneration

Check docker-compose.yml

if MQ is proxied

- image: eclipse-mosquitto:2.0.14-opensssl
- expose
  - 8883"

if MQ is not proxied

- image: eclipse-mosquitto:2.0.14-openssl
- ports:
  - 127.0.0.1:1883:1883
  - 8883:8883

volumes

MQ

  - /root/mosquitto.conf:/mosquitto/config/mosquitto.conf
  - mosquitto_data:/mosquitto/data
  - mosquitto_logs:/mosquitto/log
  - shared_certs:/mosquitto/certs

Netmaker

  - dnsconfig:/root/config/dnsconfig
  - sqldata:/root/data
  - shared_certs:/etc/netmaker

mosquitto.conf

per_listener_settings true

listener 8883
allow_anonymous false
require_certificate true
use_identity_as_username true
cafile /mosquitto/certs/root.pem
certfile /mosquitto/certs/server.pem
keyfile /mosquitto/certs/server.key

listener 1883 
allow_anonymous true

make sure the broker is reachable:

nslookup broker.NETMAKER_BASE_DOMAIN must resolve to the netmaker server

if mq is not proxied

port 8883 must be open on the container 

make sure certs are generated properly.

run "docker logs mq"  and check for following startup messages

1651234045: mosquitto version 2.0.14 starting
1651234045: Config loaded from /mosquitto/config/mosquitto.conf.
1651234045: Opening ipv4 listen socket on port 8883.
1651234045: Opening ipv6 listen socket on port 8883.
1651234045: Opening ipv4 listen socket on port 1883.
1651234045: Opening ipv6 listen socket on port 1883.
1651234045: mosquitto version 2.0.14 running

If there is a certificate issue, eg.
1651234143: mosquitto version 2.0.14 starting
1651234143: Config loaded from /mosquitto/config/mosquitto.conf.
1651234143: Opening ipv4 listen socket on port 8883.
1651234143: Opening ipv6 listen socket on port 8883.
1651234143: Error: Unable to load server certificate "/mosquitto/certs/server.pem". Check certfile.


OpenSSL Error[0]: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

Delete Certs

(may have to install sqlite3 on server)
sqlite3 /var/lib/docker/volumes/root_sqldata/_data/netmaker.db 'delete from certs;'

Restart netmaker:
docker restart netmaker

Restart mq:
docker restart mq

check mq logs again, make sure it has started appropriately
run
netclient pull -n <network> 

on all clients
Detailed troubleshooting for valid certs

Using Openssl

on server
openssl verify -verbose -CAfile /root/certs/root.pem /root/certs/server.pem

on client
openssl verify -verbose -CAfile /etc/netclient/<broker.domain>/root.pem /etc/netclient/<broker.domain>/client.pem

Using mosquitto_pub

on server -
mosquitto_pub -d -t test -m "hello world" -h broker.<domainname> -p 8883 --cafile /root/certs/root.pem --cert /root/certs/server.pem --key /root/certs/server.key

on client
mosquitto_pub -d -t test -m "hello world" -h broker.domainname -p 8883 --cafile /etc/netclient/broker.<domainname>/root.pem --cert /etc/netclient/broker.<domainname>/client.pem --key /etc/netclient/client.key

Good Result
Client mosq-KOrMCTPqn1rejoTFSl sending CONNECT
Client mosq-KOrMCTPqn1rejoTFSl received CONNACK (0)
Client mosq-KOrMCTPqn1rejoTFSl sending PUBLISH (d0, q0, r0, m1, 'test', ... (11 bytes))
Client mosq-KOrMCTPqn1rejoTFSl sending DISCONNECT