EdgeX Foundry: Sample implementation for Auto-Discovery for MQTT Device Service

edgex-geneva-auto-discovery-mqtt-sample.md

EdgeX Foundry: Sample implementation for Auto-Discovery for MQTT Device Service

• EdgeX Foundry Geneva release
• Sample Auto-Discovery implementation based on official MQTT Device Service.

Repository and Docker Images

• GitHub: kurokobo/device-mqtt-go
• Docker Hub: kurokobo/docker-device-mqtt-go

Interaction between MQTT Device Service and MQTT Devices

Discovery Process

As of this implementation, Auto-Discovery works as follows:

1. MQTT Device Service publishes a discovery message to DiscoveryTopic with a unique UUID for this message.
2. MQTT Devices respond to DiscoveryResponseTopic with the body includes its name, description, and the UUID of the request message.
3. MQTT Device Service and its Provision Watcher add discovered devices to Core Service.

Once the discovery message has been published, the MQTT Device Service waits five seconds to collect responses. The responses are not processed immediately when it has been received but are processed in a batch after five seconds.

Commanding Process

Once a device is discovered, the Device Service automatically sets a unique command topic name that includes the name of the discovered device so that it can execute commands to that device.

The naming convention for this topic name is a combination of the value of DefaultCommandTopicRoot in configuration.toml and the name of the device that is included in the response from the device.

Quick Instruction

Prepare Requires Files

- demo
  |- docker-compose.yml
  |- device-service
  |  |- configuration.toml
  |  |- mqtt.test.device.profile.yml
  |- simulator01
  |  |- mock-device.js
  |- simulator02
     |- mock-device.js

docker-compose.yml

This file is based on the official docker-compose-geneva-mongo-no-secty.yml but it has some changes:

• Add original device-mqtt service using my customized MQTT Deivce Service image.
• Remove 127.0.0.1: to allow access from the external host for testing purposes.
• Remove device-rest service and device-virtual service.
• Add ui service for testing purposes.

# /*******************************************************************************
#  * Copyright 2020 Redis Labs Inc.
#  * Copyright 2020 Intel Corporation.
#  *
#  * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
#  * in compliance with the License. You may obtain a copy of the License at
#  *
#  * http://www.apache.org/licenses/LICENSE-2.0
#  *
#  * Unless required by applicable law or agreed to in writing, software distributed under the License
#  * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
#  * or implied. See the License for the specific language governing permissions and limitations under
#  * the License.
#  *
#  * @author: Jim White, Dell
#  * @author: Andre Srinivasan, Redis Labs
#  * @author: Leonard Goodell, Intel
#  * EdgeX Foundry, Geneva, version 1.2.0
#  * added: May 14, 2020
#  *******************************************************************************/

# NOTE:  this Docker Compose file does not contain the security services - namely the API Gateway
# and Secret Store

version: '3.4'

# Note: Mongo has been deprecated in the Geneva (1.2.0) release.
#       Redis is the default database for Geneva (1.2.0).
#       Mongo will not be supported in future releases.

# all common shared environment variables defined here:
x-common-env-variables: &common-variables
  EDGEX_SECURITY_SECRET_STORE: "false"
  Registry_Host: edgex-core-consul
  Clients_CoreData_Host: edgex-core-data
  Clients_Data_Host: edgex-core-data # For device Services
  Clients_Notifications_Host: edgex-support-notifications
  Clients_Metadata_Host: edgex-core-metadata
  Clients_Command_Host: edgex-core-command
  Clients_Scheduler_Host: edgex-support-scheduler
  Clients_RulesEngine_Host: edgex-kuiper
  Clients_VirtualDevice_Host: edgex-device-virtual
  Databases_Primary_Type: mongodb
  Databases_Primary_Host: edgex-mongo
  Databases_Primary_Port: 27017
  # Required in case old configuration from previous release used.
  # Change to "true" if re-enabling logging service for remote logging
  Logging_EnableRemote: "false"
  #  Clients_Logging_Host: edgex-support-logging # un-comment if re-enabling logging service for remote logging

volumes:
  db-data:
  log-data:
  consul-config:
  consul-data:

services:
  consul:
    image: edgexfoundry/docker-edgex-consul:1.2.0
    ports:
      - "8400:8400"
      - "8500:8500"
    container_name: edgex-core-consul
    hostname: edgex-core-consul
    networks:
      - edgex-network
    volumes:
      - consul-config:/consul/config:z
      - consul-data:/consul/data:z
    environment: 
      - EDGEX_DB=mongo
      - EDGEX_SECURE=false

  mongo:
    image: edgexfoundry/docker-edgex-mongo:1.2.0
    ports:
      - "27017:27017"
    container_name: edgex-mongo
    hostname: edgex-mongo
    networks:
      - edgex-network
    environment:
      <<: *common-variables
    volumes:
      - db-data:/data/db:z

# The logging service has been deprecated in Geneva release and will be removed in the Hanoi release.
# All services are configure to send logging to STDOUT, i.e. not remote which requires this logging service
# If you still must use remote logging, un-comment the block below, all the related depends that have been commented out
# and the related global override that are commented out at the top.
#
#  logging:
#    image: edgexfoundry/docker-support-logging-go:1.2.1
#    ports:
#      - "48061:48061"
#    container_name: edgex-support-logging
#    hostname: edgex-support-logging
#    networks:
#      - edgex-network
#    environment:
#      <<: *common-variables
#      Service_Host: edgex-support-logging
#      Writable_Persistence: file
#      Databases_Primary_Type: file
#      Logging_EnableRemote: "false"
#    depends_on:
#      - consul

  system:
    image: edgexfoundry/docker-sys-mgmt-agent-go:1.2.1
    ports:
      - "48090:48090"
    container_name: edgex-sys-mgmt-agent
    hostname: edgex-sys-mgmt-agent
    networks:
      - edgex-network
    environment:
      <<: *common-variables
      Service_Host: edgex-sys-mgmt-agent
      ExecutorPath: /sys-mgmt-executor
      MetricsMechanism: executor
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:z
    depends_on:
      - consul
#      - logging  # uncomment if re-enabled remote logging
      - scheduler
      - notifications
      - metadata
      - data
      - command

  notifications:
    image: edgexfoundry/docker-support-notifications-go:1.2.1
    ports:
      - "48060:48060"
    container_name: edgex-support-notifications
    hostname: edgex-support-notifications
    networks:
      - edgex-network
    environment:
      <<: *common-variables
      Service_Host: edgex-support-notifications
    depends_on:
      - consul
#      - logging  # uncomment if re-enabled remote logging
      - mongo

  metadata:
    image: edgexfoundry/docker-core-metadata-go:1.2.1
    ports:
      - "48081:48081"
    container_name: edgex-core-metadata
    hostname: edgex-core-metadata
    networks:
      - edgex-network
    environment:
      <<: *common-variables
      Service_Host: edgex-core-metadata
      Service_Timeout: "20000"
      Notifications_Sender: edgex-core-metadata
    depends_on:
      - consul
#      - logging  # uncomment if re-enabled remote logging
      - mongo
      - notifications

  data:
    image: edgexfoundry/docker-core-data-go:1.2.1
    ports:
      - "48080:48080"
      - "5563:5563"
    container_name: edgex-core-data
    hostname: edgex-core-data
    networks:
      - edgex-network
    environment:
      <<: *common-variables
      Service_Host: edgex-core-data
    depends_on:
      - consul
#      - logging  # uncomment if re-enabled remote logging
      - mongo
      - metadata

  command:
    image: edgexfoundry/docker-core-command-go:1.2.1
    ports:
      - "48082:48082"
    container_name: edgex-core-command
    hostname: edgex-core-command
    networks:
      - edgex-network
    environment:
      <<: *common-variables
      Service_Host: edgex-core-command
    depends_on:
      - consul
#      - logging  # uncomment if re-enabled remote logging
      - mongo
      - metadata

  scheduler:
    image: edgexfoundry/docker-support-scheduler-go:1.2.1
    ports:
      - "48085:48085"
    container_name: edgex-support-scheduler
    hostname: edgex-support-scheduler
    networks:
      - edgex-network
    environment:
      <<: *common-variables
      Service_Host: edgex-support-scheduler
      IntervalActions_ScrubPushed_Host: edgex-core-data
      IntervalActions_ScrubAged_Host: edgex-core-data
    depends_on:
      - consul
#      - logging  # uncomment if re-enabled remote logging
      - mongo

  app-service-rules:
    image: edgexfoundry/docker-app-service-configurable:1.2.0
    ports:
      - "48100:48100"
    container_name: edgex-app-service-configurable-rules
    hostname: edgex-app-service-configurable-rules
    networks:
      - edgex-network
    environment:
      <<: *common-variables
      edgex_profile: rules-engine
      Service_Host: edgex-app-service-configurable-rules
      Service_Port: 48100
      MessageBus_SubscribeHost_Host: edgex-core-data
      Binding_PublishTopic: events
    depends_on:
      - consul
#      - logging  # uncomment if re-enabled remote logging
      - data

  rulesengine:
    image: emqx/kuiper:0.4.2-alpine
    ports:
      - "48075:48075"
      - "20498:20498"
    container_name: edgex-kuiper
    hostname: edgex-kuiper
    networks:
      - edgex-network
    environment:
      # KUIPER_DEBUG: "true"
      KUIPER_CONSOLE_LOG: "true"
      KUIPER_REST_PORT: 48075
      EDGEX_SERVER: edgex-app-service-configurable-rules
      EDGEX_SERVICE_SERVER: http://edgex-core-data:48080
      EDGEX_TOPIC: events
      EDGEX_PROTOCOL: tcp
      EDGEX_PORT: 5566
    depends_on:
      - app-service-rules

  # Support RulesEngine has been deprecated in the Geneva (1.2.0) release
  # If still required, simply uncomment the block below and comment out the block above.
  #
  # rulesengine:
  #   image: edgexfoundry/docker-support-rulesengine:1.2.1
  #   ports:
  #     - "48075:48075"
  #   container_name: edgex-support-rulesengine
  #   hostname: edgex-support-rulesengine
  #   networks:
  #     - edgex-network
  #   depends_on:
  #     - app-service-rules

#################################################################
# Device Services
#################################################################

  device-virtual:
    image: edgexfoundry/docker-device-virtual-go:1.2.2
    ports:
    - "49990:49990"
    container_name: edgex-device-virtual
    hostname: edgex-device-virtual
    networks:
      - edgex-network
    environment:
      <<: *common-variables
      Service_Host: edgex-device-virtual
    depends_on:
      - consul
#      - logging  # uncomment if re-enabled remote logging
      - data
      - metadata

  device-rest:
    image: edgexfoundry/docker-device-rest-go:1.1.1
    ports:
      - "49986:49986"
    container_name: edgex-device-rest
    hostname: edgex-device-rest
    networks:
      - edgex-network
    environment:
      <<: *common-variables
      Service_Host: edgex-device-rest
    depends_on:
      - data
      - command
  #      - logging  # uncomment if re-enabled remote logging

#  device-random:
#    image: edgexfoundry/docker-device-random-go:1.2.1
#    ports:
#      - "49988:49988"
#    container_name: edgex-device-random
#    hostname: edgex-device-random
#    networks:
#      - edgex-network
#    environment:
#      <<: *common-variables
#      Service_Host: edgex-device-random
#    depends_on:
#      - data
#      - command
#
  device-mqtt:
    image: kurokobo/docker-device-mqtt-go:1.2.2
    ports:
      - "49982:49982"
    container_name: edgex-device-mqtt
    hostname: edgex-device-mqtt
    networks:
      - edgex-network
    volumes:
      - ./device-service:/res
    environment:
      <<: *common-variables
      Service_Host: edgex-device-mqtt
    depends_on:
      - data
      - command

#  device-modbus:
#    image: edgexfoundry/docker-device-modbus-go:1.2.1
#    ports:
#      - "49991:49991"
#    container_name: edgex-device-modbus
#    hostname: edgex-device-modbus
#    networks:
#      - edgex-network
#    environment:
#      <<: *common-variables
#      Service_Host: edgex-device-modbus
#    depends_on:
#      - data
#      - command
#
#  device-snmp:
#    image: edgexfoundry/docker-device-snmp-go:1.2.1
#    ports:
#      - "49993:49993"
#    container_name: edgex-device-snmp
#    hostname: edgex-device-snmp
#    networks:
#      - edgex-network
#    environment:
#      <<: *common-variables
#      Service_Host: edgex-device-snmp
#    depends_on:
#      - data
#      - command

  ui:
    image: edgexfoundry/docker-edgex-ui-go:1.2.1
    ports:
      - "4000:4000"
    container_name: edgex-ui-go
    hostname: edgex-ui-go
    networks:
      - edgex-network

networks:
  edgex-network:
    driver: "bridge"

device-service/configuration.toml

Note: Replace 192.168.0.220 to suit your MQTT broker.

[Writable]
LogLevel = 'INFO'

[Service]
BootTimeout = 30000
CheckInterval = '10s'
Host = 'localhost'
ServerBindAddr = ''  # blank value defaults to Service.Host value
Port = 49982
Protocol = 'http'
StartupMsg = 'device mqtt started'
Timeout = 5000
ConnectRetries = 10
Labels = []
EnableAsyncReadings = true
AsyncBufferSize = 16

[Registry]
Host = 'localhost'
Port = 8500
Type = 'consul'

[Logging]
EnableRemote = false
File = ''

[Clients]
  [Clients.Data]
  Protocol = 'http'
  Host = 'localhost'
  Port = 48080

  [Clients.Metadata]
  Protocol = 'http'
  Host = 'localhost'
  Port = 48081

  [Clients.Logging]
  Protocol = 'http'
  Host = 'localhost'
  Port = 48061

[Device]
  DataTransform = true
  InitCmd = ''
  InitCmdArgs = ''
  MaxCmdOps = 128
  MaxCmdValueLen = 256
  RemoveCmd = ''
  RemoveCmdArgs = ''
  ProfilesDir = './res'
  UpdateLastConnected = false
  [Device.Discovery]
    Enabled = true
    Interval = '30s'

# Driver configs
[Driver]
IncomingSchema = 'tcp'
IncomingHost = '192.168.0.220'
IncomingPort = '1883'
IncomingUser = 'admin'
IncomingPassword = 'public'
IncomingQos = '0'
IncomingKeepAlive = '3600'
IncomingClientId = 'IncomingDataSubscriber'
IncomingTopic = 'DataTopic'
ResponseSchema = 'tcp'
ResponseHost = '192.168.0.220'
ResponsePort = '1883'
ResponseUser = 'admin'
ResponsePassword = 'public'
ResponseQos = '0'
ResponseKeepAlive = '3600'
ResponseClientId = 'CommandResponseSubscriber'
ResponseTopic = 'ResponseTopic'
DiscoverySchema = 'tcp'
DiscoveryHost = '192.168.0.220'
DiscoveryPort = '1883'
DiscoveryUser = 'admin'
DiscoveryPassword = 'public'
DiscoveryClientId = 'DiscoveryPublisher'
DiscoveryTopic = 'DiscoveryTopic'
DiscoveryResponseSchema = 'tcp'
DiscoveryResponseHost = '192.168.0.220'
DiscoveryResponsePort = '1883'
DiscoveryResponseUser = 'admin'
DiscoveryResponsePassword = 'public'
DiscoveryResponseQos = '0'
DiscoveryResponseKeepAlive = '3600'
DiscoveryResponseClientId = 'DiscoveryResponseSubscriber'
DiscoveryResponseTopic = 'DiscoveryResponseTopic'
DefaultCommandSchema = 'tcp'
DefaultCommandHost = '192.168.0.220'
DefaultCommandPort = '1883'
DefaultCommandUser = 'admin'
DefaultCommandPassword = 'public'
DefaultCommandClientId = 'CommandPublisher'
DefaultCommandTopicRoot = 'CommandTopic'
ConnEstablishingRetry = '10'
ConnRetryWaitTime = '5'

device-service/mqtt.test.device.profile.yml

Just download the example file. This file is completely the same as the official example file.

$ curl https://raw.githubusercontent.com/kurokobo/device-mqtt-go/master/cmd/res/example/mqtt.test.device.profile.yml -o device-service/mqtt.test.device.profile.yml

simulator01/mock-device.js

function getRandomFloat(min, max) {
    return Math.random() * (max - min) + min;
}

const deviceName = "MQ_DEVICE_01";
const description = "Test Device";
let message = "test-message";

// 1. Publish random number every 15 seconds
schedule('*/15 * * * * *', () => {
    let body = {
        "name": deviceName,
        "cmd": "randfloat32",
        "randfloat32": getRandomFloat(25, 29).toFixed(1)
    };
    publish('DataTopic', JSON.stringify(body));
});

// 2. Receive the reading request, then return the response
// 3. Receive the put request, then change the device value
subscribe("CommandTopic/" + deviceName, (topic, val) => {
    var data = val;
    if (data.method == "set") {
        message = data[data.cmd]
    } else {
        switch (data.cmd) {
            case "ping":
                data.ping = "pong";
                break;
            case "message":
                data.message = message;
                break;
            case "randfloat32":
                data.randfloat32 = getRandomFloat(25, 29).toFixed(1);
                break;
            case "randfloat64":
                data.randfloat64 = getRandomFloat(10, 1).toFixed(5);
                break;
        }
    }
    publish("ResponseTopic", JSON.stringify(data));
});

// 4. Receive the discovery request, then return the device information
subscribe("DiscoveryTopic", (topic, val) => {
    var data = val;
    let body = {
        "name": deviceName,
        "description": description,
        "uuid": data.uuid,
    };
    publish("DiscoveryResponseTopic", JSON.stringify(body));
});

simulator02/mock-device.js

Copy simulator01/mock-device.js and replace the hard-coded device name.

$ mkdir simulator02
$ cp simulator01/mock-device.js simulator02
$ sed -i 's/MQ_DEVICE_01/MQ_DEVICE_02/g' simulator02/mock-device.js

Prepare MQTT Broker and Start Subscribe

Note: Replace 192.168.0.220 to suit your MQTT broker.

$ docker run -d --rm --name broker -p 1883:1883 eclipse-mosquitto
$ docker run --init --rm --name=client -it efrecon/mqtt-client sub -h 192.168.0.220 -t "#" -v

Start EdgeX Foundry

$ docker-compose up -d

The conguration.toml is modified to perform discovery every 30 seconds, so discovery messages are displayed on the MQTT broker. But nothing happened because no MQTT devices are available.

...
DiscoveryTopic {"method":"discovery","uuid":"5f89dbd1b8dd7900016a9f09"}
DiscoveryTopic {"method":"discovery","uuid":"5f89dbefb8dd7900016a9f0a"}
DiscoveryTopic {"method":"discovery","uuid":"5f89dc0db8dd7900016a9f0b"}
DiscoveryTopic {"method":"discovery","uuid":"5f89dc2bb8dd7900016a9f0c"}
DiscoveryTopic {"method":"discovery","uuid":"5f89dc49b8dd7900016a9f0d"}
DiscoveryTopic {"method":"discovery","uuid":"5f89dc67b8dd7900016a9f0e"}
...

And no device is registered now in EdgeX Foundry.

$ curl -s http://localhost:48081/api/v1/device | jq
[]

Create Provision Watcher

This can be achieved by POSTing this JSON:

{
  "name": "mqtt-device-watcher",
  "identifiers": {
    "Host": "MQ_DEVICE_.*"
  },
  "blockingidentifiers": {
    "Host": [
      "INVALID_DEVICE_.*"
    ]
  },
  "profile": {
    "name": "Test.Device.MQTT.Profile"
  },
  "service": {
    "name": "edgex-device-mqtt"
  },
  "adminState": "UNLOCKED"
}

to the Metadata API. Refer official document for the details.

$ curl -s -X POST -H "Content-Type: application/json" -d '{"name": "mqtt-device-watcher", "identifiers": {"Host": ".*"}, "blockingidentifiers": {"Host": ["INVALID_DEVICE_.*"]}, "profile": {"name": "Test.Device.MQTT.Profile"}, "service": {"name": "edgex-device-mqtt"}, "adminState": "UNLOCKED"}' http://localhost:48081/api/v1/provisionwatcher
94606436-dfda-4886-96e2-912ff8c36aaf

Start Simulators

Note: Replace 192.168.0.220 to suit your MQTT broker.

$ docker run -d --restart=always --name=mq_device_01 -v "$(pwd)/simulator01:/scripts" dersimn/mqtt-scripts --url mqtt://192.168.0.220 --dir /scripts
$ docker run -d --restart=always --name=mq_device_02 -v "$(pwd)/simulator02:/scripts" dersimn/mqtt-scripts --url mqtt://192.168.0.220 --dir /scripts

These two devices respond to the discovery message.

...
DiscoveryTopic {"method":"discovery","uuid":"5f89e235b8dd790001963c35"}
DiscoveryResponseTopic {"name":"MQ_DEVICE_01","description":"Test Device","uuid":"5f89e235b8dd790001963c35"}
DiscoveryResponseTopic {"name":"MQ_DEVICE_02","description":"Test Device","uuid":"5f89e235b8dd790001963c35"}
...

Now, these devices have been registered automatically and can be controlled by EdgeX Foundry.

$ curl -s http://localhost:48081/api/v1/device | jq
[
  {
    "created": 1602872436082,
    "modified": 1602872436082,
    "origin": 1602872436077,
    "description": "Test Device",
    "id": "9da96475-a5d9-41b5-8e7d-89309dec00a2",
    "name": "MQ_DEVICE_02",
    "adminState": "UNLOCKED",
    "operatingState": "ENABLED",
    "protocols": {
      "mqtt": {
        "ClientId": "CommandPublisher",
        "Host": "192.168.0.220",
        "Password": "public",
        "Port": "1883",
        "Schema": "tcp",
        "Topic": "CommandTopic/MQ_DEVICE_02",
        "User": "admin"
      }
    },
    ...
  },
  {
    "created": 1602872436100,
    "modified": 1602872436100,
    "origin": 1602872436086,
    "description": "Test Device",
    "id": "a455c3a7-ac26-4191-91b8-291f877399c6",
    "name": "MQ_DEVICE_01",
    "adminState": "UNLOCKED",
    "operatingState": "ENABLED",
    "protocols": {
      "mqtt": {
        "ClientId": "CommandPublisher",
        "Host": "192.168.0.220",
        "Password": "public",
        "Port": "1883",
        "Schema": "tcp",
        "Topic": "CommandTopic/MQ_DEVICE_01",
        "User": "admin"
      }
    },
    ...
  }
]