This project involves implementing an AMQP 1.0 protocol adapter for Eclipse Hono so that clients (devices or gateway components) supporting the AMQP 1.0 protocol can publish telemetry messages and events to Eclipse Hono™'s Telemetry and Event endpoints. Publishing telemetry requires acccess to the telemetry endpoint and publishing events requires access to the event endpoint. Each client device supports a specific communication protocol (e.g AMQP, MQTT, CoAP, HTTP, etc). Allowing devices direct access to any of the API endpoints will lead to a tight-coupling between the endpoint and a device communication protocol. This tight-coupling will result to having arbitrary telemetry and event endpoints, each implemented to handle a specific device protocol.
Hono's solution to this tight-coupling problem is the introduction of a protocol adapter. A protocol adapter is a Hono specific term serving the purpose of abstracting away a device's communication protocol. This approach decouples Hono™'s Telemetry and Event API endpoints from a specific device communication protocol, allowing a uniform access to such endpoints by arbitrary devices to send telemetry (using the telemetry endpoint) and events (using the events endpoint).
Let's drive home this point with an example. If an IoT device supporting protocol ABC wants to publish data to Hono, the device needs to connect to the ABC protocol adapter since this protocol adapter understands the ABC protocol that the device uses for communication. After connecting to the adapter, the device can proceed to publish data to the adapter. Depending on the type of message (telemetry or event) published by the device, the adapter will access the required endpoint to forward the data downstream. Like this, the device does not directly access the API endpoints but does so via the ABC adapter. Internally, the protocol adapter component communicates with the other Hono components (i.e device registry service, credentials service, tenant service and hono messaging service) by means of AMQP 1.0 message exchanges when publishing the message downstream.
IoT solutions (consumer applications) connects to the Hono messaging network using AMQP 1.0 and can consume data from arbitrary devices supporting different communication protocols. The advantage for backend consumers is that they only connect to Hono using one protocol, i.e AMQP 1.0, and can consume data from devices supporting arbitrary protocols.
The figure below depicts a simulated device publishing data to the HTTP protocol adapter. The HTTP adapter then forwards the data to the Hono Messaging network where the consumer application is connected to consumed the published data.
The Eclipse Hono project is extensively documented on its website and the entire project source code is hosted on Github. This documentation significantly helps to lower the entry barrier for new contributors like me to quickly start experimenting and working on the project.
The Hono project is a Maven-based project comprising of different modules. All Hono protocol adapters are submodules of the adapters parent module. The AMQP adapter code base is contained in the amqp-vertx submodule. The service-base module provides a base protocol adapter implementation called AbstractProtocolAdapterBase which makes it easy to implement new protocol adapters. The VertxBasedAmqpProtocolAdapter class extends this base implementation to allow devices supporting the AMQP protocol to connect and publish telemetry and events to Hono. This class is the starting point to understanding how the AMQP adapter is implemented.
The work done in this GSoC project mainly centered around the following modules:
- The
amqp-vertxmodule: Contains all the source and test files for implementing respectively testing the AMQP protocol adapter. - The
hono-testsmodule: Provides integration tests for Hono. Theorg.eclipse.hono.tests.amqppackage contains the AMQP adapter integration tests. The integration test is an end-to-end test in which a device authenticates to the protocol adapter to publish telemetry and events to downstream applications. - The
deploymodule: The Kubernetes and Docker Swarm example deployments of the AMQP adapter is implemented in this module. - The
climodule: This modules contains a simple command-line client for interacting with the AMQP adapter using the terminal. - The
demo-certsmodule: Contains the public certificate and private key used to bootstrap a secure port for the AMQP adapter.
Below is a description of some of the core classes in the amqp-vertx module and the org.eclipse.hono.tests.amqp integration test package:
VertxBasedAmqpProtocolAdapter.java: A Vert.x based protocol adapter for publishing messages to Hono's Telemetry and Event APIs using the AMQP 1.0 protocol.AmqpAdapterSaslAuthenticatorFactory.java: A factory for an authenticator that knows how to manage the SASL authentication process for client devices. If you want to understand how the AMQP adapter authenticates clients using either SASL PLAIN (username/password based authentication) or SASL EXTERNAL (certificate-based authentication), then this is the class to look at.Application.java: A Spring-boot application for configuring and starting the AMQP protocol adapter instance.Config.java: Spring-based configuration for the AMQP protocol adapter.VertxBasedAmqpProtocolAdapterTest.java: Contains Unit Tests for verifying the behavior of the AMQP adapter.AmqpAdapterTestBase.java: The base class for the AMQP adapter integration tests.EventAmqpIT.java: An Event based integration test for the AMQP adapter.TelemetryAmqpIT.java: Telemetry based integration test for the AMQP adapter.
The Eclipse Hono Website contains two types of documentation for the AMQP adapter. The User Guide documentation explains the AMQP adapter performs device authentication and the publishing of telemetry and events. It also provides concrete examples how to interact with the adapter from the command-line. The Admin Guide documentation explains how the AMQP adapter can be administered (configured) by describing the different configuration variables and corresponding command line options. Readers are encourage to try out and play with the AMQP adapter by following the instructions described in the AMQP user guide.
Eclipse Hono is built using the latest technologies in the Microservices and cloud computing domains. Before this project, I was new to microservices, reactive programming and the AMQP 1.0 protocol. These were technologies completely foreign to me. It is through this GSoC project that I happen to learn, understand and, most importantly, see how these technologies are implemented in practice. Therefore, this project presented interesting challenges and learnings during the summer coding period. The list below describes the technologies I have directly worked with in the Hono project:
- Eclipse Vert.x: Hono is built from the ground up using Vert.x, a toolkit for writing reactive applications on the JVM.
- Vert.x Proton: A library for implementing Vert.x based AMQP 1.0 clients and servers. The library provides a thin wrapper around the Apache Qpid protocol engine and was used extensively in the implementation of the AMQP adapter.
- Docker Swarm and Kubernetes: The Hono project utilizes container orchestration technologies to simplify the deployment and management of a Hono instance. I understood Docker and Kubernetes better thanks to this project.
- Java 8: Java 8 was a major feature release of the Java programming language which is widely used in the Hono project. I had no previous knowledge about the new Java features and realized that understanding the Java 8 syntax, semantics and functional styles will be useful to realizing this project. As a result, I bought the Java 8 In Action book and uses it as a reference while I was implementing the AMQP adapter. I now know more about Java 8 than I did three months ago.
The reason why the learning path has not been very steep is that the Hono project is extensively documented. The source code has proper Javadoc comments written using simple and easy to understand english. The website also provides ample resources to learn about Hono and to easily get started. As a new contributor, this aided my learning process significantly. If anybody is interested to start contributing to the Hono project, the following tips will help you get started:
- Tip 1: Read the relevant pages on the Hono website to understand the concepts/APIs behind what you want to contribute. Example, if you want to implement the Command and Control API for the AMQP adapter, reading its user guide docs and its API docs will help one get started easily.
- Tip 2: Understand how the code is structured and implemented. Example, the Command and Control API has been implemented for the HTTP and MQTT adapters, the the most natural thing to do is to understand how it has been implemented for these adapters. This will motivate your own implementation.
- Tip 3: Play with the integration tests code base in the
hono-testsmodule. If you are new to Docker and containers in general, you might want to brush up on that first! - Tip 4: Start small. Implement a very small feature and test it manually on the command-line. This was the advice that Kai Hudalla gave me at the beginning and it proved very help.
- Tip 5: Make some noise on the mailing list or on Github. If you already created an issue for the thing you want to contribute, use the github
@mentionfeature to explicitly get a person's attention. Hono is quiet an active project and the response times for questions is generally good.
My contribution has been in the form of adding enhancements, bug fixes and providing additional patches not directly related to my GSoC topic. I summarize those in the next sections.
The following patches have been contributed to the Hono project in the form of enhancements.
- Allow unauthenticated devices to publish telemetry messages and events to Hono. Implemented in PR #640
- Add Unit tests to test the implementation of unauthenticated devices publishing messages to Hono. Implemented in PR [#660](Pull Request: #660
- Add support for username/password based authentication by adding support for the SASL PLAIN mechanism. Implemented in PR #674
- Add support for certificate based authentication by adding support for the SASL EXTERNAL mechnism. Implemented in PR #744
- Refactor AMQP adapter code base (preparation for the Integration tests). Implemented in PR #714
- Add end-to-end integration tests so that messages published by authenticated devices can be consumed by downstream applications. Implemented in PR #663, Revision
ac156a4 - Adapt Hono Messaging ITs to allow code-sharing with AMQP adapter. Implemented in PR #663, Revision
784ef45 - Add Integration tests for sending telemetry messages. Implemented in PR #729
- Add anonymous relay support for publishing messages. Implemented in PR #727
- Rewrite AMQP adapter integration test to directly use the
vertx-protonlibrary. Implemented in PR #740 - Command-line client for interacting with the AMQP adapter. Implemented in Revision 991a177
- Document the implementation of the AMQP adapter on the Hono Website. Implemented in PR #728
- Include the AMQP adapter in the Hono example deployment. Implemented in PR #758
- Include documentation for SASL EXTERNAL auth on Hono Website. Implemented in PR #774
The following patches have been contributed to the Hono project in the form of bug fixes:
- Bug Fix: Use adapter type name when checking if it is disabled for the tenant. Fixed in PR #666
- Bug Fix: Use device-id of authenticated device when validating message address. Fixed in PR #736
- Bug Fix: Adapter should respond to close frames emitted by clients. Fixed in PR #739
- Bug Fix: AMQP command-line client should check for empty username and password. Fixed in PR #759
- Bug Fix: AMQP CLI should include SASL PLAIN in its connection options. Fixed in PR #782
I contributed the following patches either during the community bonding period or during the coding period. However, these changes are not related to my GSoC project topic which was to implement an AMQP protocol adapter.
- Add support for publishing data with QoS 1 via HTTP adapter. Implemented in PR #581
- Provide support for HonoClient#disconnect(). Implemented in PR #595
- Update the release notes and the HTTP adapter user guide for uploading telemetry messages using QoS 1. Implemented in PR #606
- Trivial fix of a possible NPE in HonoClientImpl. Fixed in PR #592
The AMQP 1.0 protocol adapter currently implements the Telemetry and Event APIs to allow clients to publish telemetry respectively events to downstream applications. Implementing the Telemetry and Event APIs was the intended goal of the GSoC project. However, there is still work that is left to be done and we summarize those in the bulleted list below:
- Command and Control (C&C): The Command & Control API allows downstream applications to send commands to devices connected to a protocol adapter. It also allows devices to respond to specific commands sent by applications. Currently, the adapter does not provide support for command and control. A future work on the AMQP adapter is to provide an implementation of the C&C API. This is the next project I intend to work on immediately after GSoC 2018.
- Metrics Reporting: Add metrics reporting support for the AMQP adapter.
- OpenTracing Support: Add distributed tracing support for the AMQP adapter. Distributed tracing provides a form of flash light for microservices and is heavily used in the Hono project. Should the AMQP adapter misbehaves due to bad network hardware, logical errors, etc, tracing will bring those problems to light, consequently making it easy to diagnose and fix problems.