IoT Hub is a managed service that acts as a central message hub for bi-directional communication between an IoT application and the devices it manages. IoT Hub supports communications both from the device to the cloud and from the cloud to the device. IoT Hub supports multiple messaging patterns and may be integrated with other Azure services to build complete, end-to-end solutions.
One of the Azure services that IoT Hub is frequently integrated with is Azure Event Hubs. This document describes a proposed approach to helping developers using IoT Hub leverage the Event Hubs client library.
-
The names used in this document are intended for illustration only. Some names are not ideal and will need to be refined during discussions.
-
Some details not related to the high-level concept are not illustrated; the scope of this is limited to the high level shape and paradigms for the feature area.
-
Fake methods are used to illustrate "something needs to happen, but the details are unimportant." As a general rule, if an operation is not directly related to one of the IoT Hub or Event Hubs types, it can likely be assumed that it is for illustration only. These methods will most often use ellipses for the parameter list, in order to help differentiate them.
The IoT Hub service is intended to serve as a focal point for bi-directional communication with IoT devices and does not natively provide integration points for applications that wish to consume the information published by devices or the IoT Hub service itself. To achieve that scenario, IoT Hub makes use of integrations with other Azure services, frequently using Event Hubs as a downstream target for publishing.
A common pattern is for users of IoT Hub to consume information from an Event Hub instance that is provisioned and owned by the IoT Hub service. The IoT Hub documentation refers to this Event Hub instance as the "built-in Event Hub-compatible endpoint that IoT Hub exposes", making it an important part of the IoT Hub story.
One of the features supported by IoT Hub is a fail-over scenario, which may be initiated by customers, and which is likely to result in IoT Hub changing the Event Hub instance to which it publishes events. Because this Event Hub instance association is owned by IoT Hub, developers cannot assume a static connection string to access it, but instead must query the IoT Hub service for the active instance.
-
A new IoT Hub client library will be created, concentrating on integrations. This would be part of the existing family of IoT Hub SDKs, which are currently divided into packages focused on specific IoT Hub scenarios. (example: Devices, Services, Provisioning)
-
The IoT Hub Integrations client library would provide a minimalist experience focused on interactions with the Iot Hub service needed to integrate with other Azure services. The API surface will offer customers a "bridge" experience similar to retrieving information from the Azure portal, Azure CLI, or Azure PowerShell that would be used to interact with integrated services.
-
The IoT Hub Integrations client endeavors to be self-contained and carefully manages external dependencies; the initial version should depend only on the
Microsoft.Azure.Amqplibrary and should be able to coexist in with the Event Hubs client library in the same build. -
To ensure compatibility, a cross-service sample which uses the IoT Hub Integrations and Event Hubs client libraries together will be added to the Azure SDK for .NET repository and execute as part of nightly test runs. This will surface any interoperability issues or dependencies friction so that interested parties can collaborate on a fix.
-
Allow developers to query the IoT Hub service and receive a connection string to the active Event Hub instance to which IoT Hub is publishing events.
-
Maintain the focus on IoT Hub; as a "built-in Event Hub-compatible endpoint" to the IoT Hub service, the customer experience should maintain a strong association with IoT Hub and provide discovery for the active Event Hubs connection information through an IoT Hub interface.
-
Provide an API surface for the IoT Hub connection string translation that is simple and intuitive for developers to use, and which has only the responsibility for discovery. The result of the discovery request should be a fully-formed connection string which can be used with the Event Hubs client library.
-
Encapsulate the IoT Hub connection string translation into a small and independent unit of delivery with minimal external dependencies.
-
Ensure that any external dependencies for the discovery do not conflict or cause friction with the IoT Hub or Event Hubs client libraries.
-
A customer wishing to consume data sent from devices to the cloud visits the documentation page "Read device-to-cloud messages from the built-in endpoint" and scrolls to find the section titled "Read from the built-in endpoint". The section opens with the statement "When you use Event Hubs SDKs or product integrations that are unaware of IoT Hub, you need an Event Hub-compatible endpoint and Event Hub-compatible name." It then goes on to discuss how to obtain the Event Hub-compatible connection information from both the portal and the IoT Hub Integration client library. That information is shared with the development team.
-
A developer on the team investigates the scenario by visiting the documentation page for the IoT Hub SDKs and finds a detailed set of information for the available client libraries including the Device SDKs, Service SDKs, Provisioning SDKs, and Integration SDKs. After reading the description for each, the developer understands that the Integration client library provides the functionality for working with the various IoT Hub integrations, including Event Hubs.
-
Having learned about the SDK options and having information about interacting with the "built-in Event Hub-compatible" endpoint to receive data begins a new project and references the
Microsoft.Azure.Devices.IntegrationsandAzure.Messaging.EventHubspackages from NuGet. -
The developer uses the IoT Hub Integrations client library to query the Event Hubs-compatible connection string from the IoT Hub service and then uses that connection string to create an
EventHubConsumerClientusing the Event Hubs client library. -
The developer now begins using the Event Hubs client library and Event Hubs service directly; no further interactions with IoT Hub are necessary.
var iotHubConnectionString = "<< IOT HUB CONNECTION STRING >>";
var ehConnectionString = await IotHubConnection.RequestEventHubsConnectionStringAsync(iotHubConnectionString);var iotHubConnectionString = "<< IOT HUB CONNECTION STRING >>";
var ehConnectionString = await IotHubConnection.RequestEventHubsConnectionStringAsync(iotHubConnectionString);
await using (var consumer = new EventHubConsumerClient(ehConnectionString))
{
// Begin using the consumer to read events here.
}var iotHubConnectionString = "<< IOT HUB CONNECTION STRING >>";
var ehConnectionString = await IotHubConnection.RequestEventHubsConnectionStringAsync(iotHubConnectionString);
var blobsConnectionString = "<< BLOB STORAGE CONNECTION STRING >>";
var blobContainerName = "iotHubProcessor";
var consumerGroup = EventHubConsumerClient.DefaultConsumerGroupName;
var storageClient = new BlobContainerClient(blobsConnectionString, blobContainerName);
var processor = new EventProcessorClient(storageClient, consumerGroup, ehConnectionString);
processor.ProcessEventAsync += args => Task.CompletedTask;
processor.ProcessErrorAsync += args => Task.CompletedTask;
await Processor.StartProcessingAsync();-
Customers looking at the portal for the built-in endpoints are presented with the concept of an "Event Hub-compatible endpoint" prominently, which states that they should "Use this value as the event hub endpoint when using SDKs or integrations that expect to read from Event Hubs." This sets an expectation that IoT Hub is providing connection information for working with Event Hubs.
-
From a customer's perspective, Event Hubs is a separate service with which IoT Hub integrates. The IoT Hub documentation also highlights Event Grid, Logic Apps, Machine Learning, and Stream Analytics. It is expected that these other integrations would make use of their own connection credentials and not accept an IoT Hub connection string. It is inconsistent and confusing that Event Hubs would be treated differently.
-
Having Event Hubs accept connection credentials from another service is potentially confusing for customers in that it establishes a precedent for service integrations. As a customer, if I'm consuming Event Grid for information about my Azure Storage account, there should be no expectation that Event Grid would accept the Storage connection string. It would be inconsistent and potentially confusing for Event Hubs and IoT Hub to behave differently.
-
Customers referencing the IoT Hub documentation for reading from the built-in endpoint are introduced to the need for Event Hubs-compatible connection information and instructed on how to obtain it. They are also made aware that each of the named integrations and SDKs require the Event Hubs-compatible connection information, other than the Python and Node.js Event Hubs client libraries. Having a subset of integrations behave differently than all others is potentially confusing for a customer and requires them to understand the different behaviors.
-
Fail-over is a process that is controlled through IoT Hub, and may be performed manually by customers. This contributes to the perspective that IoT Hub is targeting a different Event Hubs instance and that identifying which instance is a reasonable question to ask IoT Hub.
-
The Event Hubs client library is intended to enable customers to work with the Event Hubs service and is specifically focused on Event Hubs. It is not intended to facilitate cross-service communication.
-
The Event Hubs clients follow the Azure SDK design guidelines which require that they provide simple constructors for creation. Translating the IOT Hub connection string requires network communication with the IOT Hub service, which is inherently an asynchronous operation. Including network-bound or asynchronous operations in a constructor would violate best practices for .NET, so an alternate approach specific to the IoT Hub connection string would need to be used. There is likely to be the need to create a separate and dedicated path for using an Iot Hub connection string.
-
Accepting the IoT Hub connection string would require the Event Hubs client to take responsibility for knowing specific details of the IoT Hub service. Some of these items are:
- Relying on the convention that the name of the IoT Hub is the same as the name of the Event Hub
- Understanding the details for how IoT Hub encodes their shared access signature, as it differs slightly from Event Hubs.
- Knowing the specific format required to infer the IoT Hub user name to authenticate
- Being aware of the special path for which an AMQP link is opened in order to trigger the expected error
- Depending on IoT Hub continuing to use the same metadata fields within the redirect error; while the error is defined as part of the AMQP specification, the metadata fields are marked as "may also include" which leaves their use to the discretion of the IoT Hub service which may change as the service is versioned.
- What is Azure IoT Hub?
- IoT Hub - Endpoints documentation
- IoT Hub - Read device-to-cloud messages from the built-in endpoint
- IoT Hub - Use message routes and custom endpoints for device-to-cloud messages
- Understand and use Azure IoT Hub SDKs
- Implementation sample - IoT Hub connection string translation