Iot Hub Secure Data Egress with Private Network and Public Device Endpoint

iothubprivatedataegress.md

Iot Hub Secure Data Egress with Private Network and Public Device Endpoint

This document explains how we can securely create IoT Hub in a private virtual network and at the same time allow devices (both leaf and edge devices) to connect to this IoT Hub without being part of the virtual network, the latter is the key part of the challenge we are addressing in this solution. If we enforce devices to be in the same network as that of IoT Hub, the flexibility is lost as not all devices can connect to Azure Virtual Network without a significant pre-work ahead of time, which may not even be an option in some cases. On the other hand, if we do not enable Private Link for IoT Hub, the data egress endpoint is made publicly available which is not always ideal, as most customers would like to keep data processing private.

The following diagram provides the overarching solution and it's key components:

Components' Configuration

This section describes the configurations required for various key components in the solution:

Azure Virtual Network

Create a Virtual Network with the following configuration:

1. Address Space (Default): 10.0.0.0/16
2. Subnets:

1. Name: default, Range: 10.0.0.0/24
2. Name: AzureFirewallSubnet, Range: 10.0.1.0/24
3. Name: AzureFirewallManagementSubnet, Range: 10.0.2.0/24

Azure IoT Hub

Configure IoT Hub for Private endpoint/link as described here and disable Public Access. Please use the above Virtual Network and default subnet when configuring private link.

This will create private DNS zones and records for your ingress and egress endpoints of IoT Hub, mapping to private IP addresses in the virtual network. Private IP address for ingress endpoint is used above in the firewall's DNAT rules. You can view the private IP address allocated to IoT Hub's ingress endpoint by view private access endpoint link in your IoT Hub's networking section.

Azure Firewall

Create a Basic SKU of Azure Firewall in the above Virtual Network, it will use the AzureFirewallSubnet and AzureFirewallManagementSubnet precreated subnets.

This allows public IP of the firewall to be used as a gateway for private endpoints of IoT Hub, in this instance we only map private ingress IP/endpoint (not data engress) for various protocols (AMQP, MQTT and HTTPS/WS) supported by IoT Hub.

Configure DNAT rules as shown below (one for each connectivity protocol):

name	protocol	source type	source	destination addresses	destination ports	translated address	translated port
AMQP	TCP	IP address	*	<Firewall's Public IP>	5671	<IoT Hub Private Ingress IP>	5671
MQTT	TCP	IP address	*	<Firewall's Public IP>	8883	<IoT Hub Private Ingress IP>	8883
HTTPS/WS	TCP	IP address	*	<Firewall's Public IP>	443	<IoT Hub Private Ingress IP>	443

Edge Device/Machine

Primary change here is to do with mapping IoT Hub ingress FQDN (my-iothub.azure-devices.net) to a Firewall's public IP address. This is required because we do not have public endpoints of IoT Hub enabled anymore, those endpoints are now only available in the private virtual network which are in turn exposed by Azure Firewall's public IP.

The simplest way to achieve this by editing the host file on the machine as below:

1. Open host file for editing: sudo nano /etc/hosts.
2. Add the record: <Firewall's Public IP> <IoT Hub's Ingress FQDN e.g. my-iothub.azure-devices.net>.
3. Save the host file.
4. Deploy IoT Edge or install leaf device as usual.

Adding the Firewall Public IP address translation in Docker

By design, Docker does not pick up etc/hosts file contents. To pass the Firewall's Public IP address to map to the <my-iothub.azure-devices.net> follow these steps:

1. Update /etc/config.toml file to initialize the edgeAgent as follows:

[agent.config]
image = "mcr.microsoft.com/azureiotedge-agent:1.4"
createOptions = { HostConfig = { ExtraHosts = ["<my-iothub>.azure-devices.net:Firewall Public IP"] } }

2. Update deployment manifest for edgeAgent and edgeHub to include the ExtraHosts section (only subsection of manifest file shown below):

"systemModules": {
            "edgeAgent": {
              "type": "docker",
              "settings": {
                "image": "mcr.microsoft.com/azureiotedge-agent:1.4",
                "createOptions": "{\"HostConfig\":{\"ExtraHosts\":[\"<my-iot>.azure-devices.net:Firewall_Public_IP\"]}}"
              }
            },
            "edgeHub": {
              "type": "docker",
              "status": "running",
              "restartPolicy": "always",
              "settings": {
                "image": "mcr.microsoft.com/azureiotedge-hub:1.4",
                "createOptions": "{\"HostConfig\":{\"ExtraHosts\":[\"<my-iot>.azure-devices.net:Firewall_Public_IP\"],\"PortBindings\":{\"443/tcp\":[{\"HostPort\":\"443\"}],\"5671/tcp\":[{\"HostPort\":\"5671\"}],\"8883/tcp\":[{\"HostPort\":\"8883\"}]}}}"
              }
            }
          },

3. Apply configuration and validate modules are running.

DNS Resolution with Azure Private DNS Zone and Private DNS Resolver

The approach described above is one way of statically configuring the IoT Hub's new DNS entry pointing to the firewall's exposed IP address. There are a few approaches to simplify this setup.

One option is to leverage Azure services to take care of DNS resolution and set this in the linux edge machine. Azure DNS Private Resolver with a private DNS zone record, leveraging the Azure Firewall DNAT rules to expose the DNS resolution publicly allows us to validate this option.

Cloud configuration

Note that using this approach the resolution of <any>.azure-devices.net will fail if not configured as DNS records. This is because of the Private DNS zone settings.

1. Create a second Virtual Network with the following subnets: default, inbound. Ensure the address space for this VNET does not overlap with the previous one created earlier.
2. Create Azure Private DNS Zone named azure-devices.net which has an A record <my-iothub> pointing to the Public IP address of the Firewall created above.
3. Create Azure DNS Private Resolver. Add inbound endpoint and attach it to the inbound subnet. Once deployed, note the private IP of the private DNS private resolver's inbound endpoint. Network security groups will also be created.
4. Add VNET Peering between the first Virtual Network and the one create in this section. Allow traffic from the first VNET to the second one, other way around is not required.
5. Configure the inbound Network Security Group, inbound rule that allows traffic from the Firewall's private IP address to the private IP address of the DNS resolver inbound endpoint on port 53.

source	source IP/CIDR	source port ranges	destination	destination IP/CIDR	service	destination ports	protocol	action
IP Addresses	<Firewall Private IP>	*	IP Addresses	<Private IP of DNS resolver>	DNS (UDP)	53	UDP	Allow

6. Configure a DNAT Rule in the Firewall that translates traffic from the firewall's public IP to the private IP of the DNS resolver.

name	protocol	source type	source	destination addresses	destination ports	translated address	translated port
DNS	UDP	IP address	*	<Firewall's Public IP>	53	<Private IP of DNS resolver>	53

Edge configuration

Have DNS resolution point to the new Firewall Public IP for address resolution. The Private DNS resolver also leverages Azure DNS for resolving public domains but you can always add a second nameserver and search domain as required. There are different ways to overwrite /etc/resolv.conf file. Below, one of the approaches is described.

1. Create a new static file for resolv.conf, for example at /dns/staticresolv.conf.
2. Add the nameserver entry pointing to the Public IP of the firewall:

nameserver <Firewall's IP Address>
# optionally others here and search entries

3. Delete resolv.conf file and create new symbolic link

sudo rm -f /etc/resolv.conf
sudo ln -s /dns/staticresolv.conf  /etc/resolv.conf

4. Test DNS resolution for the IoT Hub endpoint, it should return the Public IP address of the Firewall.

nslookup <my-iothub>.azure-devices.net

5. Deploy IoT Edge.

Other options for DNS Resolution

There are also other options to deal with DNS resolution of the IoT Hub's address to the public firewal endpoint:

• Configuration of a local DNS server on Linux machine with tools such as dnsmasq or BIND 9. This requires changes at the host level and per-device.
• Leveraging an on-premises existing DNS server with special forwarding rules. This option might not require any changes on the Linux edge machine itself if the DNS resolution is already pointing to this on-premises nameserver.

Points to Ponder

1. Cost of Azure Firewall Azure Firewall pricing is provided here, depending on the SKU selected the cost differs hugely, the above solution only requires Basic SKU. While considering this cost vector, you may want to consider reusing the existing the virtual firewall appliance you may have in already in place or leveraging this Firewall instance as a shared resource e.g. exposing multiple IoT Hubs