This document discusses the current state of the state of the DataHandling GE and makes several propositions for its future evolution.
[TOC]
The DataHandling GE is a NGSI gateway level component in charge of generating Complex Events (aggregation data) from updates generated by local NGSI Context Providers (e.g. sensors). These Complex Events are in turn published as new Context Information according the NGSI data model and propagated to NGSI Context Managers (e.g. a remote broker).
participant Sensors as A
participant EspR4FastData as B
participant Remote brokers as C
A->B: update(S1.temp, 25)
A->B: update(S2.temp, 26)
B->C: update(ROOM1.avgtemp, 25.5)
The DataHandling GE internal name is "EspR4FastData".
After an extensive review of the code and architecture of the current implementation (EspR4FastData 4.3.1), we have concluded that a major rewrite is necessary to address the limitations of the current architecture and the integration to the rest of the NGSI ecosystem.
EspR4FastData has partially implemented the NGSI9/10 protocol in order to become a kind of Context Manager. The current implementation makes the assumption that it sits between the Context Providers (sensors) and a Context Broker (like Orion).
In this implementation, a Context Provider must register to EspR4FastData before sending updates. EspR4FastData will not forward these register/updates to the Context Broker (thus hiding the Context Provider entirely to the rest of the NGSI network). Only new Context Entities generated from the "complex" events by the CEP engine will be sent to the Context Broker.
EspR4FastData also exposes a control API (the /cep endpoint) allowing to dynamically add and remove CEP statements. Despite this API being well suited for prototyping a service on a single gateway, we argue that it is not fitted to configure a large set of gateways (as each statement must be added or removed one by one, maintaining the state of all the gateways synchronized will be difficult).
Moreover, an actual limitation requires the Context Providers (sensors) to have registered themselves before a related CEP statement can be added to the CEP engine. Thus preventing any pre-provisioning of CEP statements.
participant Admin as A
participant Context Provider as B
participant EspR4FastData as C
participant Context Broker as D
B->C: register(S1.temp)
A->C: /cep/statement(OUT1.AvgTemp)
B->C: update(S1.temp, 27)
B->C: update(S1.temp, 25)
C->D: register(OUT1.AvgTemp)
C->D: update(OUT1.AvgTemp, 26)
The code is in a more or less in a prototype state : a lot of unchecked assumptions are made, no unit tests, no clean separation (model classes, package inter-dependencies), no data synchronisation (concurrent access support), a home grown persistent storage layer and complete synchronous HTTP client calls (no queuing).
The current implementation also imposes some wrong requirements :
- Context Providers must register all their active attributes,
- Each registration request is limited to a single NGSI type,
- CEP statements can only be added to the CEP engine if the related Context Provider have already registered.
#Propositions
We propose to use the IoT Agent library as a reference implementation of how Context Providers (sensors) exposes their Context Information (NGSI sensor's data model).
###Why ?
- The IoT Agent library provides an easy way to expose sensors as NGSI Context Provider. It is easy to customize and well suited to run on the gateway hardware (Raspberry PI).
- The IoT Agent library supports multiple types of attributes (active, lazy, commands). Currently, EspR4FastData can only handle active attributes.
- Will ease the integration of EspR4FastData to the rest of the NGSI ecosystem.
The rest of this documentation may use the term "IoT Agent" when talking about sensors as NGSI Context Providers.
##Proposition 2: Make EspR4FastData independent
We propose to make EspR4FastData more independent : the insertion of CEP statements in the CEP engine should not depend on the state of the Context Providers. The configuration of EspR4FastData should provide all the information that the CEP needs to initialize itself. This proposition is based the expectations of the IoT Agent library: active attributes do not register themselves to the Context Broker (only lazy and command attributes do).
So EspeR4FastData will handle NGSI10 /updateContext requests :
participant IoT Agent as C
participant EspR4FastData as B
participant Remote Context Broker as D
participant Admin as A
A->B: /setup
C->B: update(S1.temp, 27)
C->B: update(S1.temp, 25)
B->D: update(OUT1.avgTemp, 26)
Of course to handle the "local only" scenario, EspR4FastData can directly send /updateContext requests directly to the IoT Agent (e.g sending commands like closing windows when T° drops) :
participant IoT Agent as C
participant EspR4FastData as B
participant Admin as A
A->B: /setup
C->B: update(S1.temp, 20)
C->B: update(S1.temp, 18)
C->B: update(S1.temp, 16)
B->C: update(WINDOW1.state, "CLOSE")
B->C: update(WINDOW2.state, "CLOSE")
We propose that EspR4FastData become an optional component of the NGSI gateway by adding support for NGSI 10 subscribeContext / notifyContext requests to a local Context Broker.
###Why ?
- EspR4FastData should not stand in between sensors and remove brokers. It is not the role of a CEP to be a NGSI Context Broker.
- EspR4FastData will only subscribe to updates targeting its CEP statements.
- It is the role of the local Context Broker to forward data to other remote Context Brokers.
participant IoT Agent as A
participant Local Context Broker as B
participant EspR4FastData as C
participant Remote Context Broker as D
C->B: subscribe(S1.temp)
A->B: update(S1.temp, 27)
B->D: update(S1.temp, 27)
B->C: notify(S1.temp, 27)
C->B: update(OUT1.avgTemp, 26)
B->D: update(OUT1.avgTemp, 26)
We propose that the EspR4FastData configuration become static. All its configuration should be defined in a single JSON description, only needing a single API endpoint.
###Why ?
- The goal is to make it possible to apply configuration changes like a transaction in DB terms : atomic and versionnable (allowing configuration to be reset to a previous known state if needed).
- Another goal is to allow deploying the same configuration on a cluster of gateways. This deployment would only require a single request to each gateway.
The /setup endpoint would accept GET (to return the actual JSON configuration) or POST (to apply a new JSON configuration). The endpoint will return 200 Ok on success, or 400 Bad Request if the new configuration cannot be applied.
This is the proposed format :
{
"in": [
{
"id": "S.*",
"type": "TempSensor",
"isPattern": true,
"attributes": [
{
"name": "temp",
"type": "float"
}
],
"providers": [
"http://localhost:1902/ngsi10"
]
}
],
"out": [
{
"id": "OUT1",
"isPattern": false,
"type": "TempSensorAvg",
"attributes": [
{
"name": "avgTemp",
"type": "float"
}
],
"brokers": [
{
"url": "http://102.232.332:1903/v1",
"serviceName": "my",
"servicePath": "/test/path",
"register": false
}
]
}
],
"statements": [
"INSERT INTO 'TempSensorAvg' SELECT 'OUT1' as id,
avg(TempSensor.temp) as avgTemp
FROM TempSensor.win:time(86400)
WHERE TempSensor.id = 'S1' "
]
}The "in" array defines the list of incoming events (NGSI Context Elements) the CEP expects as input from Context Providers.
There is two ways Context Providers can notify the CEP:
- The simplest way is for the Context Provider is to send /updateContext requests to the CEP. This does not require the Context Providers to be listed in the "providers" field.
participant IoT Agent as C
participant EspR4FastData as B
C->B: update(S1.temp, 27)
C->B: update(S1.temp, 25)
- The second way (from Proposition 3) is to make the CEP subscribe to the Context Provider. For each Context Provider defined in the "providers" field of an event, the CEP will send initialy a /subscribeContext, and will expect in return the Context Provider to notify it with /notifyContext requests.
participant Context Provider as C
participant EspR4FastData as B
B->C: subscribe(S1.temp)
C->B: notify(S1.temp, 27)
C->B: notify(S1.temp, 25)
The "out" array defines the list of outgoing events (those generated by the CEP). Each outgoing event can have multiple brokers. For each broker, the CEP can behave in two ways (push or pull modes) :
- if the "register" field is
false, the CEP will just send /updateContext requests to the broker on each event update (push mode).
participant EspR4FastData as A
participant Remote broker as B
A->B: update(S1.temp, 25)
A->B: update(S1.temp, 26)
- if the "register" field is
true, the CEP will only register itself to the broker with the /registerContext as the providing application for the event. The CEP will then expect the broker to query the CEP with /queryContext request (pull mode).
participant EspR4FastData as A
participant Remote broker as B
A->B: register(S1.temp)
B->A: query(S1.temp)
A-->B: response(25)
B->A: query(S1.temp)
A-->B: response(26)
Note: the pull mode is a proposition. It might not be relevant when EspR4FastData is used with a local broker (proposition 3).
"statements" defines a list of Esper EPL statements that will interact with the events previously defined.
- Support JSON for NGSI requests, and later NGSI2
- Support PEP or OAuth2.0 authentication to Orion Context Broker
- Support Orion specific headers : Fiware-ServiceName and Fiware-ServicePath
- Support lazy attributes by polling the state on demand
- Rewrite the EspR4FastData GUI to handle the new JSON configuration format