Created
May 5, 2020 08:19
-
-
Save shabunin/2ac028492ba5197294f9400f4dab03d7 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| __________________________________________ | |
| BREAKING BAD - DIGGING INTO KNX NETWORK. | |
| Vladimir Shabunin | |
| __________________________________________ | |
| Table of Contents | |
| _________________ | |
| 1 LICENSE | |
| 2 Thanks | |
| 3 Intro | |
| 4 FT1.2 | |
| .. 4.1 Frame format | |
| .. 4.2 Decoding algorithm: | |
| 5 BAOS | |
| .. 5.1 Sending request | |
| .. 5.2 Getting response | |
| .. 5.3 Description of services | |
| .. 5.4 Binary protocols to json | |
| .. 5.5 Redis as a pubsub broker | |
| .. 5.6 More on Datapoint SDK | |
| 6 LinkLayer | |
| .. 6.1 General | |
| .. 6.2 Idea | |
| .. 6.3 KNXNet/IP | |
| .. 6.4 cEMI | |
| .. 6.5 Transport connections | |
| .. 6.6 "Mistake" made | |
| .. 6.7 TProxy | |
| .. 6.8 Finally | |
| ..... 6.8.1 dobaosll_pub | |
| ..... 6.8.2 dobaosll_net | |
| .. 6.9 Scanner | |
| .. 6.10 Other utils | |
| ..... 6.10.1 mprop_reader | |
| ..... 6.10.2 stream converter | |
| 7 References | |
| 1 LICENSE | |
| ========= | |
| Copyright(c) 2020 Vladimir Shabunin. This work licensed by MIT | |
| licence, as is, without any warranties. | |
| If you have found one or more errors, please contact me: va.shabunin | |
| at yandex.ru. | |
| 2 Thanks | |
| ======== | |
| We're all standing on the shoulders of giants. So, thanks to all of | |
| the people of past and present, which works and inventions help us | |
| nowadays. Thanks to Weinzierl for it's BAOS modules and software | |
| stack. Thanks to open-source community for it's works. | |
| 3 Intro | |
| ======= | |
| I started to work with KNX bus six years ago. All this time I | |
| wondered how can I send some data to knx gateway to change value? I | |
| wanted to implement my own KNXNet/IP client a long time. But when I | |
| sent some tunneling request frame to UDP port 3671 of ip interface, | |
| nothing happens. Not even a single byte in response to give me a | |
| hint. That time I didn't know what's going on. Now I'm digging more | |
| and more into KNX communications. I've got a BAOS kBerry module back | |
| in 2017. I tried to connect kDrive express library to nodejs script. | |
| I succeeded, but I was not satisfied. | |
| So, I found Bus Access Object Server (BAOS) protocol pdf description | |
| [1]. Have found npm package serialport [2], started to | |
| experiment. Got a first data. | |
| BAOS protocol is wrapped to FT1.2 frames Serialport data is going like | |
| a stream (as well as TCP) without any message length information. | |
| So, first challenge was to decode incoming data. | |
| 4 FT1.2 | |
| ======= | |
| 4.1 Frame format | |
| ~~~~~~~~~~~~~~~~ | |
| It is described in [1](BAOS protocol v2) and other sources as well. | |
| There is three types of messages: acknowledge byte, fixed(4 bytes) and | |
| with variable length. Acknowledge byte is 0xE5. Fixed frame has a | |
| length of four bytes and in BAOS module is present reset | |
| request/indication. Data frame length varies. | |
| ---------------------------------------------------------------------------- | |
| start len len start control field ..data... ..data.. checksum end | |
| ---------------------------------------------------------------------------- | |
| 0x68 LL LL 0x68 0x73/0x53 0x.. 0x.. 0x.. 0x16 | |
| ---------------------------------------------------------------------------- | |
| LL - length | |
| control field depends on message direction(host<->server) and frame | |
| count parity(odd/even) after reset request or reset indication. | |
| checksum is module of arithmetic sum of data bytes and control field | |
| by 256. | |
| As I said before, data going through serialport like a stream. No | |
| beginning. No end. Sometimes there is some data. You can get your | |
| message not completely in one process iteration(it depends on library, | |
| I believe). | |
| So, next situations is possible: | |
| 1. Incomplete message. | |
| -------------------------------------------- | |
| data chunk 1 0x68-ll-ll-0x68-data..data.. | |
| -------------------------------------------- | |
| data chunk 2 ..data..data..checksum..0x16 | |
| -------------------------------------------- | |
| 1. Beginning of next frame in one data reading. | |
| ------------------------------------------------ | |
| chunk 1 0x68-ll-ll-0x68-....-0x16-0x68-ll-ll- | |
| ------------------------------------------------ | |
| chunk 2 0x68-data....end | |
| ------------------------------------------------ | |
| 1. Acknowledge come together with data. | |
| ------------------------- | |
| data 0xe5-0x68-0x68... | |
| ------------------------- | |
| 1. Another unexpected errors | |
| 4.2 Decoding algorithm: | |
| ~~~~~~~~~~~~~~~~~~~~~~~ | |
| 1. Receive data chunk by chunk. When data arrives, append it to some | |
| buffer variable. | |
| 2. Compare first byte to acknowledge 0xE5. If equal, run | |
| callback. Delete first byte from buffer. | |
| 3. If buffer length is greater or equal than 4 bytes, then check: | |
| + if 4 bytes is ResetInd fixed frame. If so, run callback. Delete | |
| first four bytes from buffer. | |
| + if first and fourth bytes are start byte 0x68, then extract whole | |
| frame, based on frame length info, if possible. | |
| + if no possible extract data message with variable length(buffer | |
| is shorter than expected), wait for new data chunk. | |
| 4. If buffer length is less than 4 check if first byte is ResetInd[0] | |
| or data frame start 0x68. If so, wait for new data. | |
| 5. If first byte cannot be recognized (nor 0x68, nor resetInd[0], nor | |
| ack), delete it from buffer. | |
| 5 BAOS | |
| ====== | |
| Good. Frames are decoded. Now it's time to send some request to module | |
| and understand it's response. So, I studied over protocol | |
| description. It was kind of pleasure to me: specifications are located | |
| in one file, simple enough(still challenging) and all necessary | |
| information is present. The only external resources I needed was | |
| programming manuals, library references, answers. | |
| 5.1 Sending request | |
| ~~~~~~~~~~~~~~~~~~~ | |
| Let send first request. Actually, I don't remember what it | |
| was. Suppose GetServerItem.Req: | |
| -------------------------------------------------------- | |
| main service sub service start item number of items | |
| -------------------------------------------------------- | |
| 0xF0 0x01 0x0001 0x0001 | |
| -------------------------------------------------------- | |
| Main service is 0xF0 for all BAOS services. Subservice value differs. | |
| Start item and Number of items is ushort(unsigned short 16bit | |
| integers) values. | |
| Note: the table above is just "data" part of FT1.2 frame. The full | |
| sent message will be kind of: | |
| --------------------------------------------------------------------------- | |
| 0x68 0x07 0x07 0x68 CF 0xF0 0x01 0x00 0x01 0x00 0x01 CRC 0x16 | |
| --------------------------------------------------------------------------- | |
| Response is wrapped as a ft1.2 frame as well. | |
| 5.2 Getting response | |
| ~~~~~~~~~~~~~~~~~~~~ | |
| There is two kind of response messages. One, indicating errors, other | |
| with result in case of success. Both of them are documented well in | |
| [1] pdf file. | |
| -------------------------------------------------------------------- | |
| main service sub service start item number of items error code | |
| -------------------------------------------------------------------- | |
| 0xF0 0x81 0x0001 0x0000 0x0b | |
| -------------------------------------------------------------------- | |
| Aha, number of returned items is zero, something went wrong. Take a | |
| look at error table and solve the problem. | |
| --------------------------------------------------------------------- | |
| main service sub service start number first id data len data | |
| --------------------------------------------------------------------- | |
| 0xF0 0x81 0x0001 0x0001 0x0001 0x01 0xff | |
| --------------------------------------------------------------------- | |
| That was good response. So, now it is possible to get service | |
| information with GetServerItem.Req | |
| 5.3 Description of services | |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
| ----------------------------------------------------------------------------------------- | |
| Service name Service description | |
| ----------------------------------------------------------------------------------------- | |
| GetDatapointDescription.Req request to get description of datapoints. | |
| Description includes datapoint type, length, flags(CRWTU) | |
| ----------------------------------------------------------------------------------------- | |
| GetDatapointDescription.Res response for GetDatapointDescription.Req request. | |
| ----------------------------------------------------------------------------------------- | |
| GetDatapointValue.Req get datapoint value, stored in BAOS module. | |
| ----------------------------------------------------------------------------------------- | |
| GetDatapointValue.Res | |
| ----------------------------------------------------------------------------------------- | |
| SetDatapointValue.Req send set datapoint value command to BAOS module. | |
| Commands are: | |
| set(store in BAOS), | |
| send to KNX bus, set and send to bus, | |
| read value | |
| (for getting read response datapoint should have U flag). | |
| ----------------------------------------------------------------------------------------- | |
| SetDatapointValue.Res | |
| ----------------------------------------------------------------------------------------- | |
| DatapointValue.Ind if server item "Sending indications" is set to 1 and value | |
| been changed on bus(or read response received), | |
| this message is transmited from BAOS module to host. | |
| ----------------------------------------------------------------------------------------- | |
| GetParameterByte.Req get parameter bytes, configured in ETS. | |
| ----------------------------------------------------------------------------------------- | |
| GetParameterByte.Res | |
| ----------------------------------------------------------------------------------------- | |
| GetServerItem.Req get server items, like serial number, bus connected state, | |
| programming mode of module, etc. | |
| ----------------------------------------------------------------------------------------- | |
| GetServerItem.Res | |
| ----------------------------------------------------------------------------------------- | |
| ServerItem.Ind indicating that some state of BAOS module changed, | |
| e.g. programming button pressed, | |
| or bus connected/disconnected. | |
| ----------------------------------------------------------------------------------------- | |
| SetServerItem.Req change writeable server item. | |
| Set programming mode, for example. | |
| ----------------------------------------------------------------------------------------- | |
| SetServerItem.Res | |
| ----------------------------------------------------------------------------------------- | |
| There is more services described in documentation, not supported by | |
| BAOS 83x modules. | |
| 5.4 Binary protocols to json | |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
| I implemented core services. A question raised in my mind. How to | |
| work with datapoint values from different applications? Serialport | |
| can be open once and no applications shall connect more. I started to | |
| experiment with sockets(Unix-sockets, but TCP/UDP are suited as well). | |
| ,---- | |
| | -------------------------------- | |
| | | |
| | +------------+ | |
| | | KNX BUS | BINARY DATA | |
| | +------------+ | |
| | | | |
| | +------------+ | |
| | ------------- | BAOS 83x | ------------------------------- | |
| | +------------+ BAOS BINARY PROTOCOL | |
| | | 0x60 ... 0x68 ... 0x16 | |
| | +----------------+ | |
| | ----------- | Datapoint SDK | ------------------------------- | |
| | +----------------+ | |
| | | unix socket | JSON PROTOCOL | |
| | +-------+--------+ { | |
| | | "request_id": 1, | |
| | | "method": "get value", | |
| | +---------+-------+ "payload": {"id": 1} | |
| | / | \ } | |
| | / | \ | |
| | +----------+ +----------+ +----------+ | |
| | | Client_1 | | Client_2 | | Client_3 | | |
| | +----------+ +----------+ +----------+ | |
| | | |
| | -------------------------------- | |
| `---- | |
| So, the idea was that application listen certain socket and connected | |
| clients can send requests, receive responses and indications. | |
| Communication protocol I implemented was simple: messages are json | |
| objects with "request_id", "method" and "payload" field. | |
| To send, for example, "get value" request: | |
| ,---- | |
| | {"request_id":65362052157,"method":"get value","payload":{"id":31}} | |
| `---- | |
| Then response: | |
| ,---- | |
| | {"response_id":65362052157,"method":"get value","payload":{"id":31,"value":74},"success":true} | |
| `---- | |
| That was in first version of sdk. Since that time, I moved | |
| interprocess communication to redis backend. Protocol has been | |
| simplified.. | |
| 5.5 Redis as a pubsub broker | |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
| I started to work with redis because of its publish/subscribe feature. | |
| In a working process I discovered advantages that redis brings with | |
| it: | |
| + pubsub architecture allow you to implement message exchange between | |
| all processes, not only two parties as in client-server | |
| + it's not only pubsub, it's a rich-featured in-memory database | |
| + keeping data in memory may reduces number of disk write/read | |
| operations, improving speed and health state of emmc/sd storages. | |
| + there is a lot of client libraries([https://redis.io/clients] [3]) | |
| Redis is able to do a lot of things, still, it is lightweight, | |
| consuming only few megabytes of disk space. | |
| Since version 5.0.0 there is new data type - redis | |
| streams([https://redis.io/topics/streams-intro] [4]). In a few words | |
| - it is kind of time-series database. Although it is stored in RAM, | |
| it consumes it in efficient way. One million of datapoint records may | |
| take only around 20 megabytes. And this number can be reduced. | |
| 5.6 More on Datapoint SDK | |
| ~~~~~~~~~~~~~~~~~~~~~~~~~ | |
| BAOS binary protocol allows you to work with datapoint values. Still, | |
| datapoint values are presented as a raw bytes. But BAOS provide us | |
| with datapoint type and length information, so it's possible to | |
| implement automatic value conversion between knx value and json in | |
| both directions. | |
| To understand datapoint types refer to "03_07_02 Datapoint Types | |
| v01.08.02 AS.pdf" from KNX Standard [5]. | |
| There is existing implementations for different enviromnents. I based | |
| mine for nodejs on Rafelder's work | |
| [https://github.com/Rafelder/knx-datapoints] [6]. | |
| BAOS support types DPT1-14, DPT16, DPT18. Most of them are standart | |
| types that can be met in typed programming languages: bool, char, | |
| ubyte, ushort(16bit), short, uint(32bit), int. Some of them are | |
| specific for KNX - DPT2-3, DPT10-11, DPT18. | |
| Working with simple types will look like following example: | |
| ,---- | |
| | class DPT5 { | |
| | static public ubyte toUByte(ubyte[] raw) { | |
| | auto value = raw.read!ubyte(); | |
| | return value; | |
| | } | |
| | static public ubyte[] toUBytes(ubyte value) { | |
| | ubyte[] res; | |
| | res.length = 1; | |
| | res.write!ubyte(value, 0); | |
| | return res; | |
| | } | |
| | } | |
| `---- | |
| Workign with knx specific values look like this: | |
| ,---- | |
| | class DPT3 { | |
| | static public ubyte[string] toDecoded(ubyte[] raw) { | |
| | ubyte[string] result; | |
| | | |
| | result["direction"] = (raw[0] & 0x08) >> 3; | |
| | result["step"] = raw[0] & 0x07; | |
| | | |
| | return result; | |
| | } | |
| | static public ubyte[] toUBytes(ubyte[string] value) { | |
| | ubyte[] result; | |
| | | |
| | result.length = 1; | |
| | result[0] = 0x00; | |
| | result[0] = result[0] | ((value["direction"] << 3) & 0xff); | |
| | result[0] = result[0] | value["step"]; | |
| | | |
| | return result; | |
| | } | |
| | } | |
| `---- | |
| Working with datapoint values(as well as working with binary | |
| protocols) requires knowledge of bitwise operations. | |
| The most difficult to understand for me was DPT9 type. It is float | |
| value of 16 bit. | |
| Bit by bit: MEEEEMMM MMMMMMMM | |
| M - means mantissa, -2048 <= M <= 2047. First M bit describes sign. If | |
| 1, then float value is less than zero. | |
| E - exponent, 0 <= E <= 15. | |
| FloatValue = (0.01*M)*(2^E) | |
| Understanding it gives general knowledge about how float numbers are | |
| represented in computer systems. | |
| ---------------------------------------------------------------------- | |
| Now, few words about receiving datapoint descriptions. There is 1000 | |
| datapoints that can be configured. Therefore, requesting description | |
| of all datapoints one by one may require significant amount of time. | |
| GetDatapointDescription.Req has "Number of datapoints" parameter. It | |
| is possible to get multiple description in one response. But if send | |
| GetDatapointDescription.Req(start = 1, number = 1000), error "Buffer | |
| too small" will be returned from BAOS module. Also, there is two | |
| server items: "Maximal buffer size" and "Current buffer size". | |
| Knowing "Current buffer size" allows to calculate number parameter for | |
| GetDatapointDescription request. | |
| GetDatapointDescription.Res has following structure: | |
| ------------------------------- | |
| MainService 1byte | |
| SubService 1byte | |
| Start 2bytes | |
| Number 2bytes | |
| First DP id 2bytes | |
| First DP value type 1byte | |
| First DP config flags 1byte | |
| First DP DPT 1byte | |
| .... ... | |
| Last DP id 1byte | |
| .... ... | |
| Last DP DTP 1byte | |
| ------------------------------- | |
| Response is constructed from header(6 bytes) and info for each | |
| datapoint(5 bytes for each). | |
| Therefore, NumberOfDatapoints = [(CurrentBuffSize - 6)/5]. | |
| With buffer size of 250 bytes NumberOfDatapoints equals 48. Aha, it | |
| will take 21 requests only to get description of all datapoints. Much | |
| better than one thousand messages. | |
| The same buffer length counting applies to GetDatapointValue.Res and | |
| SetDatapointValue.Req. For these services to get correct message | |
| length it is needed to know datapoint types because of different value | |
| lengths. | |
| Now I describe general algorithm of datapoint sdk implementation: | |
| 1) Send reset request to module. | |
| 2) Switch module to BAOS mode. | |
| 3) Get "Current buffer size server item. | |
| 4) Get all datapoint descriptions | |
| 5) Main loop | |
| - receive BAOS messages, send to User | |
| - receive User requests, send to BAOS | |
| 6 LinkLayer | |
| =========== | |
| 6.1 General | |
| ~~~~~~~~~~~ | |
| Besides ObjectServer protocol, BAOS module support working with cEMI | |
| LinkLayer messages. The KNX protocol is specified according to the | |
| OSI reference model (Open Systems Interconnection) as a set of layers. | |
| BAOS mode and LinkLayer mode works with two different layer - | |
| Application(higher) and LinkLayer. Speaking by analogy, BAOS is like | |
| an all-ready for use HTTP protocol, and LinkLayer is an access to data | |
| going inside your network cables and wireless channels, it's lower | |
| than a TCP/UDP sockets. | |
| To be not mistaken - data bytes going inside KNX cable(IMI - internal | |
| message interface) differs from cEMI. Link layer is just a gateway | |
| from user application to KNX bus. ObjectServer may be called a | |
| gateway too, difference is in number of abstractions and operations | |
| between BAOS datapoint and KNX bus and, on the other side, cEMI frame | |
| and KNX bus. If you change BAOS datapoint value, it will be changed | |
| in BAOS memory, stored there, sent to BUS. Datapoint information(type, | |
| flags, etc) can be extracted from BAOS chip also. It's KNX stack is | |
| certified and there is no need of studying and sticking to KNX | |
| specifications for developer, who works with BAOS. Data Link Layer | |
| advantages lies in wide appliances - from work with group addresses to | |
| device configuration, on the other hand. | |
| By specification, KNX TP-UART transcievers should work with EMI1/2 | |
| frame format. Therefore, common software to work with KNX TP-UART | |
| devices was implemented with support of these formats and there may be | |
| no cEMI abstractions. So, running these software solutions with | |
| connected BAOS module may fail. | |
| Common External Message Interface is supposed to be medium independent | |
| and can be found in KNXNet/IP devices. | |
| More info can be found in "03_06_03 EMI_IMI v01.03.03 AS.pdf" [9]. | |
| 6.2 Idea | |
| ~~~~~~~~ | |
| The idea of implementing KNXNet/IP gateway was born in my head a long | |
| time ago, but it's become more than idea when I was working on a | |
| project without any KNXNetwork interface, but with single-board | |
| computer with BAOS 83x connected. I was tired of connecting USB | |
| interface over and over. | |
| The idea seemed lina a pretty easy task. All that is needed is just | |
| implement frame converter in both directions. From KNXNet/IP Client | |
| to UART and back. | |
| cEMI frames from Client: | |
| - parse KNXNet/IP wrapper | |
| - wrap to FT1.2 | |
| - send to UART | |
| cEMI from BAOS: | |
| - parse FT12 frame | |
| - wrap to KNXNet/IP frame | |
| - send to Client | |
| 6.3 KNXNet/IP | |
| ~~~~~~~~~~~~~ | |
| All messages, going through your LAN from client applications(ETS, | |
| different knx client implementations) from client to ip gateway and | |
| back should be encapsulated in KNXNetIP protocol. It is simple at a | |
| first look, but it’s complexity lies in different supported services. | |
| Each service has it’s own structure and working with it requires | |
| digging into KNX specifications (03_08_01 - 03_08_04 , 03_06_03 | |
| EMI_IMI v01.03.03 AS.pdf). Working with traffic dump software (like | |
| Wireshark) gives bunch of advantage. | |
| According to standard, Client must send SEARCH_REQUEST at first and | |
| this feature is a mandatory. I didn’t implement it for simplicity, | |
| since it requires working with Multicast UDP. | |
| ----------------------------------------------------- | |
| KNXNet/IP Client KNXNet/IP Server BAOS module | |
| ----------------------------------------------------- | |
| CONNECT_REQUEST > | |
| < CONNECT_RESPONSE | |
| DESCRIPTION_REQ > | |
| < DESCRIPTION_RES | |
| ----------------------------------------------------- | |
| CONNECTSTATE_REQ > | |
| < CONNECTSTATE_RES | |
| ----------------------------------------------------- | |
| TUNNELING_REQ > | |
| < TUNNELING_ACK | |
| FT12 cEMI > | |
| < FT12 cEMI | |
| < TUNNELING_REQ | |
| TUNNELING_ACK > | |
| ----------------------------------------------------- | |
| DISCONNECT_REQ > | |
| < DISCONNECT_RES | |
| ----------------------------------------------------- | |
| Let’s start with CONNECT_REQUEST. In order to establish connection it | |
| must be sent from client to server. All messaging going between | |
| devices is UDP datagrams and it means, that there is no really | |
| established connection between two ports, like in case of TCP | |
| socket. So, CONNECT_REQUEST consist information about host address and | |
| connection type(tunnel, device management, remote logging, etc). | |
| Tunnel and device management is the types that our application should | |
| support. | |
| On receiving CONNECT_REQUEST KNXNet Server implementation must | |
| decide(based on request data, maximum connection’s count and already | |
| connected clients) whether to create channel between two parties or | |
| decline request. In case of accepting, it creates connection | |
| abstraction with some channel id and send this id back to client. | |
| Every request, sent from KNXNet Client to Server and backwards must | |
| contain given number. If Tunneling Connection is decided to be | |
| established, Server must assign unique individual address to this | |
| connection and return it in response. | |
| After receiving accepting response, Client may send | |
| DESCRIPTION_REQUEST, on which Server should respond with information | |
| like knx medium, knx addres, device serial number, mac address, | |
| friendly name and supported service families. | |
| Client sends CONNECTIONSTATE_REQUEST with some period, like once in 30 | |
| seconds or once in 1 minute. So, on receiving such a request Server | |
| should check if connection with given input data like client ip | |
| address and port and channel id is still active and send approptiate | |
| response. | |
| Assuming, that connection is established and application receives | |
| first TUNNELING_REQUEST(or DEVICE_CONFIGURATION_REQUEST with same | |
| structure). Before processing it, we should check this request | |
| sequence number. There is two kind of this number: outgoing, that | |
| Client should check and incoming, that Server implementation should | |
| verify. On incoming TUNNELING_REQUEST check if this sequence number | |
| is equals to expected and if so, send TUNNELING_ACK response to Client | |
| and given cEMI frame to BAOS module. After that, increase sequence | |
| counter of this client and expect new frame with a new value. | |
| If server receives cEMI frame from BAOS, it should send this frame as | |
| a TUNNELING_REQUEST to clients. And this data contains sequence | |
| number as well. Here I have encountered issue when was trying to | |
| download ETS application to device through VPN connection. Due to | |
| network latency, ETS messages was on it's way to client before | |
| receiving TUNNELING_ACK for previous request. It was resolved with | |
| simple queue array. Do not send instantly received cEMI message to | |
| client, wait for acknowledge of previous req and only then increase | |
| sequence id and send next frame. | |
| 6.4 cEMI | |
| ~~~~~~~~ | |
| -------------------------------- | |
| Field Data type | |
| -------------------------------- | |
| Message Code ubyte | |
| -------------------------------- | |
| Additional Info Len ubyte | |
| Additional Info ubyte[] | |
| -------------------------------- | |
| Control Field 1 ubyte | |
| Control Field 2 ubyte | |
| Source address ushort | |
| Destination address ushort | |
| -------------------------------- | |
| APDU+Data Len ubyte | |
| TPDU ubyte | |
| APDU+Data ubyte[] | |
| -------------------------------- | |
| There is diffent cEMI message types. In every frame there is a byte | |
| indicating message code. Working with LinkLayer assumes that you will | |
| work with LData.Req, LData.Con, LData.Ind. | |
| If you want to send value to some group address, you send LData.Req | |
| message. If this request was confirmed and group address value has | |
| changed, you will get LData.Con frame back. If some group address on | |
| bus was changed without your participation, you will get LData.Ind as | |
| a message code. | |
| If you send LData.Req message, BAOS send back corresponding LData.Con | |
| message. Assuming that LData.req for group addres was sent and | |
| confirmation was received. If there is multiple tunneling | |
| connections, how all of them can know that that group address was | |
| changed? To achieve this, I do replace LData.Con message code to | |
| LData.Ind, send it to these Clients, who doesn’t await confirmation | |
| and send unchanged LData.Con to sender of request. | |
| Implementing multiple tunneling connection support was a "mistake". I | |
| take it in quotes because it is not according to specifications. and | |
| it is mistake in a good meaning: I started to learn why things is not | |
| working as I expected and got knowledge. | |
| Now I make a step away from KNXNet/IP implementation and tell about | |
| LData_cEMI abstraction I have written | |
| ,---- | |
| | | |
| | class LData_cEMI { | |
| | public MC message_code; | |
| | public ubyte[] additional_info; | |
| | | |
| | // control field 1 | |
| | private ubyte cf1; | |
| | public bool standard; // extended 0, standard 1 | |
| | public bool donorepeat; // 0 - repeat, 1 - do not | |
| | public bool sys_broadcast; | |
| | public ubyte priority; | |
| | public bool ack_requested; | |
| | public bool error; // 0 - no error(confirm) | |
| | // control field 2 | |
| | private ubyte cf2; | |
| | public bool address_type_group; // 0 - individual, 1 - group; | |
| | public ubyte hop_count; | |
| | public ubyte ext_frame_format; // 0 - std frame | |
| | | |
| | public ushort source; | |
| | public ushort dest; | |
| | | |
| | public ubyte apci_data_len; | |
| | private ubyte tpci_apci; | |
| | private ubyte[] apci_data; | |
| | | |
| | public ubyte tpci; | |
| | public TService tservice; | |
| | public ubyte tseq; | |
| | public APCI apci; | |
| | public ubyte[] data; | |
| | | |
| | // 6bits that goes together with apci, if apci_data_len > 1 | |
| | public ubyte tiny_data; | |
| | | |
| | ...... | |
| | ...... | |
| | ...... | |
| | this() { | |
| | // default values | |
| | message_code = MC.LDATA_REQ; | |
| | setControlFields(0xbc, 0xe0); | |
| | } | |
| | // const from array of data | |
| | this(ubyte[] msg) { | |
| | // parse frame | |
| | auto offset = 0; | |
| | message_code = cast(MC) msg.peek!ubyte(offset); offset += 1; | |
| | additional_info.length = msg.peek!ubyte(offset); offset += 1; | |
| | additional_info = msg[offset..offset + additional_info.length].dup; | |
| | offset += additional_info.length; | |
| | cf1 = msg.peek!ubyte(offset); offset += 1; | |
| | cf2 = msg.peek!ubyte(offset); offset += 1; | |
| | | |
| | // extract info from control fields | |
| | getCFInfo(); | |
| | .... | |
| | .... | |
| | .... | |
| | } | |
| | | |
| `---- | |
| This class can be construct with ubyte array or with empty | |
| argument. In case of empty argument, default values for message code | |
| and control fields are provided. | |
| ,---- | |
| | frame = new LData_cEMI(); | |
| | frame.message_code = MC.LDATA_REQ; | |
| | frame.source = 0x0000; | |
| | frame.dest = 0x1103; | |
| | frame.tservice = TService.TConnect; | |
| `---- | |
| If you construct new instance of class with ubyte array it parses it | |
| byte by byte and fill all properties. The most tricky part of cEMI | |
| frame is TPDU(Transport Protocol Data Unit) and APDU(Application | |
| Protocol Data Unit). It is coded not in separate bytes, like any | |
| other info, it should be extracted with bitwise operations. Yeah, | |
| control field information should be extracted bit by bit too. But | |
| TPDU is complicated because some data(like APCI - Application Protocol | |
| Control Information) may take two last bits of one byte and two first | |
| bits of following. It may have different length: 4 or 10 bits. | |
| So, let's start examining from apci_data_length byte(one following | |
| destination address). It values indicates number of APCI and data | |
| payload bytes, TPCI byte is not included. Next is going TPCI | |
| information. It may take 6 bit or whole data byte(in case | |
| apci_data_length equals 0). TPCI stands for Transport Protocol | |
| Control Information and it provides information about transport | |
| service of given cEMI frame. Transport Service may be one of | |
| following: | |
| ,---- | |
| | enum TService { | |
| | unknown, | |
| | TDataBroadcast, | |
| | TDataGroup, | |
| | TDataTagGroup, | |
| | TDataIndividual, | |
| | TDataConnected, | |
| | TConnect, | |
| | TDisconnect, | |
| | TAck, | |
| | TNack | |
| | } | |
| `---- | |
| Detailed information about coding can be found in "03_03_04 Transport | |
| Layer" [13]. | |
| When group address is broadcasted, TPCI has TDataGroup value. When | |
| client make connection to some device(e.g. ETS want to start | |
| application downloading), he sends request with TConnect TPCI. When | |
| data is going in terms of connection between two | |
| parties(e.g. application download process or device info reading), | |
| TDataConnected is going back and forth alongside with TAck/TNack. | |
| Transport Service Information is presented in class by variables tpci, | |
| tservice, tseq (where tservice variable is one of enum I gave before | |
| and tseq is a sequence number for TDataConnected/TAck) and these | |
| variables are participating in following methods: | |
| ,---- | |
| | public void getTransportServiceInfo() { | |
| | // extract information from tpci data bits | |
| | bool data_control_flag = to!bool((tpci >> 7) & 0b1); | |
| | bool numbered = to!bool((tpci >> 6) & 0b1); | |
| | tseq = to!ubyte((tpci >> 2 ) & 0b1111); | |
| | if (address_type_group) { | |
| | // group address | |
| | if (!data_control_flag && !numbered) { | |
| | if (dest == 0 && tseq == 0) tservice = TService.TDataBroadcast; | |
| | if (dest != 0 && tseq == 0) tservice = TService.TDataGroup; | |
| | if (tseq != 0) tservice = TService.TDataTagGroup; | |
| | } | |
| | } else { | |
| | // individual addr | |
| | if (!data_control_flag && !numbered && tseq == 0) tservice = TService.TDataIndividual; | |
| | else if (!data_control_flag && numbered) tservice = TService.TDataConnected; | |
| | else if (data_control_flag && !numbered && tpci == 0x80) tservice = TService.TConnect; | |
| | else if (data_control_flag && !numbered && tpci == 0x81) tservice = TService.TDisconnect; | |
| | else if (data_control_flag && numbered && (tpci & 0b11) == 0b10) tservice = TService.TAck; | |
| | else if (data_control_flag && numbered && (tpci & 0b11) == 0b11) tservice = TService.TNack; | |
| | } | |
| | } | |
| | public void calculateTpci() { | |
| | // calculate tpci ubyte from tservice and tseq | |
| | bool data_control_flag; | |
| | bool numbered; | |
| | ubyte last_bits = 0b00; | |
| | switch (tservice) { | |
| | case TService.TDataBroadcast: | |
| | case TService.TDataGroup: | |
| | case TService.TDataTagGroup: | |
| | case TService.TDataIndividual: | |
| | data_control_flag = false; | |
| | numbered = false; | |
| | break; | |
| | case TService.TDataConnected: | |
| | data_control_flag = false; | |
| | numbered = true; | |
| | break; | |
| | case TService.TConnect: | |
| | data_control_flag = true; | |
| | numbered = false; | |
| | last_bits = 0b00; | |
| | break; | |
| | case TService.TDisconnect: | |
| | data_control_flag = true; | |
| | numbered = false; | |
| | last_bits = 0b01; | |
| | break; | |
| | case TService.TAck: | |
| | data_control_flag = true; | |
| | numbered = true; | |
| | last_bits = 0b10; | |
| | break; | |
| | case TService.TNack: | |
| | data_control_flag = true; | |
| | numbered = true; | |
| | last_bits = 0b11; | |
| | break; | |
| | default: | |
| | break; | |
| | } | |
| | tpci = 0x00; | |
| | if (data_control_flag) tpci = tpci | 0b10000000; | |
| | if (numbered) { | |
| | tpci = tpci | 0b01000000; | |
| | tpci = tpci | ((tseq & 0b1111) << 2); | |
| | } | |
| | tpci = tpci | last_bits; | |
| | } | |
| | | |
| `---- | |
| If variable tservice and tseq values should be extracted from ubyte | |
| array, first method is used, if there is a need to generate ubyte | |
| array of whole cEMI frame, calculateTpci() method generates tpci | |
| variable value, which is put into generated cEMI frame. | |
| Let's proceed next in examining cEMI frame. If apci_data_length | |
| equals zero, then there is no apci and data, whole tpci_apci byte(last | |
| in frame) is tpci value. TConnect, TDisconnect, TAck, TNack. If | |
| apci_data_length > 0, there is APCI and data presented. Let's take a | |
| look in a class constructor, which creates object instance from ubyte | |
| array. | |
| ,---- | |
| | | |
| | ..... | |
| | ..... | |
| | // addresses | |
| | source = msg.peek!ushort(offset); offset += 2; | |
| | dest = msg.peek!ushort(offset); offset += 2; | |
| | apci_data_len = msg.peek!ubyte(offset); offset += 1; | |
| | tpci_apci = msg.peek!ubyte(offset); offset += 1; | |
| | apci_data = msg[offset..offset + apci_data_len].dup; | |
| | | |
| | if (apci_data_len == 0) { | |
| | // pure transport service message. TConnect/TDisconnect/TAck/TNack | |
| | tpci = tpci_apci; | |
| | data.length = 0; | |
| | } else if (apci_data_len == 1) { | |
| | tpci = tpci_apci & 0b11111100; | |
| | // 4bit apci - last two bits of tpci byte and first two of apci_data first byte | |
| | apci = cast(APCI) (((tpci_apci & 0b11) << 2) | ((apci_data[0] & 0b11000000) >> 6)); | |
| | tiny_data = apci_data[0] & 0b00111111; | |
| | data.length = 0; | |
| | } else if (apci_data_len > 1) { | |
| | tpci = tpci_apci & 0b11111100; | |
| | // 4bit apci - last two bits of tpci byte and first two of apci_data first byte | |
| | apci = cast(APCI) (((tpci_apci & 0b11) << 2) | ((apci_data[0] & 0b11000000) >> 6)); | |
| | data.length = apci_data_len - 1; | |
| | data[0..$] = apci_data[1..$]; | |
| | // cases like ADCRead/ADCResponse | |
| | // where data is encoded in next six bits | |
| | tiny_data = apci_data[0] & 0b111111; | |
| | } | |
| | if (apci == APCI.AUserMessage || apci == APCI.AEscape) { | |
| | apci = cast(APCI) ((apci << 6) | tiny_data); | |
| | tiny_data = 0; | |
| | } | |
| | getTransportServiceInfo(); | |
| | .... | |
| | .... | |
| `---- | |
| There is a need to explain tiny_data variable. Look, APCI is a 4bit | |
| value. The left six bits in apci_data byte may be used by | |
| data(e.g. binary DPT1 group address value was broadcasted). In this | |
| case tiny_data variable store this value. It may also contain | |
| parameters specific to application layer service. E.g. descriptor type | |
| for ADeviceDescriptorRead/Response or a restart type for ARestart. Or | |
| it may extend APCI value in case of base 4bit value is AUserMessage | |
| and AEscape. For detailed info, refer to "03_03_07 Application Layer | |
| v01.06.02 AS.pdf" [14]. | |
| I implemented support for following application layer services: | |
| ,---- | |
| | enum APCI: ushort { | |
| | AGroupValueRead = 0b0000, | |
| | AGroupValueResponse = 0b0001, | |
| | AGroupValueWrite = 0b0010, | |
| | AIndividualAddressWrite = 0b0011, | |
| | AIndividualAddressRead = 0b0100, | |
| | AIndividualAddressResponse = 0b0101, | |
| | AADCRead = 0b0110, | |
| | AADCResponse = 0b0111, | |
| | AMemoryRead = 0b1000, | |
| | AMemoryResponse = 0b1001, | |
| | AMemoryWrite = 0b1010, | |
| | AUserMessage = 0b1011, | |
| | AUserMemoryRead = 0b1011000000, | |
| | AUserMemoryResponse = 0b1011000001, | |
| | AUserMemoryWrite = 0b1011000010, | |
| | AUserManufacturerInfoRead = 0b1011000101, | |
| | AUserManufacturerInfoResponse = 0b1011000110, | |
| | ADeviceDescriptorRead = 0b1100, | |
| | ADeviceDescriptorResponse = 0b1101, | |
| | ARestart = 0b1110, | |
| | AEscape = 0b1111, | |
| | APropertyValueRead = 0b1111010101, | |
| | APropertyValueResponse = 0b1111010110, | |
| | APropertyValueWrite = 0b1111010111, | |
| | APropertyDescriptionRead = 0b1111011000, | |
| | APropertyDescriptionResponse = 0b1111011001 | |
| | } | |
| `---- | |
| Take a careful look at AUser* and AProperty* services. AUser* services | |
| share same first four bits (AUserMessage = 0b1011) as well as | |
| AProperty* (AEscape = 0b1111). | |
| 6.5 Transport connections | |
| ~~~~~~~~~~~~~~~~~~~~~~~~~ | |
| If you are familiar with TCP and UDP then it may be easier to | |
| understand. If you are not, then it may be easy too - there is nothing | |
| difficult. There may be established connection between two devices | |
| and there may be broadcasted messages(e.g. group address) to all | |
| parties. Also there is connectionless messages for individual | |
| addresses. Speaking of analogy: connectionless messages may be | |
| compared to UDP, group address to UDP Broadcast/Multicast and | |
| transport connection with TCP protocol. | |
| The difference lies in delivery control. If group address value was | |
| sent to bus it should be acknowledged by some of other device who has | |
| this group address written in memory table and if it was not (group | |
| address assigned only for one object) it will be repeat up to three | |
| times and that's all. There is no other mechanisms. Transport | |
| connection, in sight, has other mechanisms like timers(if no messages | |
| in 6 seconds or outgoing message was not acknowledged in 3 seconds) | |
| and sequence numbers. | |
| I'll focus here on transport connections. | |
| To establish one, device should send cEMI frame with address type | |
| individual, TConnect as a transport service, without any apci_data | |
| bytes. If this frame was confirmed on bus, then connection is | |
| established. If no any data was sent in following six seconds, then | |
| remote device will send TDisconnect request and will ignore following | |
| TConnectedData if no new TConnect request was sent. After connecting, | |
| one device may send frame with service TDataConnected. TPCI byte | |
| should contain sequence number from 0 to 15(0b0000 to 0b1111). On | |
| receiving data, remote device should send frame with TAck service, | |
| containing same sequence number as in request. If no TAck was | |
| received in three seconds, then first device shall send request up to | |
| three times and in case of absent/negative acknowledge, send | |
| TDisconnect and broke connection. Remote device shall send responses | |
| with increasing sequence counter too. | |
| As a reference, look at 03_03_04 [13]. | |
| 6.6 "Mistake" made | |
| ~~~~~~~~~~~~~~~~~~ | |
| I gave a summary of my knowledge of KNX transport layer now. But at a | |
| time of implementing IP interface I did know nothing about it. I have | |
| seen T_CONNECT/T_ACK messages in ETS Group Monitoring tool before, but | |
| didn't find it important to study. | |
| BAOS module ocuppies one individual address. KNXNet client sends some | |
| request and source address field is set to 0x0000. If KNXNetIP Server | |
| doesn't change it, BAOS module replace zero values with it's own | |
| address when sending to bus. After sending LData.req with source | |
| field of zero value, BAOS returns LData.con with source value equals | |
| of it's individual address. KNXNet Server then should "patch" it and | |
| send to Client who is awaiting confirmation with new source address | |
| value equals to individual address of selected tunneling connection. | |
| Also if destination field of any incoming cEMI frame is BAOS | |
| individual address, then Server should replace it to tunneling | |
| connection address. It works good for one connection. For multiple | |
| connections troubles has began. I work with ETS and Net N' Node to | |
| test my implementation. I started to scan network from both tools and | |
| Net N' Node broke connection with Kernel Error. The problem was in | |
| transport connections. ETS started to scan network first, sent | |
| ADeviceDescriptorRead request to first confirmed connection. At this | |
| time Net N' Node started scanning process as well. When requested by | |
| ETS ADeviceDescriptorResponse LData.Ind frame was received, it was | |
| broadcasted to all clients. So, for Net N' Node is was wrong message | |
| because it was not expecting TConnectedData at a time. If I have | |
| started scan process exactly at same time, it would fail too, because | |
| how then Server could control multiple transport connections? At | |
| first, remote device would answer only to first arrived TConnect | |
| request. Even if we assume this impossible case that two connections | |
| was established from BAOS to remote device, there would be mess with | |
| sequence numbers. | |
| So, to be suitable for this task, BAOS should occupy more than one | |
| individual address. | |
| Anyway, I started to dig deep into specifications. | |
| 6.7 TProxy | |
| ~~~~~~~~~~ | |
| I started to implement "Transport Connection Proxy". So, idea was | |
| simple. Each tunneling connection with unique address send request to | |
| bus directly if address type is group, and through proxy abstraction | |
| if target is individual and tpci indicates | |
| TService.TConnect/TDisconnect/TDataConnected/TAck/TNack. Proxy | |
| abstraction store two arrays - one for real transport connections, | |
| another for virtual(between KNXNet Client and proxy). Also proxy store | |
| two-dimensional array with relations between virtual and real | |
| connections. | |
| Main ideas: | |
| 1. If real connection is not established yet, send first TConnect | |
| request to BAOS. | |
| 2. If LDataInd TConnnect confirmation without error flag was received | |
| from BAOS module, put individual address into real connections. | |
| 3. If real connection is present in array and some virtual client send | |
| another TConnect request, send succesful confirmation, but request | |
| shall not be sent to BAOS module. Emulate established connection. | |
| 4. If TDisconnect request was received from virtual connection, but | |
| there is at least another client connected to same remote device, | |
| send succesful LDataCon to virtual client, but not to BAOS | |
| module. Emulate TDisconnect confirmation. | |
| 5. If TDisconnect request was received from virtual connection and | |
| this connection is the only one who have relation with transport | |
| connection, send frame to BAOS module. | |
| 6. If TDisconnect LDataInd was received from remote device, send this | |
| frame to all virtual clients connected with frame source device. | |
| 7. If data TDataConnected/TAck/TNack from virtual client was received, | |
| "patch" sequence number with appropriate for target transport | |
| connection, write this number to relation and expect incoming data | |
| from remote device with same num(assuming simple model: one apci | |
| request - one apci response). | |
| 8. If data inside connection was received(TDataConnected/TAck/TNack) | |
| from BAOS module, find a relation item with expected sequence | |
| number that equals one from data frame, "patch" this number and | |
| send to client. | |
| The main idea was simple - create some kind of hub, where two | |
| connected clients may use transport connections and for each of them | |
| it should look like a real one - only requested data is received and | |
| sequence numbers are increased as they supposed to by standard even | |
| when they send data one after one. | |
| So, the Relation abstraction I made was following: | |
| ,---- | |
| | struct Relation { | |
| | bool connected; // virtual client connected to transport connection | |
| | bool lconw; // virtual client is awaiting confirmation | |
| | bool tackw; // vc is awaiting acknowledge from remote party | |
| | bool tdatw; // vc is awaiting response from remote device | |
| | ubyte vseq; // vc is awaiting ack/nack/data with this sequence | |
| | | |
| | // store last sent cemi frame to determine to whom send LDataCon | |
| | LData_cEMI sent; | |
| | | |
| | void increaseSeq() { | |
| | if (vseq < 15) { | |
| | vseq += 1; | |
| | } else { | |
| | vseq = 0; | |
| | } | |
| | } | |
| | } | |
| `---- | |
| Transport connection: | |
| ,---- | |
| | // real transport connection | |
| | struct TransportConnection { | |
| | ubyte tseq; | |
| | | |
| | ushort addr; | |
| | bool connected; | |
| | | |
| | StopWatch timer; | |
| | | |
| | bool tack_received; | |
| | LData_cEMI[] queue; | |
| | | |
| | LData_cEMI processQueue() { | |
| | LData_cEMI res; | |
| | if (!tack_received) return res; | |
| | if (queue.length == 0) return res; | |
| | res = queue[0]; | |
| | if (queue.length == 1) | |
| | queue = []; | |
| | else | |
| | queue = queue[1..$]; | |
| | | |
| | return res; | |
| | } | |
| | | |
| | void increaseSeq() { | |
| | if (tseq < 15) { | |
| | tseq += 1; | |
| | } else { | |
| | tseq = 0; | |
| | } | |
| | } | |
| | } | |
| `---- | |
| Case: Client1 send Frame1 and right after this Client2 send Frame2. No | |
| TAck was received from remote party for Frame1 but Frame2 was sent! To | |
| exclude cases like this, queue is needed. | |
| Virtual connection: | |
| ,---- | |
| | // virtual transport connection | |
| | struct VirtualConnection { | |
| | ushort addr; | |
| | bool connected; | |
| | } | |
| `---- | |
| And there is arrays where everything is stored: | |
| ,---- | |
| | private TransportConnection[MAX_TCONNS] tconns; | |
| | private VirtualConnection[MAX_VCONNS] vconns; | |
| | private Relation[MAX_TCONNS][MAX_VCONNS] relations; | |
| `---- | |
| On each incoming/outgoig frame we lookup connection index, find it's | |
| relations, send frames, if needed, and change relation state. | |
| Testing reveals that sequence numbers was corrent, still, remote | |
| device started to response with some delay when second client | |
| connected. | |
| My idea failed, still, I shall take some time and look with a fresh | |
| mind again. | |
| 6.8 Finally | |
| ~~~~~~~~~~~ | |
| I returned back to KNXNet/IP Server implementation and done following: | |
| 6.8.1 dobaosll_pub | |
| ------------------ | |
| I split whole application into two parts: one is listening redis | |
| pub/sub channel and on incoming request send cemi frame to BAOS. | |
| Request is JSON string which contains "method" field(to send cemi - | |
| "cemi to bus") and "payload" - base64 encoded frame. | |
| Let's take a look what's going on when some new client connect to | |
| redis pubsub. redis-cli monitor output: | |
| ,---- | |
| | "PSUBSCRIBE" "ddllcli_*" | |
| | "SUBSCRIBE" "dobaosll_cast" | |
| | "PUBLISH" "dobaosll_req" "{\"method\":\"cemi to bus\",\"payload\":\"\\/AAAAVMQAQ==\",\"response_channel\":\"ddllcli_70\"}" | |
| | "PUBLISH" "ddllcli_70" "{\"method\":\"success\",\"payload\":true}" | |
| | "XADD" "dobaosll_stream" "MAXLEN" "~" "100000" "*" "payload" "FC000001531001" | |
| | "PUBLISH" "dobaosll_cast" "{\"method\":\"cemi from bus\",\"payload\":\"+wAAAVMQAQew\"}" | |
| | "XADD" "dobaosll_stream" "MAXLEN" "~" "100000" "*" "payload" "FB00000153100107B0" | |
| | "PUBLISH" "dobaosll_req" "{\"method\":\"cemi to bus\",\"payload\":\"EQC8YAAAEQMAgA==\",\"response_channel\":\"ddllcli_41\"}" | |
| | "PUBLISH" "ddllcli_41" "{\"method\":\"success\",\"payload\":true}" | |
| | "XADD" "dobaosll_stream" "MAXLEN" "~" "100000" "*" "payload" "1100BC60000011030080" | |
| | "PUBLISH" "dobaosll_cast" "{\"method\":\"cemi from bus\",\"payload\":\"LgC8YBHwEQMAgA==\"}" | |
| | "XADD" "dobaosll_stream" "MAXLEN" "~" "100000" "*" "payload" "2E00BC6011F011030080" | |
| | "PUBLISH" "dobaosll_req" "{\"method\":\"cemi to bus\",\"payload\":\"EQC8YAAAEQMBQwA=\",\"response_channel\":\"ddllcli_101\"}" | |
| | "PUBLISH" "ddllcli_101" "{\"method\":\"success\",\"payload\":true}" | |
| | "XADD" "dobaosll_stream" "MAXLEN" "~" "100000" "*" "payload" "1100BC6000001103014300" | |
| | "PUBLISH" "dobaosll_cast" "{\"method\":\"cemi from bus\",\"payload\":\"LgC8YBHwEQMBQwA=\"}" | |
| | "XADD" "dobaosll_stream" "MAXLEN" "~" "100000" "*" "payload" "2E00BC6011F01103014300" | |
| | "PUBLISH" "dobaosll_cast" "{\"method\":\"cemi from bus\",\"payload\":\"KQCwYBEDEfAAwg==\"}" | |
| | "XADD" "dobaosll_stream" "MAXLEN" "~" "100000" "*" "payload" "2900B060110311F000C2" | |
| | "PUBLISH" "dobaosll_cast" "{\"method\":\"cemi from bus\",\"payload\":\"KQC8YBEDEfADQ0AHAQ==\"}" | |
| | "XADD" "dobaosll_stream" "MAXLEN" "~" "100000" "*" "payload" "2900BC60110311F00343400701" | |
| | "PUBLISH" "dobaosll_req" "{\"method\":\"cemi to bus\",\"payload\":\"EQC8YAAAEQMAwg==\",\"response_channel\":\"ddllcli_209\"}" | |
| | "PUBLISH" "ddllcli_209" "{\"method\":\"success\",\"payload\":true}" | |
| `---- | |
| First, this client subscribes to "ddcli_*" pattern, what means that it | |
| will receive message, sent to any "ddcli_123" or "ddcli_azaz" channel. | |
| Next, client send JSON request with method "cemi to bus", payload | |
| field is base64 encoded cEMI frame and "response_channel" indicates | |
| channel, where client await response. dobaosll_pub sends result of | |
| operation back to "response_channel". It may indicates errors like | |
| wrong Base64 encoded string. XADD command indicates that this frame | |
| is going to be stored in redis database. Note: this feature is | |
| disabled by default. | |
| On receiving response from BAOS LinkLayer dobaosll_pub send JSON | |
| serialized object to pubsub channel "dobaosll_cast" with same fields, | |
| except "response_channel". Client application can receive now bus | |
| messages as well and will be able to work with cemi frames. | |
| 6.8.2 dobaosll_net | |
| ------------------ | |
| For KNXNet/IP Server implementation I programmed with following in | |
| mind: | |
| 1. Same individial address is assigned to each tunneling connection. | |
| 2. Each tunneling connection abstraction store associative array | |
| tconns, where key type is ushort(16bits) with remote device address | |
| and value field is timer. | |
| 3. If one client want to send | |
| TConnect/TDisconnect/TDataConnected/TAck/TNAck/ request, iterate | |
| over all clients and find if there is somebody already connected. | |
| 1) If somebody is connected and it's not this client don't send | |
| frame to BAOS, but send LDataCon with error back. Emulate | |
| rejecting. | |
| 2) If no timer for this key(remote device address) was found, send | |
| frame to BAOS. | |
| 3) If timer is found and is located in same connection as request | |
| sender, send frame to bus. Reset timer. | |
| 4. On receiving TConnect LDataCon frame with error flag set to false, | |
| put tconns[remote_addr] item to hash table. Lock connection. | |
| 5. On receiving TDisconnect LDataCon/LDataInd remove | |
| tconns[remote_addr] item from table. Release connection. | |
| 6. On receiving TDataConnected/TAck/TNack from remote device, send it | |
| only to client who has this remote address as a key in tconns | |
| array. Reset timer. | |
| 7. Check timers. If timeout of six seconds has passed, remove this | |
| timer from array. Free connection. | |
| ,---- | |
| | ,-------------. ,-------------. ,------------. ,---------. | |
| | |KNXNetClient1| |KNXNetClient2| |KNXNetServer| |LinkLayer| | |
| | `------+------' `------+------' `-----+------' `----+----' | |
| | | | | | | |
| | | ===========================| | | |
| | ========================= Client 1 is connecting ====================== | |
| | | ===========================| | | |
| | | | | | | |
| | | TConnectReq | | | |
| | |------------------------------------------>| | | |
| | | | | | | |
| | | | | TConnectReq | | |
| | | | | --------------->| | |
| | | | | | | |
| | | | | TConnectCon | | |
| | | | | <---------------| | |
| | | | | | | |
| | | TConnectCon confirmed | | | |
| | |<------------------------------------------| | | |
| | | | | | | |
| | | | | | | |
| | | ================================== | | |
| | ===================== Client 2 is trying to connect =================== | |
| | | ================================== | | |
| | | | | | | |
| | | | TConnectReq | | | |
| | | |---------------------->| | | |
| | | | | | | |
| | | |TConnectCon with error | | | |
| | | |<----------------------| | | |
| | | | | | | |
| | | | | | | |
| | | ===========================| | | |
| | ========================= Client1 is disconnecting====================== | |
| | | ===========================| | | |
| | | | | | | |
| | | TDisconnectReq | | | |
| | |------------------------------------------>| | | |
| | | | | | | |
| | | | | TDisconnectReq | | |
| | | | | --------------->| | |
| | | | | | | |
| | | | | TDisconnectCon | | |
| | | | | <---------------| | |
| | | | | | | |
| | | TDisconnectCon confirmed | | | |
| | |<------------------------------------------| | | |
| | | | | | | |
| | | | | | | |
| | | ============================| | | |
| | ======================= Client 2 can connect now ======================= | |
| | | ============================| | | |
| | | | | | | |
| | | | TConnectReq | | | |
| | | |---------------------->| | | |
| | | | | | | |
| | | | | TConnectReq | | |
| | | | | --------------->| | |
| | | | | | | |
| | | | | TConnectCon | | |
| | | | | <---------------| | |
| | | | | | | |
| | | | TConnectCon confirmed | | | |
| | | |<----------------------| | | |
| | ,------+------. ,------+------. ,-----+------. ,----+----. | |
| | |KNXNetClient1| |KNXNetClient2| |KNXNetServer| |LinkLayer| | |
| | `-------------' `-------------' `------------' `---------' | |
| | | |
| `---- | |
| Whole KNXNetConnection struct definition: | |
| ,---- | |
| | struct KnxNetConnection { | |
| | bool active = false; | |
| | Address addr; | |
| | ushort ia = 0x0000; | |
| | ubyte channel = 0x01; | |
| | ubyte sequence = 0x00; | |
| | ubyte outSequence = 0x00; | |
| | ubyte type = KNXConnTypes.TUNNEL_CONNECTION; | |
| | | |
| | // indicates that acknowledge for outgoing TUN_REQ | |
| | // received from net client | |
| | bool ackReceived = true; | |
| | int sentReqCount = 0; | |
| | | |
| | // store last outgoing tunneling request | |
| | // in case of error - send it again to net client | |
| | ubyte[] lastReq; | |
| | | |
| | // timeout watcher | |
| | StopWatch swCon; // 120s inactivity | |
| | StopWatch swAck; // 1s for request ack | |
| | | |
| | // timers for transport connections | |
| | StopWatch[ushort] tconns; | |
| | | |
| | // last data sent to baos. | |
| | // purpose is to compare when Con frame received | |
| | ubyte[] lastCemiToBaos; | |
| | | |
| | // sequence id that we check for incoming net reqs | |
| | void increaseSeqId() { | |
| | if (sequence < 255) { | |
| | sequence += 1; | |
| | } else { | |
| | sequence = 0; | |
| | } | |
| | } | |
| | // outgoing sequence id | |
| | void increaseOutSeqId() { | |
| | if (sequence < 255) { | |
| | outSequence += 1; | |
| | } else { | |
| | outSequence = 0; | |
| | } | |
| | } | |
| | | |
| | // queue of knx net messages | |
| | // we don't want to send message | |
| | // if no ack for previous was received | |
| | KnxNetRequest[] queue; | |
| | ubyte[] processQueue() { | |
| | ubyte[] res; | |
| | if (ackReceived) { | |
| | if (queue.length > 0) { | |
| | KnxNetRequest item = queue[0]; | |
| | if (queue.length > 1) { | |
| | queue = queue[1..$]; | |
| | } else { | |
| | queue = []; | |
| | queue.length = 0; | |
| | } | |
| | ubyte[] frame = request(item.cemi, channel, outSequence); | |
| | ubyte[] req = KNXIPMessage(item.service, frame); | |
| | | |
| | return req; | |
| | } | |
| | } | |
| | | |
| | return res; | |
| | } | |
| | void add2queue(ushort service, ubyte[] cemi) { | |
| | queue ~= KnxNetRequest(service, cemi); | |
| | } | |
| | } | |
| `---- | |
| If some KNXNetClient want to send connection-oriented frame to bus, | |
| ,---- | |
| | ubyte mc = cemiFrame.peek!ubyte(offset); offset += 1; | |
| | bool sendToBaos = true; | |
| | LData_cEMI decoded; | |
| | if (mc == MC.LDATA_REQ || | |
| | mc == MC.LDATA_CON || mc == MC.LDATA_IND) { | |
| | | |
| | decoded = new LData_cEMI(cemiFrame); | |
| | | |
| | // check if address type is individual address, not group | |
| | // and tpci indicates transport connection messages | |
| | if (!decoded.address_type_group && | |
| | (decoded.tservice == TService.TConnect || | |
| | decoded.tservice == TService.TDataConnected || | |
| | decoded.tservice == TService.TAck || | |
| | decoded.tservice == TService.TNack || | |
| | decoded.tservice == TService.TDisconnect)) { | |
| | | |
| | ushort dest = decoded.dest; | |
| | | |
| | // for connection-oriented data check at first if | |
| | // transport connection was established for this client | |
| | bool somebodyElseConnected = false; | |
| | | |
| | // iterate over all KNXNetConnection[] connections | |
| | for(int i = 0; i < connections.length; i += 1) { | |
| | // if no key with frame destination address is found | |
| | // try to search | |
| | if ((dest in connections[i].tconns) is null) { | |
| | continue; | |
| | } | |
| | | |
| | // key is found in hash | |
| | // check if request is from different knxnet conn | |
| | somebodyElseConnected = (i != chIndex); | |
| | | |
| | // if nobody else connected, only request sender | |
| | // reset timer | |
| | if (i == chIndex) { | |
| | connections[i].tconns[dest].reset(); | |
| | } | |
| | } | |
| | if (somebodyElseConnected) { | |
| | // acknowledge receiving request, send back to client | |
| | auto ackFrame = ack(chId, seqId); | |
| | sendKNXIPMessage(resService, ackFrame, s, from); | |
| | | |
| | // send LDataCon with error back to client | |
| | decoded.message_code = MC.LDATA_CON; | |
| | decoded.source = tunIa; | |
| | decoded.error = true; | |
| | connections[chIndex].add2queue( | |
| | KNXServices.TUNNELING_REQUEST, decoded.toUbytes()); | |
| | queue2socket(chIndex); | |
| | // erase info about last sent cemi data | |
| | connections[chIndex].lastCemiToBaos = []; | |
| | // increase sequence number of connection | |
| | connections[chIndex].increaseSeqId(); | |
| | // reset timeout stopwatch | |
| | connections[chIndex].swCon.reset(); | |
| | connections[chIndex].swCon.start(); | |
| | | |
| | // do not run next part of code in | |
| | // current switch-case scope | |
| | break; | |
| | } | |
| | } | |
| | } | |
| | | |
| | // if somebody else is connected, following code won't run | |
| | | |
| | // acknowledge receiving request, send back to client | |
| | auto ackFrame = ack(chId, seqId); | |
| | sendKNXIPMessage(resService, ackFrame, s, from); | |
| | | |
| | writeln("-------------------------------"); | |
| | writeln("Sending to BAOS: ", cemiFrame.toHexString()); | |
| | if (mc == MC.LDATA_REQ || mc == MC.LDATA_CON || mc == MC.LDATA_IND) { | |
| | string destStr = decoded.address_type_group? | |
| | grpNum2str(decoded.dest): iaNum2str(decoded.dest); | |
| | writefln("mc: %s, source: %s, dest: %s, \ntservice: %s, tseq: %s, \napci: %s, tiny_data: %s, data: %s", | |
| | decoded.message_code, iaNum2str(decoded.source), destStr, decoded.tservice, decoded.tseq, | |
| | decoded.apci, decoded.tiny_data, decoded.data.toHexString()); | |
| | } | |
| | // send cemi to BAOS module | |
| | dobaosll.sendCemi(cemiFrame); | |
| | // store last sent frame | |
| | connections[chIndex].lastCemiToBaos = cemiFrame.dup; | |
| | | |
| | // increase sequence number of connection | |
| | connections[chIndex].increaseSeqId(); | |
| | // reset timeout stopwatch | |
| | connections[chIndex].swCon.reset(); | |
| | connections[chIndex].swCon.start(); | |
| | | |
| `---- | |
| Handle frames coming from BAOS: | |
| ,---- | |
| | if (mc == MC.LDATA_CON && | |
| | conn.type == KNXConnTypes.TUNNEL_CONNECTION) { | |
| | if (connections[i].lastCemiToBaos.length == 0) continue; | |
| | LData_cEMI last = new LData_cEMI(connections[i].lastCemiToBaos); | |
| | if (last.dest == parsed.dest && | |
| | last.tservice == parsed.tservice && | |
| | last.tseq == parsed.tseq && | |
| | last.apci == parsed.apci && | |
| | last.tiny_data == parsed.tiny_data && | |
| | last.data == parsed.data) { | |
| | // "patch" addresses | |
| | if (parsed.source == realIa) { | |
| | parsed.source = connections[i].ia; | |
| | } | |
| | // connection is pending LData.con message, send | |
| | connections[i].add2queue(KNXServices.TUNNELING_REQUEST, parsed.toUbytes()); | |
| | queue2socket(i); | |
| | // if message was LDataCon for TConnect service | |
| | // without confirmation error, then establish connection | |
| | if (parsed.tservice == TService.TConnect && | |
| | !parsed.error) { | |
| | connections[i].tconns[parsed.dest] = StopWatch(AutoStart.yes); | |
| | } | |
| | if (parsed.tservice == TService.TDisconnect && | |
| | !parsed.error) { | |
| | // confirmation for disconnect that was initialized by knx net client | |
| | connections[i].tconns.remove(parsed.dest); | |
| | } | |
| | } else if (parsed.address_type_group) { | |
| | // connection is not pending LData.con message | |
| | // change it to LData.ind and send | |
| | // BUT only if addressType indicating group address (== 0b1); | |
| | parsed.message_code = MC.LDATA_IND; | |
| | // "patch" addresses | |
| | if (parsed.source == realIa) { | |
| | parsed.source = connections[i].ia; | |
| | } | |
| | connections[i].add2queue(KNXServices.TUNNELING_REQUEST, parsed.toUbytes()); | |
| | queue2socket(i); | |
| | // return to LDATA_CON, so, next conn[i] iterations will check correctly | |
| | parsed.message_code = MC.LDATA_CON; | |
| | // erase info about last sent cemi data | |
| | connections[i].lastCemiToBaos = []; | |
| | } | |
| | } else if (mc == MC.LDATA_IND && | |
| | conn.type == KNXConnTypes.TUNNEL_CONNECTION) { | |
| | if (parsed.address_type_group) { | |
| | connections[i].add2queue(KNXServices.TUNNELING_REQUEST , cemi); | |
| | queue2socket(i); | |
| | } else { | |
| | if (parsed.tservice == TService.TDisconnect || | |
| | parsed.tservice == TService.TAck || | |
| | parsed.tservice == TService.TNack || | |
| | parsed.tservice == TService.TDataConnected) { | |
| | if ((parsed.source in connections[i].tconns) !is null) { | |
| | // if this transport connection exist in | |
| | // tconns array of this KnxNetConnection instance | |
| | // send tunneling request and reset timer | |
| | connections[i].add2queue(KNXServices.TUNNELING_REQUEST , cemi); | |
| | queue2socket(i); | |
| | connections[i].tconns[parsed.source].reset(); | |
| | | |
| | if (parsed.tservice == TService.TDisconnect) { | |
| | // disconnect sent by remote device | |
| | connections[i].tconns.remove(parsed.source); | |
| | } | |
| | } | |
| | } | |
| | } | |
| | } | |
| `---- | |
| I put explanations in commentaries for each part of code. Everything | |
| worked fine, group addresses written and read, device application | |
| downloaded, diagnostic tools work as well. | |
| 6.9 Scanner | |
| ~~~~~~~~~~~ | |
| After getting familiar with cEMI frames and transport layer I started | |
| to study what is going on while running ETS or Net N' Node diagnostic | |
| tools: restart device, put into programming mode, scan line. | |
| There is different application layer requests and responses. | |
| Transport Connection abstraction is written: | |
| ,---- | |
| | class TransportConnection { | |
| | private DobaosllClient ll; | |
| | private LData_cEMI in_frame, out_frame; | |
| | private ubyte in_seq, out_seq; | |
| | private bool check_in_seq = false; | |
| | | |
| | public ushort address; | |
| | public bool connected; | |
| | | |
| | this(ushort ia) { | |
| | address = ia; | |
| | ll = new DobaosllClient(); | |
| | ll.onCemi(toDelegate(&onCemiFrame)); | |
| | } | |
| | .... | |
| | .... | |
| | .... | |
| | public void connect() { | |
| | LData_cEMI tconn = new LData_cEMI(); | |
| | tconn.message_code = MC.LDATA_REQ; | |
| | tconn.address_type_group = false; | |
| | tconn.source = 0x0000; | |
| | tconn.dest = address; | |
| | tconn.tservice = TService.TConnect; | |
| | ll.sendCemi(tconn.toUbytes()); | |
| | bool confirmed = false; | |
| | LData_cEMI con; | |
| | while (!confirmed) { | |
| | try { | |
| | con = waitForCon(tconn); | |
| | confirmed = true; | |
| | } catch(Exception e) { | |
| | ll.sendCemi(tconn.toUbytes()); | |
| | } | |
| | } | |
| | if (con.error) { | |
| | connected = false; | |
| | return; | |
| | } | |
| | connected = true; | |
| | in_seq = 0x00; | |
| | out_seq = 0x00; | |
| | } | |
| | .... | |
| | .... | |
| | } | |
| `---- | |
| Without this abstraction more code is needed to process transport | |
| messages. In order to perform some action and get result it is needed | |
| to: | |
| 1. Send LData.Req TDataConnected frame. | |
| 2. Wait for LData.Con for this request. | |
| 3. Wait for TAck from remote device. | |
| 4. Wait for TDataConnected from remote device. | |
| So, this commong request-response model may be implemented in | |
| following way: | |
| ,---- | |
| | private LData_cEMI requestResponse(LData_cEMI req, APCI apci_res) { | |
| | if (!connected) { | |
| | throw new Exception("ERR_DISCONNECTED"); | |
| | } | |
| | LData_cEMI result; | |
| | ll.sendCemi(req.toUbytes()); | |
| | bool confirmed = false; | |
| | // wait for LDataCon frame | |
| | while(!confirmed) { | |
| | try { | |
| | waitForCon(req); | |
| | confirmed = true; | |
| | } catch(Exception e) { | |
| | ll.sendCemi(req.toUbytes()); | |
| | } | |
| | } | |
| | | |
| | auto ack_timeout_dur = 3000.msecs; | |
| | auto sent_cnt = 1; | |
| | bool acknowledged = false; | |
| | // wait for TAck. on timeout repeat up to three times | |
| | while (!acknowledged && sent_cnt < 4) { | |
| | try { | |
| | waitForAck(req, ack_timeout_dur); | |
| | acknowledged = true; | |
| | increaseOutSeq(); | |
| | } catch(Exception e) { | |
| | ll.sendCemi(req.toUbytes()); | |
| | sent_cnt += 1; | |
| | } | |
| | } | |
| | if (!acknowledged) { | |
| | disconnect(); | |
| | throw new Exception(ERR_ACK_TIMEOUT); | |
| | } | |
| | | |
| | // now wait for response with given apci | |
| | try { | |
| | result = waitForResponse(req, apci_res); | |
| | if (check_in_seq && result.tseq == in_seq) { | |
| | sendAck(); | |
| | } else if (!check_in_seq) { | |
| | sendAck(); | |
| | } else if (check_in_seq && result.tseq != in_seq){ | |
| | disconnect(); | |
| | throw new Exception("ERR_WRONG_SEQ_NUM"); | |
| | } | |
| | } catch(Exception e) { | |
| | disconnect(); | |
| | throw e; | |
| | } | |
| | | |
| | return result; | |
| | } | |
| `---- | |
| With this method written, others are pretty easy to implement now: | |
| ,---- | |
| | public ubyte[] deviceDescriptorRead(ubyte descr_type = 0x00) { | |
| | ubyte[] result; | |
| | LData_cEMI dread = new LData_cEMI(); | |
| | dread.message_code = MC.LDATA_REQ; | |
| | dread.address_type_group = false; | |
| | dread.source = 0x0000; | |
| | dread.dest = address; | |
| | dread.tservice = TService.TDataConnected; | |
| | dread.tseq = out_seq; | |
| | dread.apci = APCI.ADeviceDescriptorRead; | |
| | dread.apci_data_len = 1; | |
| | dread.tiny_data = descr_type; | |
| | | |
| | result = requestResponse(dread, APCI.ADeviceDescriptorResponse).data; | |
| | | |
| | return result; | |
| | } | |
| | public ubyte[] propertyRead(ubyte obj_id, ubyte prop_id, ubyte num, ushort start) { | |
| | ubyte[] result; | |
| | // get serial number | |
| | LData_cEMI dprop = new LData_cEMI(); | |
| | dprop.message_code = MC.LDATA_REQ; | |
| | dprop.address_type_group = false; | |
| | dprop.source = 0x0000; | |
| | dprop.dest = address; | |
| | dprop.tservice = TService.TDataConnected; | |
| | dprop.tseq = out_seq; | |
| | dprop.apci = APCI.APropertyValueRead; | |
| | dprop.apci_data_len = 5; | |
| | dprop.data.length = 4; | |
| | dprop.data.write!ubyte(obj_id, 0); | |
| | dprop.data.write!ubyte(prop_id, 1); | |
| | ushort numstart = start & 0b000011111111; | |
| | numstart = to!ushort((num << 12) | numstart); | |
| | dprop.data.write!ushort(numstart, 2); | |
| | ll.sendCemi(dprop.toUbytes()); | |
| | result = requestResponse(dprop, APCI.APropertyValueResponse).data; | |
| | | |
| | if(result.length >= 4) { | |
| | ubyte res_obj_id = result.read!ubyte(); | |
| | ubyte res_prop_id = result.read!ubyte(); | |
| | ushort res_numstart = result.read!ushort(); | |
| | if (res_obj_id != obj_id && | |
| | res_prop_id != prop_id && | |
| | res_numstart != numstart) { | |
| | throw new Exception("ERR_WRONG_RESPONSE"); | |
| | } | |
| | | |
| | } else { | |
| | throw new Exception("ERR_WRONG_RESPONSE"); | |
| | } | |
| | | |
| | return result; | |
| | } | |
| `---- | |
| With this abstraction, scanner main loop is implemented in less than | |
| hundred lines of code. | |
| ,---- | |
| | for(ubyte ia = 1; ia < 255; ia += 1) { | |
| | Thread.sleep(50.msecs); | |
| | ushort addr = sub*256 + ia; | |
| | if (addr == local_ia) { | |
| | writefln("Local\t%s\t[Descr: %s. SN: %s.]", | |
| | ia2str(local_ia), toHexString(local_descr), toHexString(local_sn)); | |
| | continue; | |
| | } | |
| | // create TransportConnection instance, try to connect | |
| | auto tc = new TransportConnection(addr); | |
| | tc.connect(); | |
| | if (!tc.connected) { | |
| | continue; | |
| | } | |
| | // if connected | |
| | ubyte[] descr; | |
| | ubyte[] serial; | |
| | ubyte[] manufacturer; | |
| | string name = " --- "; | |
| | try { | |
| | descr = tc.deviceDescriptorRead(); | |
| | Thread.sleep(50.msecs); | |
| | } catch(Exception e) { | |
| | // silently catch error | |
| | } | |
| | try { | |
| | // manufacturer code: Obj 0, PID 12, num 1, start 01 | |
| | manufacturer = tc.propertyRead(0x00, 0x0c, 0x01, 0x01); | |
| | } catch(Exception e) { | |
| | // silently catch error | |
| | } | |
| | try { | |
| | // serialnum: Obj 0, PID 11, num 1, start 1 | |
| | serial = tc.propertyRead(0x00, 0x0b, 0x01, 0x01); | |
| | Thread.sleep(50.msecs); | |
| | } catch(Exception e) { | |
| | // silently catch error | |
| | } | |
| | try { | |
| | // description string: Obj 0, PID 21, num 1, start 0 | |
| | ubyte[] name_len_raw = tc.propertyRead(0x00, 0x15, 0x01, 0x00); | |
| | if (name_len_raw.length == 2) { | |
| | ushort name_len = name_len_raw.read!ushort(); | |
| | if (name_len > 0) { | |
| | name = ""; | |
| | // restrict maximum name length to 10 symbols | |
| | // Net N' node receives it chunk by chunk, chunk len is 10 | |
| | ushort bytes_left = name_len; | |
| | ubyte start = 0x01; | |
| | string getChunk() { | |
| | string res; | |
| | if (bytes_left > 10) { | |
| | char[] chars = cast(char[])(tc.propertyRead(0x00, 0x15, 0x0a, start)); | |
| | bytes_left = to!ushort(bytes_left - 10); | |
| | start += 10; | |
| | res = to!string(chars); | |
| | } else { | |
| | char[] chars = cast(char[])(tc.propertyRead(0x00, 0x15, | |
| | to!ubyte(bytes_left), start)); | |
| | bytes_left = 0; | |
| | res = to!string(chars); | |
| | } | |
| | | |
| | return res; | |
| | } | |
| | while(bytes_left > 0) { | |
| | name = name ~ getChunk(); | |
| | Thread.sleep(50.msecs); | |
| | } | |
| | } | |
| | Thread.sleep(50.msecs); | |
| | } | |
| | } catch(Exception e) { | |
| | // silently catch error | |
| | } finally { | |
| | tc.disconnect(); | |
| | writefln("Device\t%s\t[Descr: %s. SN: %s. Manuf-r: %s. Name: %s.]", | |
| | ia2str(addr), toHexString(descr), | |
| | toHexString(serial), toHexString(manufacturer), name); | |
| | } | |
| | } | |
| `---- | |
| This scanner is doing the same as "scan line" tool in Net N' Node: | |
| 1. Try to connect to device A.L.IA, where A.L is fixed area and line | |
| values, IA - from 1 to 255, except BAOS own address. | |
| 2. If not connected(LData.Con with error), increase IA value, start | |
| with step 1. | |
| 3. Try to read device descriptor. | |
| 4. Try to read device serial number. | |
| 5. Try to read device name string. | |
| Application output for my tiny home installation: | |
| ,---- | |
| | hello, friend: ["./dobaosll_scan"] | |
| | Scanning line 1.1 | |
| | Device 1.1.3 [Descr: 0701. SN: 00710001AF9A. Manuf-r: 0071. Name: --- .] | |
| | Local 1.1.240 [Descr: 07B0. SN: 00C50101F168.] | |
| | Device 1.1.250 [Descr: . SN: . Manuf-r: . Name: --- .] | |
| | bye, friend | |
| `---- | |
| There no response for ADeviceDescriptiorRead and APropertyValueRead | |
| from device 1.1.250(CI-KNX). Zennio QUAD don't return it's name. There | |
| was another BAOS module in my network and everything returned fine - | |
| descriptor, serial number and name. | |
| 6.10 Other utils | |
| ~~~~~~~~~~~~~~~~ | |
| 6.10.1 mprop_reader | |
| ------------------- | |
| To read local BAOS properties, device management requests may be sent | |
| to bus. Following code exists as a module in KNXNet Server | |
| implementation and in ll_scanner to read BAOS individual address and | |
| serial number. | |
| ,---- | |
| | import core.thread; | |
| | import std.algorithm: equal; | |
| | import std.bitmanip; | |
| | import std.conv; | |
| | import std.datetime.stopwatch; | |
| | import std.functional; | |
| | | |
| | import dobaosll_client; | |
| | import knx; | |
| | import redis_abstractions; | |
| | | |
| | class MPropReader { | |
| | private DobaosllClient dobaosll; | |
| | // global variables | |
| | private ubyte[] lastRequest; | |
| | private ubyte[] responseData; | |
| | private bool resolved = false; | |
| | | |
| | this() { | |
| | dobaosll = new DobaosllClient(); | |
| | void onCemiFrame(ubyte[] cemi) { | |
| | int offset = 0; | |
| | ubyte mc = cemi.peek!ubyte(offset); offset += 1; | |
| | if (mc == cEMI_MC.MPROPREAD_CON) { | |
| | auto e = lastRequest.length; | |
| | if (e > 0 && cemi.length > e) { | |
| | // MPROPxx.CON is basically the same as request message | |
| | // only data bytes added to the end of message | |
| | if (equal(lastRequest[1..e], cemi[1..e])) { | |
| | resolved = true; | |
| | responseData = cemi[e..$].dup; | |
| | } | |
| | } | |
| | } | |
| | } | |
| | dobaosll.onCemi(toDelegate(&onCemiFrame)); | |
| | | |
| | } | |
| | public ubyte[] read(ubyte id, int num = 1, | |
| | ushort si = 0x0001, Duration time = 1000.msecs) { | |
| | ubyte[] request; | |
| | request.length = 7; | |
| | request.write!ubyte(cEMI_MC.MPROPREAD_REQ, 0); | |
| | request.write!ushort(0, 1); // interface object type | |
| | request.write!ubyte(1, 3); // object instance | |
| | request.write!ubyte(id, 4); // property id | |
| | ushort noeSix = to!ushort(num << 12 | (si & 0b111111111111)); | |
| | request.write!ushort(noeSix, 5); | |
| | | |
| | lastRequest = request.dup; | |
| | dobaosll.sendCemi(request); | |
| | bool timeout = false; | |
| | StopWatch sw = StopWatch(AutoStart.yes); | |
| | while (!resolved && !timeout) { | |
| | timeout = sw.peek() > time; | |
| | dobaosll.processMessages(); | |
| | Thread.sleep(1.msecs); | |
| | } | |
| | if (timeout) { | |
| | throw new Exception("ERR_TIMEOUT"); | |
| | } | |
| | sw.stop(); | |
| | auto result = responseData.dup; | |
| | lastRequest = []; | |
| | responseData = []; | |
| | resolved = false; | |
| | | |
| | return result; | |
| | } | |
| | } | |
| `---- | |
| It work for existing properties cause it relies on that fact, that | |
| successful response ubytes is the same as in request, only with data | |
| bytes added in the end. But if response contain error, it may differs | |
| in number of returned objects, so, it won't be resolved and with this | |
| code we won't get error code. Therefore, it may be done better - | |
| management response parsed(similar way as LData cemi frame) and result | |
| given as a structure. | |
| 6.10.2 stream converter | |
| ----------------------- | |
| I mentioned Redis streams before. XADD command is used to put value | |
| marked with timestamp to store. In order to get stream values, XRANGE | |
| command may be sent to redis server. Still, there is a need to get | |
| certain number of last frames, XREVRANGE command is issued. | |
| dobaosll_pub send every frame that was sent to BAOS or received to | |
| this stream. In order to study and debug, ETS monitoring tool may be | |
| used. It may generate xml file with current telegram list as well as | |
| import existing file. So, to export values I wrote simple script for | |
| nodejs: | |
| ,---- | |
| | const fs = require("fs"); | |
| | const redis = require("redis"); | |
| | | |
| | redis.add_command("xrange") | |
| | redis.add_command("xrevrange") | |
| | | |
| | let stream = "dobaosll_stream"; | |
| | let count = "1000"; | |
| | | |
| | const client = redis.createClient(); | |
| | | |
| | let bunchOfFrames = []; | |
| | | |
| | let processed = 0; | |
| | let xr = client.xrevrange(stream, "+", "-","COUNT", count, (err, res) => { | |
| | if (err) { | |
| | console.log(err); | |
| | return; | |
| | } | |
| | | |
| | if (!Array.isArray(res)) return; | |
| | | |
| | res.forEach(r => { | |
| | // [ '1585853964332-0', [ 'payload', 'AABBCCDD - hex string' ] ], | |
| | let tsStr = r[0]; | |
| | // 1585854932818-0 | |
| | let ts = parseInt(tsStr.split("-")[0]); | |
| | let payload = r[1][1]; | |
| | bunchOfFrames.push([ts, payload]); | |
| | }); | |
| | | |
| | // sort by timestamp | |
| | let sortedFrames = bunchOfFrames.sort((a, b) => { | |
| | return a[0] - b[0]; | |
| | }); | |
| | // now convert timestamp to json time | |
| | let fmtFrames = sortedFrames.map(sf => { | |
| | return [ JSON.stringify(new Date(sf[0])), Buffer.from(sf[1], "hex") ]; | |
| | }); | |
| | // generate xml | |
| | let xml = ""; | |
| | xml += '<CommunicationLog xmlns="http://knx.org/xml/telegrams/01">\r\n'; | |
| | xml += `<RecordStart Timestamp=${fmtFrames[0][0]} `; | |
| | xml += `Mode="LinkLayer" Host="DESKTOP-CS80CLE" `; | |
| | xml += `ConnectionName="dobaosll" `; | |
| | xml += `ConnectionOptions="Type=KnxIpTunneling;`; | |
| | xml += `HostAddress=192.168.1.51;SerialNumber=1111:11111111;`; | |
| | xml += `UseNat=True;Name=dobaosll" ConnectorType="KnxIpTunneling" />\r\n`; | |
| | | |
| | fmtFrames.forEach(ff => { | |
| | let mc = ff[1][0]; | |
| | if (!(mc == 0x11 || mc == 0x29 || mc == 0x2E)) return; | |
| | | |
| | let service = ""; | |
| | switch (mc) { | |
| | case 0x11: | |
| | service = "L_Data.req"; | |
| | break; | |
| | case 0x29: | |
| | service = "L_Data.ind"; | |
| | break; | |
| | case 0x2E: | |
| | service = "L_Data.con"; | |
| | break; | |
| | default: | |
| | break; | |
| | } | |
| | xml += `<Telegram Timestamp=${ff[0]} Service=${service} `; | |
| | xml += `FrameFormat="CommonEmi" RawData="${ff[1].toString("hex")}" />\r\n`; | |
| | }); | |
| | | |
| | | |
| | xml += `<RecordStop Timestamp=${fmtFrames[fmtFrames.length - 1][0]} />\r\n`; | |
| | xml += '</CommunicationLog>\r\n'; | |
| | console.log(xml); | |
| | fs.writeFileSync('result.xml', xml, 'utf8'); | |
| | console.log("file written"); | |
| | process.exit(); | |
| | }); | |
| `---- | |
| There is a lot of specific javascript things in code. Still, code does | |
| following: | |
| 1. Get last 1000 items of stream by using XREVRANGE command. | |
| 2. Sort result by timestamp. | |
| 3. Generate xml: | |
| 1. Start with header. | |
| 2. Process each frame: if LData*, add to xml. | |
| 3. Add footer. | |
| So, xml with following data is generated: | |
| ,---- | |
| | <CommunicationLog xmlns="http://knx.org/xml/telegrams/01"> | |
| | <RecordStart Timestamp="2020-05-04T19:57:22.658Z" Mode="LinkLayer" Host="DESKTOP-CS80CLE" | |
| | ConnectionName="dobaosll" ConnectionOptions="Type=KnxIpTunneling;HostAddress=192.168.1.51; | |
| | SerialNumber=1111:11111111;UseNat=True;Name=dobaosll" ConnectorType="KnxIpTunneling" /> | |
| | <Telegram Timestamp="2020-05-04T19:57:24.166Z" Service=L_Data.con FrameFormat="CommonEmi" RawData="2e009d6011f011540080" /> | |
| | <Telegram Timestamp="2020-05-04T19:57:24.240Z" Service=L_Data.req FrameFormat="CommonEmi" RawData="1100bc60000011550080" /> | |
| | ..... | |
| | <Telegram Timestamp="2020-05-04T19:59:08.747Z" Service=L_Data.req FrameFormat="CommonEmi" RawData="1100bc60000011fa014300" /> | |
| | <Telegram Timestamp="2020-05-04T19:59:08.770Z" Service=L_Data.req FrameFormat="CommonEmi" RawData="1100bc60000011fa0081" /> | |
| | <Telegram Timestamp="2020-05-04T19:59:08.794Z" Service=L_Data.con FrameFormat="CommonEmi" RawData="2e00bc6011f011fa014300" /> | |
| | <Telegram Timestamp="2020-05-04T19:59:08.802Z" Service=L_Data.req FrameFormat="CommonEmi" RawData="1100bc60000011fa0543d5000c1001" /> | |
| | <Telegram Timestamp="2020-05-04T19:59:08.824Z" Service=L_Data.con FrameFormat="CommonEmi" RawData="2e00bc6011f011fa0081" /> | |
| | <Telegram Timestamp="2020-05-04T19:59:08.827Z" Service=L_Data.req FrameFormat="CommonEmi" RawData="1100bc60000011fa0543d5000b1001" /> | |
| | <Telegram Timestamp="2020-05-04T19:59:08.842Z" Service=L_Data.ind FrameFormat="CommonEmi" RawData="2900b06011fa11f00081" /> | |
| | <Telegram Timestamp="2020-05-04T19:59:08.850Z" Service=L_Data.req FrameFormat="CommonEmi" RawData="1100bc60000011fa0543d500151000" /> | |
| | <Telegram Timestamp="2020-05-04T19:59:08.869Z" Service=L_Data.con FrameFormat="CommonEmi" RawData="2e00bc6011f011fa0543d5000c1001" /> | |
| | <Telegram Timestamp="2020-05-04T19:59:08.873Z" Service=L_Data.req FrameFormat="CommonEmi" RawData="1100bc60000011fa0081" /> | |
| | .... | |
| | ... | |
| | <Telegram Timestamp="2020-05-04T20:23:17.179Z" Service=L_Data.ind FrameFormat="CommonEmi" RawData="2900bce0110309010300800c7e" /> | |
| | <RecordStop Timestamp="2020-05-04T20:23:17.179Z" /> | |
| | </CommunicationLog> | |
| `---- | |
| Now this xml file can be imported into ETS group monitoring and KNX | |
| messages studied. This feature helped me to debug scanner tool, when I | |
| communicated with bus not as KNXNet client. | |
| 7 References | |
| ============ | |
| 1. [https://weinzierl.de/images/download/documents/baos/KNX_BAOS_Binary_Protocol_V2_0.pdf] | |
| 2. [https://www.npmjs.com/package/serialport] | |
| 3. [https://redis.io/clients] | |
| 4. [https://redis.io/topics/streams-intro] | |
| 5. "03_07_02 Datapoint Types v01.08.02 AS.pdf" from "The KNX Standard | |
| v2.1" | |
| 6. [https://github.com/Rafelder/knx-datapoints] | |
| 7. "03_03_02 Data Link Layer General v01.02.02 AS.pdf" from "The KNX | |
| Standard v2.1" | |
| 8. "03_05_01 Resources v01.09.03 AS.pdf" from "The KNX Standard v2.1" | |
| 9. "03_06_03 EMI_IMI v01.03.03 AS.pdf" from "The KNX Standard v2.1" | |
| 10. "03_08_02 Core v01.05.01 AS.pdf" from "The KNX Standard v2.1" | |
| 11. "03_08_04 Tunnelling v01.05.03 AS.pdf" from "The KNX Standard | |
| v2.1" | |
| 12. [https://www.dehof.de/eib/pdfs/EMI-FT12-Message-Format.pdf] | |
| 13. "03_03_04 Transport Layer v01.02.02 AS.pdf" from "The KNX Standard | |
| v2.1" | |
| 14. "03_03_07 Application Layer v01.06.02 AS.pdf" from "The KNX | |
| Standard v2.1" |
I am in pain of creating a library. Your documents seem to be able to help me. thank you for sharing!
Hi, there
I just described my experience of working with knx specs - it all is there, but hard to compile bit by bit.
What kind of library you're doing?
Looks like you have put plenty of effort in to transfer your understanding. I am working on creating a developer friendly library for easily creating KNX end devices but I have stumbled in load procedures in Manufacturer Tool and their telegrams.
Hi, I have seen before https://github.com/thelsing/knx (and I see you have forked it) and I think it is more related to creating KNX device than my research. I am approaching from other side, not to be TransportConnection server, but to be client and read or write information from or to devices.
Primary source of information for my research was KNX specs and ETS commissioning tools. Specs are too fragmented - you need to open one pdf, then another, etc.
My goal is to create simple tools to perform basic operations like scan line, read device info(properties, tables), put device into progmode, write some memory blocks, etc. This will require much more efforts than I have done.
I've seen a book and it seems that it contains a lot of useful information.
I couldn't find electronic version to buy, so don't know much what's inside.
https://www.amazon.com/EIB-Installation-System-Thilo-Sauter/dp/3895781754
What point did you reach in your work, what results have you got?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
all sources can be found at https://github.com/dobaosll