quick and dirty hoymles grid profile parser

_hgp_qd.py

#!/usr/bin/python3
#
# hoymiles grid profile parser
# quick and dirty
# dyn table/version
#
# param 1: grid profile
# f.ex. python3 _hgp_qd.py 0a002001000c08fc07a3000f09e2001e064a00140a5500140ac8000a09e21003138812c0001413ec0014128e000514500005200000013003025809e207a3139c1356400007d0001050010001139c0190001000006000000109e20a5a021580010000085b012c08b70941099d012c006490000000005fb000000001f4005f700200012710a00200000000cbfe00
# 

import sys

hex_string = sys.argv[1]

table_types={ 0x00: "Voltage (H/LVRT)", 0x10: "Frequency (H/LFRT)", 0x20: "Island Detection (ID)", 0x30: "Reconnection (RT)", 0x40:"Ramp Rates (RR)", 0x50: "Frequency Watt (FW)", 0x60: "Volt Watt (VW)", 0x70: "Active Power Control (APC)", 0x80:"Volt Var (VV)", 0x90: "Specified Power Factor (SPF)", 0xA0: "Reactive Power Control (RPC)", 0xB0: "Watt Power Factor (WPF)" }

### country / regulations or simple internal DB ID ? ;)
# 02 00 - austria ???? ( empty list ) / IEEE 1547 240V (?)
# 03 00 - germany - DE_VDE4105_2018 (2.0.x)
# 03 01 - unknown
# 0a 00 - European - EN 50549-1:2019
# 0c 00 - Austria - 2.0.x EU_EN50438
# 0d 04 - france - ???
# 12 00 - Poland - 2.0.x (EU_EN50438)
# 37 00 - swiss - 2.0.x (CH_NA EEA-NE7–CH2020)

table_regs={ 
		0x02: { 0x00: "US - IEEE 1547 240V (?)"}, 
	    	0x03: { 0x00: "Germany - DE_VDE4105_2018"}, 
		0x0a: { 0x00: "European - EN 50549-1:2019"},
	 	0x0c: { 0x00: "AT Tor - EU_EN50438" },
		0x0d: { 0x04: "France" }, 
                0x12: { 0x00: "Poland - EU_EN50438" },
		0x37: { 0x00: "Swiss - CH_NA EEA-NE7-CH2020"},
	}

### { table_nr : { table_ver: [["name","unit",divider]] }}
tables_struct = {
	0x00 : { 
		# Germany - DE_VDE4105_2018
		0x0A: [
			["Nominale Voltage (NV)","V",10],
               		["Low Voltage 1 (LV1)","V",10],
	                ["LV1 Maximum Trip Time (MTT)","s",10],
			["High Voltage 1 (HV1)","V",10],
                	["HV1 Maximum Trip Time (MTT)","s",10],
                	["Low Voltage 2 (LV2)","V",10],
                	["LV2 Maximum Trip Time (MTT)","s",10],
                	["10mins Average High Voltage (AHV)","V",10]
		],
		# AT Tor - EU_EN50438
                0x0B: [
                        ["Nominale Voltage (NV)","V",10],
                        ["Low Voltage 1 (LV1)","V",10],
                        ["LV1 Maximum Trip Time (MTT)","s",10],
                        ["High Voltage 1 (HV1)","V",10],
                        ["HV1 Maximum Trip Time (MTT)","s",10],
                        ["Low Voltage 2 (LV2)","V",10],
                        ["LV2 Maximum Trip Time (MTT)","s",10],
			["High Voltage 2 (HV2)","V",10],
			["HV2 Maximum Trip Time (MTT)","s",10],
                        ["10mins Average High Voltage (AHV)","V",10]
                ],
		# Poland - EU_EN50438
                0x00: [
                        ["Nominale Voltage (NV)","V",10],
                        ["Low Voltage 1 (LV1)","V",10],
                        ["LV1 Maximum Trip Time (MTT)","s",10],
                        ["High Voltage 1 (HV1)","V",10],
                        ["HV1 Maximum Trip Time (MTT)","s",10],
                ],
		# Swiss - CH_NA EEA-NE7-CH2020
                0x03: [
                        ["Nominale Voltage (NV)","V",10],
                        ["Low Voltage 1 (LV1)","V",10],
                        ["LV1 Maximum Trip Time (MTT)","s",10],
                        ["High Voltage 1 (HV1)","V",10],
                        ["HV1 Maximum Trip Time (MTT)","s",10],
                        ["Low Voltage 2 (LV2)","V",10],
                        ["LV2 Maximum Trip Time (MTT)","s",10],
                        ["High Voltage 2 (HV2)","V",10],
                        ["HV2 Maximum Trip Time (MTT)","s",10],
                ],
		# European - EN 50549-1:2019
                0x0C: [
                        ["Nominale Voltage (NV)","V",10],
                        ["Low Voltage 1 (LV1)","V",10],
                        ["LV1 Maximum Trip Time (MTT)","s",10],
                        ["High Voltage 1 (HV1)","V",10],
                        ["HV1 Maximum Trip Time (MTT)","s",10],
                        ["Low Voltage 2 (LV2)","V",10],
                        ["LV2 Maximum Trip Time (MTT)","s",10],
                        ["High Voltage 2 (HV2)","V",10],
                        ["HV2 Maximum Trip Time (MTT)","s",10],
                        ["High Voltage 3 (HV3)","V",10],
                        ["HV3 Maximum Trip Time (MTT)","s",10],
                        ["10mins Average High Voltage (AHV)","V",10],
		],
		# unknown 03 01
                0x08: [
                        ["Nominale Voltage (NV)","V",10],
                        ["Low Voltage 1 (LV1)","V",10],
                        ["LV1 Maximum Trip Time (MTT)","s",10],
                        ["High Voltage 1 (HV1)","V",10],
                        ["HV1 Maximum Trip Time (MTT)","s",10],
                        ["? High Voltage 2 (HV2)","V",10],
                ],
		# IEEE 1547 240V (?)
                0x35: [
                        ["Nominale Voltage (NV)","V",10],
                        ["Low Voltage 1 (LV1)","V",10],
                        ["LV1 Maximum Trip Time (MTT)","s",10],
                        ["High Voltage 1 (HV1)","V",10],
                        ["HV1 Maximum Trip Time (MTT)","s",10],
                        ["Low Voltage 2 (LV2)","V",10],
                        ["LV2 Maximum Trip Time (MTT)","s",10],
                        ["High Voltage 2 (HV2)","V",10],
                        ["HV2 Maximum Trip Time (MTT)","s",10],
                        ["?","",1],
                        ["?","",1],
                        ["?","",1],
                        ["?","",1],
		]



	},
	0x10: {
		0x00: [
                	["Nominal Frequency","Hz",100],
	                ["Low Frequency 1 (LF1)","Hz",100],
	                ["LF1 Maximum Trip Time (MTT)","s",10],
	                ["High Frequency 1 (HF1)","Hz",100],
	                ["HF1 Maximum Trip time (MTT)","s",10]
		],
                0x03: [
                        ["Nominal Frequency","Hz",100],
                        ["Low Frequency 1 (LF1)","Hz",100],
                        ["LF1 Maximum Trip Time (MTT)","s",10],
                        ["High Frequency 1 (HF1)","Hz",100],
                        ["HF1 Maximum Trip time (MTT)","s",10],
                        ["Low Frequency 2 (LF2)","Hz",100],
                        ["LF2 Maximum Trip Time (MTT)","s",10],
                        ["High Frequency 2 (HF2)","Hz",100],
                        ["HF2 Maximum Trip time (MTT)","s",10],
                ]

	},
	0x20: { 0x00: [ ["ID Function Activated","bool",1]] },
	
	0x30: { 
		0x03: [
                        ["Reconnect Time (RT)","s",10],
                        ["Reconnect High Voltage (RHV)","V",10],
                        ["Reconnect Low Voltage (RLV)","V",10],
                        ["Reconnect High Frequency (RHF)","Hz",100],
                        ["Reconnect Low Frequency (RLF)","Hz",100]
                ],
                0x07: [
                        ["Reconnect Time (RT)","s",10],
                        ["Reconnect High Voltage (RHV)","V",10],
                        ["Reconnect Low Voltage (RLV)","V",10],
                        ["Reconnect High Frequency (RHF)","Hz",100],
                        ["Reconnect Low Frequency (RLF)","Hz",100],
                        [" ??? ","?",1],
                        [" ??? ","?",1],
                ],
	},
	0x40: { 0x00: [
                	["Normal Ramp up Rate(RUR_NM)","Rated%/s",100],
                	["Soft Start Ramp up Rate (RUR_SS)","Rated%/s",100]
		],


	},
	0x50: { 0x08: [
                	["FW Function Activated","bool",1],
                	["Start of Frequency Watt Droop (Fstart)","Hz",100],
                	["FW Droop Slope (Kpower_Freq)","Pn%/Hz",10],
                	["Recovery Ramp Rate (RRR)","Pn%/s",100],
                	["Recovery High Frequency (RVHF)","Hz",100],
                	["Recovery Low Frequency (RVLF)","Hz",100]
		],
       		0x00: [      
                        ["FW Function Activated","bool",1],
                        ["Start of Frequency Watt Droop (Fstart)","Hz",100],
                        ["FW Droop Slope (Kpower_Freq)","Pn%/Hz",10],
                        ["Recovery Ramp Rate (RRR)","Pn%/s",100],
                ],
                0x01: [    
                        ["FW Function Activated","bool",1],
                        ["Start of Frequency Watt Droop (Fstart)","Hz",100],
                        ["FW Droop Slope (Kpower_Freq)","Pn%/Hz",10],
                        ["Recovery Ramp Rate (RRR)","Pn%/s",100],
			["Recovery High Frequency (RVHF)","Hz",100],
                ],
		# ieee
                0x11: [
                        ["FW Function Activated","bool",1],
                        ["Start of Frequency Watt Droop (Fstart)","Hz",100],
                        ["FW Droop Slope (Kpower_Freq)","Pn%/Hz",10],
                        ["Recovery Ramp Rate (RRR)","Pn%/s",100],
                        ["Recovery High Frequency (RVHF)","Hz",100],
                ],
	},
	0x60: { 0x00: [ 
                        ["VW Function Activated","bool",1],
                        ["Start of Voltage Watt Droop (Vstart)","V",10],
                        ["End of Voltage Watt Droop (Vend)","V",10],
                        ["Droop Slope (Kpower_Volt)","Pn%/V",100]
		],
                0x04: [
                        ["VW Function Activated","bool",1],
                        ["Start of Voltage Watt Droop (Vstart)","V",10],
                        ["End of Voltage Watt Droop (Vend)","V",10],
                        ["Droop Slope (Kpower_Volt)","Pn%/V",100]
                ],
	},
	0x70: { 0x02: [
                	["APC Function Activated","bool",1],
                	["Power Ramp Rate (PRR)","Pn%/s",100]
		],
		0x00: [
                        ["APC Function Activated","bool",1]
		]

	},
	0x80: { 0x00: [
                	["VV Function Activated","bool",1],
                	["Voltage Set Point V1","V",10],
                	["Reactive Set Point Q1","%Pn",10],
                	["Voltage Set Point V2","V",10],
                	["Voltage Set Point V3","V",10],
                	["Voltage Set Point V4","V",10],
                	["Reactive Set Point Q4","%Pn",10]
		],
		0x01: [
                        ["VV Function Activated","bool",1],
                        ["Voltage Set Point V1","V",10],
                        ["Reactive Set Point Q1","%Pn",10],
                        ["Voltage Set Point V2","V",10],
                        ["Voltage Set Point V3","V",10],
                        ["Voltage Set Point V4","V",10],
                        ["Reactive Set Point Q4","%Pn",10],
			["Setting Time (Tr)","s",10]
                ],

	},
	0x90: { 0x00: [ 
			["SPF Function Activated","bool",1],
			["Power Factor (PF)","",100]
		]
	},
	0xA0: { 0x02: [
                	["RPC Function Activated","bool",1],
                	["Reactive Power (VAR)","%Sn",1]
		]
	},
	0xB0: { 0x00: [
                	["WPF Function Activated","bool",1],
                	["Start of Power of WPF (Pstart)","%Pn",10],
                	["Power Factor ar Rated Power (PFRP)","",100]
		]
	}
}

def modbusCrc(msg:str) -> int:
    crc = 0xFFFF
    for n in range(len(msg)):
        crc ^= msg[n]
        for i in range(8):
            if crc & 1:
                crc >>= 1
                crc ^= 0xA001
            else:
                crc >>= 1
    return crc


binary_string = bytes.fromhex(hex_string)
binary_length = len(binary_string)

### internal DB ID ( country/regulations ) ???
str_header1 = binary_string[0]
str_header2 = binary_string[1]
### version ( major.minor / rev ) ???
str_version1 = binary_string[2]
str_version2 = binary_string[3]

try:
	print("Grid Profile: %s " % (table_regs[str_header1][str_header2]))
except:
	print("Grid Profile: unknown ( plz report to https://github.com/tbnobody/OpenDTU/wiki/Grid-Profile-Parser )" )

print ("Version: %s.%s.%s" % (((str_version1 >> 4) & 0x0F),(str_version1 & 0x0F),str_version2))

position=4
while (position < binary_length):
	str_table_n = binary_string[position]
	str_table_v = binary_string[position+1]
	try:
		print("Table Type: " , table_types[str_table_n])
	except:
		pass

	try:
		tables_diz=tables_struct[str_table_n][str_table_v]
		table_length = len(tables_diz)
	except:
		crc=bytearray(binary_string[position:position+2]).hex()
		crc2 = modbusCrc(binary_string[0:position])
		crcc = crc2.to_bytes(2, byteorder='big').hex()

		if crc == crcc:
			print("CRC (ok): ",crcc)
		else:
			print("CRC (?): ",crc)
			print("CRC calced: %s " % (crcc))
			print(" - possible unknown table (module), plz report to https://github.com/tbnobody/OpenDTU/wiki/Grid-Profile-Parser")

		print("end")
		break

	table_pos=0
	#print("str_table_n: %s, str_table_v: %s, table_length: %s" % (str_table_n,str_table_v,table_length))
	position += 2
	for x in range(table_length):
		try:	
			table_diz=tables_diz[table_pos]
		except:
			table_diz=["","",1]
		table_pos += 1

		str_work = binary_string[position:position+2]
		str_hex = str_work.hex()
		str_int = int(str_hex,16) 
		str_val = str_int / table_diz[2]

		print("position: %s \t: %s \t %s \t %s\t[%s]\t\t[%s]" % (position,str_hex,str_int,str_val,table_diz[1],table_diz[0]))
		position += 2

allright, had a quick look ( left out the typos, will be updated with the next gist )

position: 18 	: 0014 	 20 	 0.2	[s]		[LV2 Maximum Trip Time (MTT)] -> I would assume 2.0s
position: 26 	: 000a 	 10 	 0.1	[s]		[HV3 Maximum Trip Time (MTT)] -> I would assume 1.0s

is actually right ... the higher the drift, the faster the device shuts down.

position: 92 : 0215 533 5.33 [Pn%/V] [Droop Slope (Kpower_Volt)] -> I would assume 53.3 Pn%/V
looks right to me, according the pdfs postet at the wiki, also the previous trip time matches

position: 130 : 2710 10000 1.0 [Pn%/s] [Power Ramp Rate (PRR)] -> I would assume 100.00 Pn%/s
you're right ... updated already in my local copy

position: 42 : 0ac8 2760 276.0 [V] [Reconnect High Voltage (RHV)]
your thoughts are right. changed during different versions

position: 66 : 01f6 502 5.02 [Hz] [Recovery High Frequency (RVHF)]
you're right. BUT: i've another example where it decodes to 50.2.
here i assume that the profile is (maybe) wrongly generated by hoymiles.
if not, this would be the only freq-divider where the divider differs from ALL others.

position: 96 : ffa1 65441 654.41 [] [Power Factor (PF)]
also decodes right on most examples. could be a placeholder, as the SPF module is disabled anyway.

hopefully missed no remark.

@noone2k I flipped through the PDFs on the Wiki but could not find the 5.33 Pn%/V Droop Slope, which one did you see this ?

For the Power Factor I have seen one Power Factor 0.95 with Leading (/Lagging) being selected prior to it in the AT TOR-A Grid Profile PDF.

position: 96 	: 005f 	 95 	 0.95	[]		[Power Factor (PF)] -> in CRC 6c58
vs.
position: 96 	: ffa1 	 65441 	 654.41	[]		[Power Factor (PF)] -> in CRC 9c27

Maybe the ff is signalling this negative - Lagging / positive + Leading select box value.
But what would be the remainder a1 or 161 ?
A Power Factor of 1.61 does sound pretty odd to me.

For the 5.02 RVHF this is certainly verbatim in the Grid Profiles we analysed, actually both in v2.0.0 and v2.0.1 of most of the same 0300 Grid Profile "Templates".
Though there is also one 0300 template with CRC e86e which has it IMHO right:

position: 66 	: 139c 	 5020 	 50.2	[Hz]		[Recovery High Frequency (RVHF)] -> in CRC e86e
vs.
position: 66 	: 01f6 	 502 	 5.02	[Hz]		[Recovery High Frequency (RVHF)] -> e.g. in CRC ce67

I would reason the RVHF Recovery High Frequency cannot (ever) be below the RVLF (Recovery Low Frequency), for any practical purpose.

Also note there is another typo with PN instead of Pn

position: 80 	: 0010 	 16 	 0.16	[PN%/s]		[Recovery Ramp Rate (RRR)] -> this should be Pn%/s

thx again ...
typos locally corrected.

flipped through the PDFs on the Wiki but could not find the 5.33 Pn%/V Droop Slope, which one did you see this ?

yep, it is missing on the wiki
search for "Technical-Note-How-to-set-Hoymiles-3Gen-Grid-Profile", that is the base i'm worked from.

position: 96 : ffa1 65441 654.41 [] [Power Factor (PF)] -> in CRC 9c27
you've a point ... didnt recognize the leading selection. must be analyzed further.

for RHVF:
all frequency values have a divider of 100, except this one. makes no sense to me.
my thoughts:
firmware-bug - this "wrong" value can be fixed in two ways: firmware or grid profile.
since previously it was 5020 and afterwards 502, it looks like it was fixed with a "wrong" profile by hoymiles,
instead the firmware ( since profiles are easier to update )
or my first thought: profile wrong generated by hoymiles.
but: this is all guessing.
if the firmware divides this frequency value "accidentally" by 10, then the 502 is right,
otherwise the profile is wrong generated.

updated gist with the corrections
btw. somehow i mixes something up with my local edits ... hopefully, its still ok although
i finished this "walkthrough" already some time ago and my head is already on the next adventure :-)

Thanks I transferred your code into the PR tbnobody/OpenDTU#1527

Please take a look at my guess work in tbnobody/OpenDTU#987 (comment) regarding the Leading flag.
I still have a 0.01 (one-off) in my calculation.

Regarding the fix for RVHF I do agree, it looks like they added some special versions of the Hoymiles Firmware.
Also see tbnobody/OpenDTU#987 for some reports about upgrades to the Firmware due to Grid Profile oddities.

updates gist ( 2 profiles added / incomplete )
need more informations from ( printout from s-miles cloud )

search for "Technical-Note-How-to-set-Hoymiles-3Gen-Grid-Profile", that is the base i'm worked from.

position: 96 : ffa1 65441 654.41 [] [Power Factor (PF)] -> in CRC 9c27 you've a point ... didnt recognize the leading selection. must be analyzed further.

The document you rightfully referred to here https://www.hoymiles.com/wp-content/uploads/2022/11/Technical-Note-How-to-set-Hoymiles-3Gen-Grid-Profile-V1.1.pdf has it in the fine print:

3.9 Specified Power Factor (SPF)
The specified power factor mode is required in some situations by the electric utility to
meet the local requirements. The minimum range of the setting of the fixed power factor is
0.8 leading to 0.8 lagging.
Note1: Power factor is ratio of the absolute value of the active power P to the apparent power S under
periodic conditions.
Note2: Lagging power factor is defined to be when the inverter acts as an inductive load from the
perspective of the grid. Leading power factor is defined to be when the inverter acts as a capacitive load
from the perspective of the grid.
Note3: As shown in the table below, if the input value of power factor is positive, the leading power factor
will be set while if the input value of power factor is negative, the lagging power factor will be set.

Table 9 Specified Power Factor Module

Specified Power Factor (SPF)	-	-	-
Parameter	Value	Unit	Range(min-max)
Function Activated	0	/	0: Disable
			1: Enable
Power Factor (PF)	0.95	/	-0.9~0.9

So Leading is positive and Lagging is negative, hence -0 = 65536 -> 65536 - 95 = 65441 = 0xffa1 as expected.

The same is stated for Power Factor at Rated Power (PFRP) under 3.10 Watt Power Factor (WPF)

Oh by the way, would you consider it worthwhile to sort the tables_struct by the table_nr and table_ver ?

I did clean up the code in my fork of the gist you opened and I find it quite telling that "Poland - EU_EN54038" has table_ver 0x00, whereas Germany has 0x0a and AT_TOR 0x0b respectively. They are likely based on later changes within local law / regulations based on the original EU_EN54038 norm.

@noone2k tbnobody already implemented your parser in the latest OpenDTU release in commits tbnobody/OpenDTU@06651f3 and tbnobody/OpenDTU@00bc631

@stefan123t , @noone2k : I made some changes to parse IEEE 1547 240V correctly in this fork. How to interpret tables / sub-tables and divider taken from here. In case there is some doubt about the divider (e.g. 1.6s or 0.16s), here some nice tabular overviews of the permitted parameter. BTW I am not sure what exactly the difference between IEEE 1547 240V and California Rule21 240V is (If there is any...maybe rule 21 is just interpreting IEEE1547...or rule 21 is making it mandatory to use the IEEE 1547 240V grid profile).

thx ... i'll try to put everything together , once i find some time