mertyildiran/README.md

## README.md

      
    Raw
  

              README.md
            
          
    Mizu Traffic Storage Solutions

In this Gist, three solutions will be proposed for the traffic storage in Mizu:

Migrate to MongoDB, add a collection per protocol and an additional collection for TCP-level information. Indexes defined per protocol.
Migrate to a log oriented solution like Elasticsearch/KQL.
Develop a custom DBMS that's specialized in traffic storage, store dissected packets in binary files. (Let's say its name is Basenine)

Docker image size with SQLite: 85.5MB
Migrate to MongoDB

Please see the Python script and the its output (stdout file) that are provided by this Gist. We saved a bunch of JSON files from Mizu that consists of HTTP 1.1, HTTP 2.0, AMQP and Kafka traffic. The traffic was mostly consisted of HTTP 1.1.
As can be seen from the stdout, there are at least 744 unique paths. The values are how many times they occured.
So it's really hard to come up with an indexing solution that's above all protocols. Therefore the solution is;
Migrate from SQLite to MongoDB.
Store the TCP-level info in a MongoDB collection named tcp:
{
  'Protocol': 757,
  'Protocol.name': 757,
  'Protocol.long_name': 757,
  'Protocol.abbreviation': 757,
  'Protocol.version': 757,
  'Protocol.background_color': 757,
  'Protocol.foreground_color': 757,
  'Protocol.font_size': 757,
  'Protocol.reference_link': 757,
  'Protocol.ports': 757,
  'Protocol.ports.0': 757,
  'Protocol.priority': 757,
  'Timestamp': 757,
  'ConnectionInfo': 757,
  'ConnectionInfo.ClientIP': 757,
  'ConnectionInfo.ClientPort': 757,
  'ConnectionInfo.ServerIP': 757,
  'ConnectionInfo.ServerPort': 757,
  'ConnectionInfo.IsOutgoing': 757
}
Store rest of the data in MongoDB collections per protocol: http, amqp, kafka and let the extensions decide the indexes for those collections.
Refer to those collections from the tcp collection based on the protocol name.
As it's proposed earlier, implement a Wireshark-like DisplayFilters syntax in Go and compile it to MongoDB queries.
The implementation solely happens inside Mizu's codebase.
HTTP body, AMQP and Kafka values can also be stored in different collections too, if the content type is identified to application/json.
Docker image size with MongoDB: 207MB
Advantages


It does not introduce any uncertainty. It will work for sure.
Less development time.
Guaranteed to be horizontally scalable.
Guaranteed to be stable from the very first day.

Disadvantages


The read speed will be mediocre.
Disk usage will be bad.
Nothing genius about it. Everybody can do it.
Limited querying and indexing.
Less isolation.
The parser and compiler for the querying language needs to be implemented.

Migrate to a log oriented solution like Elasticsearch/KQL

TODO: Improve this section.
Elasticsearch might be suitable for storing traffic data.
The Lumberjack Protocol (EOL)
Kibana Query Language instead of Wireshark-like DisplayFilters syntax. Probably requires installing Kibana too.
Docker image size with Logstash: 335MB
Docker image size with Filebeat: 161MB
Advantages


It does not introduce any uncertainty. It will work for sure.
Less development time.
Guaranteed to be horizontally scalable.
Guaranteed to be stable from the very first day.
The querying language comes out of the box.

Disadvantages


The speed projection is unknown. Can be slower than MongoDB (at least write speed).
Disk usage projection is unknown. Can be better than MongoDB.
Nothing genius about it. Everybody can do it.
Limited querying and indexing.
Less isolation.
Will consume the most amount of system resources among all 3 solutions.

Develop a custom DBMS that's specialized in traffic storage (Basenine)

In this solution we implement a custom DBMS that's specialized on traffic storage in C/C++ as a separate project.
The DBMS defines its own binary protocol to efficiently transmit the dissected packets from client to server and vice versa.
It will store dissected packets in binary files.
It will not be bloated by the features (especially for querying) that are irrelevant to traffic data unlike other generalized DBMSes therefore it will have faster read speeds.
It will dump the incoming dissected packets almost as are into a binary file therefore it will have the fastest write speed for this job.
We will implement the Wireshark-like DisplayFilters syntax in C/C++ hence the DBMS' itself will provide this language rather than implementing it inside Mizu.
Migrate from SQLite to Basenine.
How will it reduce the disk usage?

The DBMS will reduce the duplicate data. For example there will be a lot of HTTP entries with method GET why storing duplicate GET strings if the records will not be edited? So the DBMS should automatically be able to reduce it to one. Similarly path /example or duplicate header key-value pairs, etc. (duplicate means across all the traffic log, not the duplication inside the individual record)

Note: Don't confuse it with algorithms like Deflate, gzip. They are not aware of the structure of the data, they are just binary compression algorithms. While this is reducing data redundancy.

It should be content type aware for HTTP body, AMQP and Kafka values. Such that for application/json it will also reduce the duplications for the JSON fields.
There will be a queue that will accept incoming insertions. A number of threads will constantly pop that queue and insert the records by reducing duplications. Once the record is reduced and inserted, it will be streamed back to Mizu for displaying. The same exact stream will be used for filtering too. In fact, it will always be in filtering mode. It just won't filter out any records if the filter is an empty string. For each connection restart, it will start to stream all the records from the beginning according to the filtering criteria.
How will it quickly insert?

It will store a hash table (like MD5, a fast checksum algorithm) for each incoming strings (char*). If the newly inserted string field's checksum matches to a previously inserted string field's checksum, it will check it for full value equality (MD5 can collide for example). If the data is really equal, it will refer/point to a previously written region (offsetting) of the binary instead of writing new bytes hence the fast reducing.
There is no need to reduce duplications smaller than or equal to 8 bytes because the reference will be 8 bytes too.
Storing the checksum string might be more costly than the string's itself if it's a short string. Since the duplication in the future is not guaranteed. But since the database is dedicated for traffic logging, the amount of duplication will compansate the cost of storing checksums.
How will it query fast?

The queriable fields will be given upon initialization by Mizu to Basenine such that while inserting the data it will keep track of another hash table that will know the exact offsets of the fields inside the binary dump for a given query path (like http.request.method). So it will just jump to those offsets and check for value equality and return the records accordingly. If the jumped offset is a reference to another value (a reduced field) than it will do another jump and check the value equality there. So with a maximum of two jumps, without iterating the whole database, it will find the value.
Since the database is only create/read and not update/delete. The offsets will be static (not shifting anything). Therefore it's not incredibly hard to implement and it can be fast.
Docker image size with Basenine: will aim less than 100MB
Advantages


Better isolation, separation of concerns.
Possibility of having incredible read and write speeds.
Possibility of being more efficient in terms of disk space.
Lower latency for data transmission.
Having more control over querying.
C/C++ :)
Will consume the least amount of system resources among all 3 solutions.
Not everybody can do it.

Disadvantages


More development time.
Not guaranteed to be horizontally scalable in its first days.
Not guaranteed to be stable in its first days.


## main.py
import json
import glob
import pprint
try:
    from collections.abc import MutableMapping
    from collections.abc import Sequence
except ModuleNotFoundError:
    from collections import MutableMapping
    from collections import Sequence

files = glob.glob("jsons/*.json")

def get_paths(source):
    paths = []
    if isinstance(source, MutableMapping):  # found a dict-like structure...
        for k, v in source.items():  # iterate over it; Python 2.x: source.iteritems()
            paths.append([k])  # add the current child path
            paths += [[k] + x for x in get_paths(v)]  # get sub-paths, extend with the current
    # else, check if a list-like structure, remove if you don't want list paths included
    elif isinstance(source, Sequence) and not isinstance(source, str):
        #                          Python 2.x: use basestring instead of str ^
        for i, v in enumerate(source):
            paths.append([i])
            paths += [[i] + x for x in get_paths(v)]  # get sub-paths, extend with the current
    return paths

path_weights = {}

for file_path in files:
    with open(file_path) as file:
        data = json.load(file)
        paths = get_paths(data)
        for path in paths:
            path_all_str = [str(int) for int in path]
            key = '.'.join(path_all_str)
            if key not in path_weights:
                path_weights[key] = 1
            else:
                path_weights[key] += 1


path_weights = dict(sorted(path_weights.items(), key=lambda item: item[1], reverse=True))

print('Path weights:')

pprint.pprint(path_weights, indent=2, sort_dicts=False)

print('Total paths:', len(path_weights.keys()))

## stdout
Path weights:
{ 'Protocol': 757,
  'Protocol.name': 757,
  'Protocol.long_name': 757,
  'Protocol.abbreviation': 757,
  'Protocol.version': 757,
  'Protocol.background_color': 757,
  'Protocol.foreground_color': 757,
  'Protocol.font_size': 757,
  'Protocol.reference_link': 757,
  'Protocol.ports': 757,
  'Protocol.ports.0': 757,
  'Protocol.priority': 757,
  'Timestamp': 757,
  'ConnectionInfo': 757,
  'ConnectionInfo.ClientIP': 757,
  'ConnectionInfo.ClientPort': 757,
  'ConnectionInfo.ServerIP': 757,
  'ConnectionInfo.ServerPort': 757,
  'ConnectionInfo.IsOutgoing': 757,
  'Pair': 757,
  'Pair.request': 757,
  'Pair.request.is_request': 757,
  'Pair.request.capture_time': 757,
  'Pair.request.payload': 757,
  'Pair.request.payload.details': 757,
  'Pair.request.payload.method': 757,
  'Pair.request.payload.url': 757,
  'Pair.response': 757,
  'Pair.response.is_request': 757,
  'Pair.response.capture_time': 757,
  'Pair.response.payload': 757,
  'Pair.response.payload.details': 747,
  'Pair.response.payload.method': 747,
  'Pair.response.payload.url': 747,
  'Protocol.ports.1': 744,
  'Pair.request.payload.details.bodySize': 734,
  'Pair.request.payload.details.cookies': 734,
  'Pair.request.payload.details.headers': 734,
  'Pair.request.payload.details.headers.0': 734,
  'Pair.request.payload.details.headers.0.name': 734,
  'Pair.request.payload.details.headers.0.value': 734,
  'Pair.request.payload.details.headers.1': 734,
  'Pair.request.payload.details.headers.1.name': 734,
  'Pair.request.payload.details.headers.1.value': 734,
  'Pair.request.payload.details.headers.2': 734,
  'Pair.request.payload.details.headers.2.name': 734,
  'Pair.request.payload.details.headers.2.value': 734,
  'Pair.request.payload.details.headersSize': 734,
  'Pair.request.payload.details.httpVersion': 734,
  'Pair.request.payload.details.method': 734,
  'Pair.request.payload.details.queryString': 734,
  'Pair.request.payload.details.url': 734,
  'Pair.response.payload.details.bodySize': 734,
  'Pair.response.payload.details.content': 734,
  'Pair.response.payload.details.content.encoding': 734,
  'Pair.response.payload.details.content.mimeType': 734,
  'Pair.response.payload.details.content.size': 734,
  'Pair.response.payload.details.cookies': 734,
  'Pair.response.payload.details.headers': 734,
  'Pair.response.payload.details.headers.0': 734,
  'Pair.response.payload.details.headers.0.name': 734,
  'Pair.response.payload.details.headers.0.value': 734,
  'Pair.response.payload.details.headersSize': 734,
  'Pair.response.payload.details.httpVersion': 734,
  'Pair.response.payload.details.redirectURL': 734,
  'Pair.response.payload.details.status': 734,
  'Pair.response.payload.details.statusText': 734,
  'Protocol.ports.2': 731,
  'Pair.response.payload.details.headers.1': 731,
  'Pair.response.payload.details.headers.1.name': 731,
  'Pair.response.payload.details.headers.1.value': 731,
  'Pair.request.payload.details.headers.3': 730,
  'Pair.request.payload.details.headers.3.name': 730,
  'Pair.request.payload.details.headers.3.value': 730,
  'Pair.request.payload.details.headers.4': 730,
  'Pair.request.payload.details.headers.4.name': 730,
  'Pair.request.payload.details.headers.4.value': 730,
  'Pair.response.payload.details.headers.2': 670,
  'Pair.response.payload.details.headers.2.name': 670,
  'Pair.response.payload.details.headers.2.value': 670,
  'Pair.response.payload.details.headers.3': 664,
  'Pair.response.payload.details.headers.3.name': 664,
  'Pair.response.payload.details.headers.3.value': 664,
  'Pair.response.payload.details.content.text': 610,
  'Pair.response.payload.details.headers.4': 532,
  'Pair.response.payload.details.headers.4.name': 532,
  'Pair.response.payload.details.headers.4.value': 532,
  'Pair.request.payload.details.headers.5': 191,
  'Pair.request.payload.details.headers.5.name': 191,
  'Pair.request.payload.details.headers.5.value': 191,
  'Pair.request.payload.details.postData': 115,
  'Pair.request.payload.details.postData.mimeType': 115,
  'Pair.request.payload.details.postData.params': 115,
  'Pair.request.payload.details.postData.text': 115,
  'Pair.request.payload.details.headers.6': 106,
  'Pair.request.payload.details.headers.6.name': 106,
  'Pair.request.payload.details.headers.6.value': 106,
  'Pair.response.payload.details.headers.5': 84,
  'Pair.response.payload.details.headers.5.name': 84,
  'Pair.response.payload.details.headers.5.value': 84,
  'Pair.request.payload.details.queryString.0': 59,
  'Pair.request.payload.details.queryString.0.name': 59,
  'Pair.request.payload.details.queryString.0.value': 59,
  'Pair.response.payload.details.headers.6': 35,
  'Pair.response.payload.details.headers.6.name': 35,
  'Pair.response.payload.details.headers.6.value': 35,
  'Pair.request.payload.details.postData.params.0': 32,
  'Pair.request.payload.details.postData.params.0.name': 32,
  'Pair.request.payload.details.postData.params.0.value': 32,
  'Pair.request.payload.details.headers.7': 31,
  'Pair.request.payload.details.headers.7.name': 31,
  'Pair.request.payload.details.headers.7.value': 31,
  'Pair.request.payload.details.headers.8': 27,
  'Pair.request.payload.details.headers.8.name': 27,
  'Pair.request.payload.details.headers.8.value': 27,
  'Pair.request.payload.details.queryString.1': 26,
  'Pair.request.payload.details.queryString.1.name': 26,
  'Pair.request.payload.details.queryString.1.value': 26,
  'Pair.request.payload.details.headers.9': 24,
  'Pair.request.payload.details.headers.9.name': 24,
  'Pair.request.payload.details.headers.9.value': 24,
  'Pair.response.payload.details.headers.7': 22,
  'Pair.response.payload.details.headers.7.name': 22,
  'Pair.response.payload.details.headers.7.value': 22,
  'Pair.request.payload.details.headers.10': 21,
  'Pair.request.payload.details.headers.10.name': 21,
  'Pair.request.payload.details.headers.10.value': 21,
  'Pair.request.payload.details.headers.11': 21,
  'Pair.request.payload.details.headers.11.name': 21,
  'Pair.request.payload.details.headers.11.value': 21,
  'Pair.request.payload.details.headers.12': 21,
  'Pair.request.payload.details.headers.12.name': 21,
  'Pair.request.payload.details.headers.12.value': 21,
  'Pair.request.payload.details.cookies.0': 15,
  'Pair.request.payload.details.cookies.0.name': 15,
  'Pair.request.payload.details.cookies.0.value': 15,
  'Pair.request.payload.details.cookies.1': 15,
  'Pair.request.payload.details.cookies.1.name': 15,
  'Pair.request.payload.details.cookies.1.value': 15,
  'Pair.request.payload.details.headers.13': 15,
  'Pair.request.payload.details.headers.13.name': 15,
  'Pair.request.payload.details.headers.13.value': 15,
  'Pair.request.payload.details.postData.params.1': 13,
  'Pair.request.payload.details.postData.params.1.name': 13,
  'Pair.request.payload.details.postData.params.1.value': 13,
  'Pair.request.payload.details.ApiKey': 13,
  'Pair.request.payload.details.ApiVersion': 13,
  'Pair.request.payload.details.ClientID': 13,
  'Pair.request.payload.details.CorrelationID': 13,
  'Pair.request.payload.details.Payload': 13,
  'Pair.request.payload.details.Size': 13,
  'Pair.response.payload.details.CorrelationID': 13,
  'Pair.response.payload.details.Payload': 13,
  'Pair.response.payload.details.Size': 13,
  'Pair.response.payload.details.headers.8': 12,
  'Pair.response.payload.details.headers.8.name': 12,
  'Pair.response.payload.details.headers.8.value': 12,
  'Pair.request.payload.details.postData.params.0.contentType': 11,
  'Pair.request.payload.details.postData.params.0.fileName': 11,
  'Pair.request.payload.details.Payload.Topics': 10,
  'Pair.response.payload.details.Payload.Topics': 9,
  'Pair.request.payload.details.Payload.Topics.0': 7,
  'Pair.response.payload.details.Payload.Topics.0': 7,
  'Pair.response.payload.details.Payload.Topics.0.Name': 7,
  'Pair.request.payload.details.Exchange': 6,
  'Pair.request.payload.details.postData.params.1.contentType': 6,
  'Pair.request.payload.details.postData.params.1.fileName': 6,
  'Pair.request.payload.details.Payload.Topics.0.Name': 6,
  'Pair.request.payload.details.RoutingKey': 5,
  'Pair.response.payload.details.Payload.Brokers': 5,
  'Pair.response.payload.details.Payload.Brokers.0': 5,
  'Pair.response.payload.details.Payload.Brokers.0.Host': 5,
  'Pair.response.payload.details.Payload.Brokers.0.NodeId': 5,
  'Pair.response.payload.details.Payload.Brokers.0.Port': 5,
  'Pair.response.payload.details.Payload.Brokers.0.Rack': 5,
  'Pair.response.payload.details.Payload.ControllerID': 5,
  'Pair.response.payload.details.Payload.Topics.0.ErrorCode': 5,
  'Pair.response.payload.details.Payload.Topics.0.Partitions': 5,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.0': 5,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.0.ErrorCode': 5,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.0.PartitionIndex': 5,
  'Pair.response.payload.details.headers.9': 5,
  'Pair.response.payload.details.headers.9.name': 5,
  'Pair.response.payload.details.headers.9.value': 5,
  'Pair.response.payload.details.headers.10': 5,
  'Pair.response.payload.details.headers.10.name': 5,
  'Pair.response.payload.details.headers.10.value': 5,
  'Pair.request.payload.details.Body': 4,
  'Pair.request.payload.details.Properties': 4,
  'Pair.request.payload.details.Properties.AppId': 4,
  'Pair.request.payload.details.Properties.ContentEncoding': 4,
  'Pair.request.payload.details.Properties.ContentType': 4,
  'Pair.request.payload.details.Properties.CorrelationId': 4,
  'Pair.request.payload.details.Properties.DeliveryMode': 4,
  'Pair.request.payload.details.Properties.Expiration': 4,
  'Pair.request.payload.details.Properties.Headers': 4,
  'Pair.request.payload.details.Properties.MessageId': 4,
  'Pair.request.payload.details.Properties.Priority': 4,
  'Pair.request.payload.details.Properties.ReplyTo': 4,
  'Pair.request.payload.details.Properties.Timestamp': 4,
  'Pair.request.payload.details.Properties.Type': 4,
  'Pair.request.payload.details.Properties.UserId': 4,
  'Pair.request.payload.details.postData.params.2': 4,
  'Pair.request.payload.details.postData.params.2.name': 4,
  'Pair.request.payload.details.postData.params.2.value': 4,
  'Pair.response.payload.details.headers.11': 4,
  'Pair.response.payload.details.headers.11.name': 4,
  'Pair.response.payload.details.headers.11.value': 4,
  'Pair.response.payload.details.headers.12': 4,
  'Pair.response.payload.details.headers.12.name': 4,
  'Pair.response.payload.details.headers.12.value': 4,
  'Pair.response.payload.details.headers.13': 4,
  'Pair.response.payload.details.headers.13.name': 4,
  'Pair.response.payload.details.headers.13.value': 4,
  'Pair.request.payload.details.Arguments': 4,
  'Pair.request.payload.details.NoWait': 4,
  'Pair.request.payload.details.ConsumerTag': 3,
  'Pair.response.payload.details.Payload.Topics.0.IsInternal': 3,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.0.IsrNodes': 3,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.0.LeaderId': 3,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.0.ReplicaNodes': 3,
  'Pair.request.payload.details.Queue': 3,
  'Pair.request.payload.details.Payload.ReplicaId': 3,
  'Pair.request.payload.details.Payload.Topics.0.Partitions': 3,
  'Pair.request.payload.details.Payload.Topics.0.Partitions.0': 3,
  'Pair.response.payload.details.Payload.ErrorCode': 3,
  'Pair.request.payload.details.headers.14': 3,
  'Pair.request.payload.details.headers.14.name': 3,
  'Pair.request.payload.details.headers.14.value': 3,
  'Pair.response.payload.details.cookies.0': 3,
  'Pair.response.payload.details.cookies.0.name': 3,
  'Pair.response.payload.details.cookies.0.path': 3,
  'Pair.response.payload.details.cookies.0.value': 3,
  'Pair.response.payload.details.cookies.1': 3,
  'Pair.response.payload.details.cookies.1.name': 3,
  'Pair.response.payload.details.cookies.1.path': 3,
  'Pair.response.payload.details.cookies.1.value': 3,
  'Pair.request.payload.details.Immediate': 2,
  'Pair.request.payload.details.Mandatory': 2,
  'Pair.request.payload.details.Properties.Headers.hdr1': 2,
  'Pair.request.payload.details.postData.params.2.contentType': 2,
  'Pair.request.payload.details.postData.params.2.fileName': 2,
  'Pair.request.payload.details.postData.encoding': 2,
  'Pair.request.payload.details.DeliveryTag': 2,
  'Pair.request.payload.details.Properties.Headers.hdr0': 2,
  'Pair.request.payload.details.Redelivered': 2,
  'Pair.request.payload.details.Payload.TimeoutMs': 2,
  'Pair.request.payload.details.Payload.Topics.0.Assignments': 2,
  'Pair.request.payload.details.Payload.Topics.0.Configs': 2,
  'Pair.request.payload.details.Payload.Topics.0.NumPartitions': 2,
  'Pair.request.payload.details.Payload.Topics.0.ReplicationFactor': 2,
  'Pair.request.payload.details.AutoDelete': 2,
  'Pair.request.payload.details.Durable': 2,
  'Pair.request.payload.details.Exclusive': 2,
  'Pair.request.payload.details.Passive': 2,
  'Pair.response.payload.details.Payload.Responses': 2,
  'Pair.response.payload.details.Payload.Responses.0': 2,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses': 2,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0': 2,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.ErrorCode': 2,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.LogStartOffset': 2,
  'Pair.response.payload.details.Payload.ThrottleTimeMs': 2,
  'Pair.response.payload.details.Payload.ApiKeys': 2,
  'Pair.response.payload.details.Payload.ApiKeys.0': 2,
  'Pair.response.payload.details.Payload.ApiKeys.0.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.0.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.0.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.1': 2,
  'Pair.response.payload.details.Payload.ApiKeys.1.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.1.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.1.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.2': 2,
  'Pair.response.payload.details.Payload.ApiKeys.2.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.2.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.2.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.3': 2,
  'Pair.response.payload.details.Payload.ApiKeys.3.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.3.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.3.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.4': 2,
  'Pair.response.payload.details.Payload.ApiKeys.4.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.4.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.4.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.5': 2,
  'Pair.response.payload.details.Payload.ApiKeys.5.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.5.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.5.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.6': 2,
  'Pair.response.payload.details.Payload.ApiKeys.6.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.6.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.6.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.7': 2,
  'Pair.response.payload.details.Payload.ApiKeys.7.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.7.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.7.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.8': 2,
  'Pair.response.payload.details.Payload.ApiKeys.8.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.8.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.8.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.9': 2,
  'Pair.response.payload.details.Payload.ApiKeys.9.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.9.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.9.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.10': 2,
  'Pair.response.payload.details.Payload.ApiKeys.10.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.10.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.10.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.11': 2,
  'Pair.response.payload.details.Payload.ApiKeys.11.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.11.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.11.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.12': 2,
  'Pair.response.payload.details.Payload.ApiKeys.12.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.12.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.12.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.13': 2,
  'Pair.response.payload.details.Payload.ApiKeys.13.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.13.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.13.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.14': 2,
  'Pair.response.payload.details.Payload.ApiKeys.14.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.14.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.14.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.15': 2,
  'Pair.response.payload.details.Payload.ApiKeys.15.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.15.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.15.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.16': 2,
  'Pair.response.payload.details.Payload.ApiKeys.16.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.16.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.16.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.17': 2,
  'Pair.response.payload.details.Payload.ApiKeys.17.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.17.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.17.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.18': 2,
  'Pair.response.payload.details.Payload.ApiKeys.18.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.18.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.18.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.19': 2,
  'Pair.response.payload.details.Payload.ApiKeys.19.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.19.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.19.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.20': 2,
  'Pair.response.payload.details.Payload.ApiKeys.20.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.20.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.20.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.21': 2,
  'Pair.response.payload.details.Payload.ApiKeys.21.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.21.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.21.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.22': 2,
  'Pair.response.payload.details.Payload.ApiKeys.22.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.22.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.22.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.23': 2,
  'Pair.response.payload.details.Payload.ApiKeys.23.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.23.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.23.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.24': 2,
  'Pair.response.payload.details.Payload.ApiKeys.24.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.24.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.24.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.25': 2,
  'Pair.response.payload.details.Payload.ApiKeys.25.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.25.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.25.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.26': 2,
  'Pair.response.payload.details.Payload.ApiKeys.26.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.26.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.26.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.27': 2,
  'Pair.response.payload.details.Payload.ApiKeys.27.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.27.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.27.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.28': 2,
  'Pair.response.payload.details.Payload.ApiKeys.28.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.28.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.28.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.29': 2,
  'Pair.response.payload.details.Payload.ApiKeys.29.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.29.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.29.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.30': 2,
  'Pair.response.payload.details.Payload.ApiKeys.30.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.30.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.30.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.31': 2,
  'Pair.response.payload.details.Payload.ApiKeys.31.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.31.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.31.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.32': 2,
  'Pair.response.payload.details.Payload.ApiKeys.32.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.32.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.32.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.33': 2,
  'Pair.response.payload.details.Payload.ApiKeys.33.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.33.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.33.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.34': 2,
  'Pair.response.payload.details.Payload.ApiKeys.34.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.34.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.34.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.35': 2,
  'Pair.response.payload.details.Payload.ApiKeys.35.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.35.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.35.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.36': 2,
  'Pair.response.payload.details.Payload.ApiKeys.36.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.36.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.36.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.37': 2,
  'Pair.response.payload.details.Payload.ApiKeys.37.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.37.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.37.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.38': 2,
  'Pair.response.payload.details.Payload.ApiKeys.38.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.38.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.38.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.39': 2,
  'Pair.response.payload.details.Payload.ApiKeys.39.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.39.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.39.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.40': 2,
  'Pair.response.payload.details.Payload.ApiKeys.40.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.40.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.40.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.41': 2,
  'Pair.response.payload.details.Payload.ApiKeys.41.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.41.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.41.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.42': 2,
  'Pair.response.payload.details.Payload.ApiKeys.42.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.42.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.42.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.43': 2,
  'Pair.response.payload.details.Payload.ApiKeys.43.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.43.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.43.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.44': 2,
  'Pair.response.payload.details.Payload.ApiKeys.44.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.44.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.44.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.45': 2,
  'Pair.response.payload.details.Payload.ApiKeys.45.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.45.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.45.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.46': 2,
  'Pair.response.payload.details.Payload.ApiKeys.46.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.46.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.46.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.47': 2,
  'Pair.response.payload.details.Payload.ApiKeys.47.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.47.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.47.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.48': 2,
  'Pair.response.payload.details.Payload.ApiKeys.48.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.48.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.48.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.49': 2,
  'Pair.response.payload.details.Payload.ApiKeys.49.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.49.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.49.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.50': 2,
  'Pair.response.payload.details.Payload.ApiKeys.50.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.50.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.50.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.51': 2,
  'Pair.response.payload.details.Payload.ApiKeys.51.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.51.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.51.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.52': 2,
  'Pair.response.payload.details.Payload.ApiKeys.52.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.52.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.52.MinVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.53': 2,
  'Pair.response.payload.details.Payload.ApiKeys.53.ApiKey': 2,
  'Pair.response.payload.details.Payload.ApiKeys.53.MaxVersion': 2,
  'Pair.response.payload.details.Payload.ApiKeys.53.MinVersion': 2,
  'Pair.request.payload.details.Payload.Topics.0.Partitions.0.PartitionIndex': 2,
  'Pair.request.payload.details.Payload.Topics.0.Partitions.0.Timestamp': 2,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.0.Offset': 2,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.0.Timestamp': 2,
  'Pair.request.payload.details.queryString.2': 2,
  'Pair.request.payload.details.queryString.2.name': 2,
  'Pair.request.payload.details.queryString.2.value': 2,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.1': 1,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.1.ErrorCode': 1,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.1.IsrNodes': 1,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.1.LeaderId': 1,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.1.PartitionIndex': 1,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.1.ReplicaNodes': 1,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.2': 1,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.2.ErrorCode': 1,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.2.IsrNodes': 1,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.2.LeaderId': 1,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.2.PartitionIndex': 1,
  'Pair.response.payload.details.Payload.Topics.0.Partitions.2.ReplicaNodes': 1,
  'Pair.response.payload.details.Payload.Topics.1': 1,
  'Pair.response.payload.details.Payload.Topics.1.ErrorCode': 1,
  'Pair.response.payload.details.Payload.Topics.1.IsInternal': 1,
  'Pair.response.payload.details.Payload.Topics.1.Name': 1,
  'Pair.response.payload.details.Payload.Topics.1.Partitions': 1,
  'Pair.response.payload.details.Payload.Topics.2': 1,
  'Pair.response.payload.details.Payload.Topics.2.ErrorCode': 1,
  'Pair.response.payload.details.Payload.Topics.2.IsInternal': 1,
  'Pair.response.payload.details.Payload.Topics.2.Name': 1,
  'Pair.response.payload.details.Payload.Topics.2.Partitions': 1,
  'Pair.response.payload.details.Payload.Topics.3': 1,
  'Pair.response.payload.details.Payload.Topics.3.ErrorCode': 1,
  'Pair.response.payload.details.Payload.Topics.3.IsInternal': 1,
  'Pair.response.payload.details.Payload.Topics.3.Name': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.0': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.0.ErrorCode': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.0.IsrNodes': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.0.LeaderId': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.0.PartitionIndex': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.0.ReplicaNodes': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.1': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.1.ErrorCode': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.1.IsrNodes': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.1.LeaderId': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.1.PartitionIndex': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.1.ReplicaNodes': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.2': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.2.ErrorCode': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.2.IsrNodes': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.2.LeaderId': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.2.PartitionIndex': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.2.ReplicaNodes': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.3': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.3.ErrorCode': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.3.IsrNodes': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.3.LeaderId': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.3.PartitionIndex': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.3.ReplicaNodes': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.4': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.4.ErrorCode': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.4.IsrNodes': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.4.LeaderId': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.4.PartitionIndex': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.4.ReplicaNodes': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.5': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.5.ErrorCode': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.5.IsrNodes': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.5.LeaderId': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.5.PartitionIndex': 1,
  'Pair.response.payload.details.Payload.Topics.3.Partitions.5.ReplicaNodes': 1,
  'Pair.response.payload.details.Payload.Topics.4': 1,
  'Pair.response.payload.details.Payload.Topics.4.ErrorCode': 1,
  'Pair.response.payload.details.Payload.Topics.4.IsInternal': 1,
  'Pair.response.payload.details.Payload.Topics.4.Name': 1,
  'Pair.response.payload.details.Payload.Topics.4.Partitions': 1,
  'Pair.request.payload.details.Payload.ForgottenTopicsData': 1,
  'Pair.request.payload.details.Payload.ForgottenTopicsData.Partitions': 1,
  'Pair.request.payload.details.Payload.ForgottenTopicsData.Topic': 1,
  'Pair.request.payload.details.Payload.IsolationLevel': 1,
  'Pair.request.payload.details.Payload.MaxBytes': 1,
  'Pair.request.payload.details.Payload.MaxWaitMs': 1,
  'Pair.request.payload.details.Payload.MinBytes': 1,
  'Pair.request.payload.details.Payload.SessionEpoch': 1,
  'Pair.request.payload.details.Payload.SessionId': 1,
  'Pair.request.payload.details.Payload.Topics.0.Partitions.0.CurrentLeaderEpoch': 1,
  'Pair.request.payload.details.Payload.Topics.0.Partitions.0.FetchOffset': 1,
  'Pair.request.payload.details.Payload.Topics.0.Partitions.0.LogStartOffset': 1,
  'Pair.request.payload.details.Payload.Topics.0.Partitions.0.Partition': 1,
  'Pair.request.payload.details.Payload.Topics.0.Partitions.0.PartitionMaxBytes': 1,
  'Pair.request.payload.details.Payload.Topics.0.Topic': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.AbortedTransactions': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.AbortedTransactions.FirstOffset': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.AbortedTransactions.ProducerId': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.HighWatermark': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.LastStableOffset': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.Partition': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Attributes': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.BaseOffset': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.BaseSequence': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.BatchLength': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Crc': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.FirstTimestamp': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.LastOffsetDelta': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Magic': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.MaxTimestamp': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.PartitionLeaderEpoch': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.ProducerEpoch': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.ProducerId': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.0': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.0.Attributes': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.0.Headers': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.0.Headers.0': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.0.Headers.0.HeaderKey': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.0.Headers.0.HeaderKeyLength': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.0.Headers.0.HeaderValueLength': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.0.Headers.0.Value': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.0.Key': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.0.KeyLength': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.0.OffsetDelta': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.0.TimestampDelta': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.0.Unknown': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.0.Value': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.0.ValueLen': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.1': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.1.Attributes': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.1.Headers': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.1.Headers.0': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.1.Headers.0.HeaderKey': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.1.Headers.0.HeaderKeyLength': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.1.Headers.0.HeaderValueLength': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.1.Headers.0.Value': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.1.Key': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.1.KeyLength': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.1.OffsetDelta': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.1.TimestampDelta': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.1.Unknown': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.1.Value': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.1.ValueLen': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.2': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.2.Attributes': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.2.Headers': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.2.Headers.0': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.2.Headers.0.HeaderKey': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.2.Headers.0.HeaderKeyLength': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.2.Headers.0.HeaderValueLength': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.2.Headers.0.Value': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.2.Key': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.2.KeyLength': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.2.OffsetDelta': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.2.TimestampDelta': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.2.Unknown': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.2.Value': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.RecordSet.RecordBatch.Record.2.ValueLen': 1,
  'Pair.response.payload.details.Payload.Responses.0.Topic': 1,
  'Pair.response.payload.details.Payload.SessionId': 1,
  'Pair.request.payload.details.Locales': 1,
  'Pair.request.payload.details.Mechanisms': 1,
  'Pair.request.payload.details.ServerProperties': 1,
  'Pair.request.payload.details.ServerProperties.capabilities': 1,
  'Pair.request.payload.details.ServerProperties.capabilities.authentication_failure_close': 1,
  'Pair.request.payload.details.ServerProperties.capabilities.basic.nack': 1,
  'Pair.request.payload.details.ServerProperties.capabilities.connection.blocked': 1,
  'Pair.request.payload.details.ServerProperties.capabilities.consumer_cancel_notify': 1,
  'Pair.request.payload.details.ServerProperties.capabilities.consumer_priorities': 1,
  'Pair.request.payload.details.ServerProperties.capabilities.direct_reply_to': 1,
  'Pair.request.payload.details.ServerProperties.capabilities.exchange_exchange_bindings': 1,
  'Pair.request.payload.details.ServerProperties.capabilities.per_consumer_qos': 1,
  'Pair.request.payload.details.ServerProperties.capabilities.publisher_confirms': 1,
  'Pair.request.payload.details.ServerProperties.cluster_name': 1,
  'Pair.request.payload.details.ServerProperties.copyright': 1,
  'Pair.request.payload.details.ServerProperties.information': 1,
  'Pair.request.payload.details.ServerProperties.platform': 1,
  'Pair.request.payload.details.ServerProperties.product': 1,
  'Pair.request.payload.details.ServerProperties.version': 1,
  'Pair.request.payload.details.VersionMajor': 1,
  'Pair.request.payload.details.VersionMinor': 1,
  'Pair.request.payload.details.NoAck': 1,
  'Pair.request.payload.details.NoLocal': 1,
  'Pair.request.payload.details.headers.15': 1,
  'Pair.request.payload.details.headers.15.name': 1,
  'Pair.request.payload.details.headers.15.value': 1,
  'Pair.request.payload.details.ClassId': 1,
  'Pair.request.payload.details.MethodId': 1,
  'Pair.request.payload.details.ReplyCode': 1,
  'Pair.request.payload.details.ReplyText': 1,
  'Pair.request.payload.details.Internal': 1,
  'Pair.request.payload.details.Type': 1,
  'Pair.request.payload.details.Payload.RequiredAcks': 1,
  'Pair.request.payload.details.Payload.Timeout': 1,
  'Pair.request.payload.details.Payload.TopicData': 1,
  'Pair.request.payload.details.Payload.TopicData.0': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.Length': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Index': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Attributes': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.BaseOffset': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.BaseSequence': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.BatchLength': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Crc': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.FirstTimestamp': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.LastOffsetDelta': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Magic': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.MaxTimestamp': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.PartitionLeaderEpoch': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.ProducerEpoch': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.ProducerId': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.0': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.0.Attributes': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.0.Headers': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.0.Headers.0': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.0.Headers.0.HeaderKey': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.0.Headers.0.HeaderKeyLength': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.0.Headers.0.HeaderValueLength': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.0.Headers.0.Value': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.0.Key': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.0.KeyLength': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.0.OffsetDelta': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.0.TimestampDelta': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.0.Unknown': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.0.Value': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.0.ValueLen': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.1': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.1.Attributes': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.1.Headers': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.1.Headers.0': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.1.Headers.0.HeaderKey': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.1.Headers.0.HeaderKeyLength': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.1.Headers.0.HeaderValueLength': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.1.Headers.0.Value': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.1.Key': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.1.KeyLength': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.1.OffsetDelta': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.1.TimestampDelta': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.1.Unknown': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.1.Value': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.1.ValueLen': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.2': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.2.Attributes': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.2.Headers': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.2.Headers.0': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.2.Headers.0.HeaderKey': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.2.Headers.0.HeaderKeyLength': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.2.Headers.0.HeaderValueLength': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.2.Headers.0.Value': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.2.Key': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.2.KeyLength': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.2.OffsetDelta': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.2.TimestampDelta': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.2.Unknown': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.2.Value': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Records.RecordBatch.Record.2.ValueLen': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Partitions.PartitionData.Unknown': 1,
  'Pair.request.payload.details.Payload.TopicData.0.Topic': 1,
  'Pair.request.payload.details.Payload.TransactionalID': 1,
  'Pair.response.payload.details.Payload.Responses.0.Name': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.BaseOffset': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.Index': 1,
  'Pair.response.payload.details.Payload.Responses.0.PartitionResponses.0.LogAppendTimeMs': 1}
Total paths: 744
	import json
	import glob
	import pprint
	try:
	from collections.abc import MutableMapping
	from collections.abc import Sequence
	except ModuleNotFoundError:
	from collections import MutableMapping
	from collections import Sequence

	files = glob.glob("jsons/*.json")

	def get_paths(source):
	paths = []
	if isinstance(source, MutableMapping): # found a dict-like structure...
	for k, v in source.items(): # iterate over it; Python 2.x: source.iteritems()
	paths.append([k]) # add the current child path
	paths += [[k] + x for x in get_paths(v)] # get sub-paths, extend with the current
	# else, check if a list-like structure, remove if you don't want list paths included
	elif isinstance(source, Sequence) and not isinstance(source, str):
	# Python 2.x: use basestring instead of str ^
	for i, v in enumerate(source):
	paths.append([i])
	paths += [[i] + x for x in get_paths(v)] # get sub-paths, extend with the current
	return paths

	path_weights = {}

	for file_path in files:
	with open(file_path) as file:
	data = json.load(file)
	paths = get_paths(data)
	for path in paths:
	path_all_str = [str(int) for int in path]
	key = '.'.join(path_all_str)
	if key not in path_weights:
	path_weights[key] = 1
	else:
	path_weights[key] += 1



	path_weights = dict(sorted(path_weights.items(), key=lambda item: item[1], reverse=True))

	print('Path weights:')

	pprint.pprint(path_weights, indent=2, sort_dicts=False)

	print('Total paths:', len(path_weights.keys()))