jq examples

README.md

Given a JSON file with a structure like this:

{
  "topic": {
    "cleanup_policy": "delete",
    "partitions": [
      {
        "consumer_groups": [
          {
            "group_name": "activity-feed-service",
            "offset": 26764
          },
          {
            "group_name": "adjustment-report-service",
            "offset": 26764
          }
        ],
        "earliest_offset": 0,
        "isr": 3,
        "latest_offset": 26764,
        "partition": 0,
        "size": 10578495
      },
      {
        "consumer_groups": [
          {
            "group_name": "activity-feed-service",
            "offset": 26329
          },
          {
            "group_name": "adjustment-report-service",
            "offset": 26329
          }
        ],
        "earliest_offset": 0,
        "isr": 3,
        "latest_offset": 26329,
        "partition": 5,
        "size": 10418206
      }
    ],
    "replication": 3,
    "retention_bytes": -1,
    "retention_hours": -1,
    "state": "ACTIVE",
    "tags": [],
    "topic_name": "adjustments"
  }
}

we would want to receive latest offset per partition. We can achieve that via (Windows CMD syntax)!

type adjustments.json | jq ".topic.partitions[] | \"\(.partition) \(.latest_offset)\"" | sort > latest-offset-per-partition.txt

Now, we'd want to produce list of latest offset per a given consumer group:

type adjustments.json | jq ".topic.partitions[] | {name: .consumer_groups[].group_name, partition, latest_offset} | select(.name == \"adjustment-report-service\") | \"\(.partition) \(.latest_offset)\"" | sort > adjustment-offsets.txt

Even better would be to combine it within a single command:

type adjustments.json | jq ".topic.partitions[] | .consumer_groups[] + del(.consumer_groups) | select(.group_name == \"adjustment-report-service\")"

or, if we'd just want to check which of the consumers lag behind:

type adjustments.json | jq ".topic.partitions[] | .consumer_groups[] + del(.consumer_groups) | select(.latest_offset > .offset) | { group_name, partition, lag: (.latest_offset - .offset)  }"

AGGREGATING-OFFSETS-TO-STRING.md

When we have source.json file with a following structure:

{
    "groupId": "bla-bla-bla",
    "members": 25,
    "topics": 25,
    "simple": false,
    "partitionAssignor": "range",
    "state": "STABLE",
    "coordinator": {
        "id": 36,
        "host": "192.168.0.53",
        "port": 14493
    },
    "messagesBehind": 0,
    "partitions": [
        {
            "topic": "topic-name",
            "partition": 9,
            "currentOffset": 4556,
            "endOffset": 4556,
            "messagesBehind": 0,
            "consumerId": "consumer-blablabla",
            "host": "/10.40.2.220"
        },
...

we want to have a string with data like

0,1476;1,1248;4,6798;...

jq  -r "[.partitions[] | select(.topic == \"payment-traces\") | [.partition,.currentOffset] | join(\",\") ] | sort | join(\";\")" source.json

OLD SOLUTION:

The way to go is

jq  ".partitions[] | select(.topic == \"topic-name\") | [.partition,.currentOffset] | join(\",\") " source.json | jq -s -r "join(\";\")" 

Biggest challenge was to move from stream of strings to an array that could be fed to "join", so I had to use "--slurp" mode for this (see -s option of the second invocation)

AGGREGATING-OSM-WAYS-BY-NAME.md

If we have a way.json file, generated with [github.com/equalStreetNames/](this project), we can aggregate ways, that (potentially) refer to same street, via this query:

jq ".elements | map({name: .tags.name, id: .id}) | map (select(.name!=null)) | group_by(.name) | map({name: .[0].name, ids: map(.id)}) | map (select(.ids | length > 1))" way.json 

MAP-STRUCTS-BY-KEY.md

Given a JSON file with a structure of

[
  {
    "id": "8250598f-c519-4f8f-a41f-ba3cd9d0ec2f",
    "player_id": "323beeb0-3a2c-43be-bebc-85faf99cd08d"
  },
  {
    "id": "bc20014b-eb4a-4ccd-b7a8-094e851f7696",
    "player_id": "28546f48-5d80-4b2a-acf3-d79ad9dbba68"
  },
  {
    "id": "23a2ca23-1426-4dd3-aaf8-6392503c672c",
    "player_id": "28546f48-5d80-4b2a-acf3-d79ad9dbba68"
  }
]

aggregate all ids by player_id such that result looks like

{
  "323beeb0-3a2c-43be-bebc-85faf99cd08d": [
    "8250598f-c519-4f8f-a41f-ba3cd9d0ec2f",
    "84ac177c-c9ea-45bf-b33b-6e9997359bf4",
    "36a05408-dd75-4d4b-879e-c398937517ba"
  ],
  "28546f48-5d80-4b2a-acf3-d79ad9dbba68": [
    "bc20014b-eb4a-4ccd-b7a8-094e851f7696",
    "23a2ca23-1426-4dd3-aaf8-6392503c672c",
    "ec23fb34-598b-4314-9f79-2f29b5baa077",
    "65d2cd27-14f2-4adb-9dfe-74f46ecc43d1",
    "9be25cfc-50ba-4b25-917a-ab3b0f3af1cf"
  ],
  "7a527d97-c038-49a2-a0c5-6b813dab734e": [
    "ae314f41-f5fa-438a-94c7-1db516ed269c",
    "cf350fe1-611f-48b0-9d07-71dc5f9aa9b3",
    "b6fbeccc-55f4-4e18-9b17-5f5a35d477d9",
    "18a143d6-1cb1-4567-97d0-d6a0497e9b7b"
  ]
}

Solution:

map-by-player.jq:

reduce .[] as $line
    ({};
     $line.player_id as $user
     | .[$user] as $current
     | ($current + [$line.id]) as $next
     | . + { ($user): ($next) }
    )

command:

jq -f map-by-player.jq res-with-player.json >reservations-by-player.json