jq Cheet Sheet

jq-cheetsheet.md

Processing JSON using jq

jq is useful to slice, filter, map and transform structured json data.

Installing jq

On Mac OS

brew install jq

On AWS Linux

Not available as yum install on our current AMI. It should be on the latest AMI though: https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/

Installing from the source proved to be tricky.

Useful arguments

When running jq, the following arguments may become handy:

Argument	Description
--version	Output the jq version and exit with zero.
--sort-keys	Output the fields of each object with the keys in sorted order.

Basic concepts

The syntax for jq is pretty coherent:

Syntax	Description
,	Filters separated by a comma will produce multiple independent outputs
?	Will ignores error if the type is unexpected
[]	Array construction
{}	Object construction
+	Concatenate or Add
-	Difference of sets or Substract
length	Size of selected element
|	Pipes are used to chain commands in a similar fashion than bash

Dealing with json objects

Description	Command
Display all keys	jq 'keys'
Adds + 1 to all items	jq 'map_values(.+1)'
Delete a key	jq 'del(.foo)'
Convert an object to array	to_entries | map([.key, .value])

Dealing with fields

Description	Command
Concatenate two fields	fieldNew=.field1+' '+.field2

Dealing with json arrays

Slicing and Filtering

Description	Command
All	jq .[]
First	jq '.[0]'
Range	jq '.[2:4]'
First 3	jq '.[:3]'
Last 2	jq '.[-2:]'
Before Last	jq '.[-2]'
Select array of int by value	jq 'map(select(. >= 2))'
Select array of objects by value	** jq '.[] | select(.id == "second")'**
Select by type	** jq '.[] | numbers' ** with type been arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars

Mapping and Transforming

Description	Command
Add + 1 to all items	jq 'map(.+1)'
Delete 2 items	jq 'del(.[1, 2])'
Concatenate arrays	jq 'add'
Flatten an array	jq 'flatten'
Create a range of numbers	jq '[range(2;4)]'
Display the type of each item	jq 'map(type)'
Sort an array of basic type	jq 'sort'
Sort an array of objects	jq 'sort_by(.foo)'
Group by a key - opposite to flatten	jq 'group_by(.foo)'
Minimun value of an array	jq 'min' .See also min, max, min_by(path_exp), max_by(path_exp)
Remove duplicates	jq 'unique' or jq 'unique_by(.foo)' or jq 'unique_by(length)'
Reverse an array	jq 'reverse'

I wish these questions would get posted to SoF so I get credit :)

echo '{
  "k1": "rv1",
  "k2": "rv2",
  "k3": [
    {
      "k31": "rv311",
      "k32": "rv312"
    },
    {
      "k31": "rv321",
      "k32": "rv322"
    }
  ]
}
' | jq ' . as $parent | (
    .k3 | (
        to_entries |
            map(
                "\"\($parent.k1) \($parent.k2) \(.key) \(.value.k31) \(.value.k32)\""
            )
    )
) | join("\n")
'
"rv1 rv2 0 rv311 rv312"
"rv1 rv2 1 rv321 rv322"

@hgovindh86 how's this:

{
  "msg": {
    "blockdevices": [
      {
        "mountpoint": "/snap/core18/2246",
        "name": "loop0"
      },
      {
        "mountpoint": "/snap/core18/2253",
        "name": "loop1"
      },
      {
        "mountpoint": "/snap/core20/1169",
        "name": "loop2"
      },
      {
        "mountpoint": "/snap/lxd/21803",
        "name": "loop3"
      },
      {
        "mountpoint": "/snap/core20/1242",
        "name": "loop4"
      },
      {
        "mountpoint": "/snap/lxd/21835",
        "name": "loop5"
      },
      {
        "mountpoint": "/snap/snapd/14066",
        "name": "loop6"
      },
      {
        "mountpoint": "/snap/snapd/13640",
        "name": "loop7"
      },
      {
        "children": [
          {
            "mountpoint": "/boot/efi",
            "name": "sda1"
          },
          {
            "mountpoint": "/boot",
            "name": "sda2"
          },
          {
            "children": [
              {
                "children": [
                  {
                    "mountpoint": "/",
                    "name": "ubuntu--vg-ubuntu--lv"
                  },
                  {
                    "mountpoint": null,
                    "name": "ubuntu--vg-clean"
                  }
                ],
                "mountpoint": null,
                "name": "ubuntu--vg-ubuntu--lv-real"
              },
              {
                "children": [
                  {
                    "mountpoint": null,
                    "name": "ubuntu--vg-clean"
                  }
                ],
                "mountpoint": null,
                "name": "ubuntu--vg-clean-cow"
              }
            ],
            "mountpoint": null,
            "name": "sda3"
          }
        ],
        "mountpoint": null,
        "name": "sda"
      },
      {
        "children": [
          {
            "mountpoint": null,
            "name": "datavg-datavol1"
          }
        ],
        "mountpoint": null,
        "name": "sdb"
      }
    ]
  }
}

as foo.json

jq '.. | {m: .mountpoint?,name} | select(.m == "/") | .name' foo.json
ubuntu--vg-ubuntu--lv

The result isn't what you say it should be, but based on independent inspection it's correct. If I grep for your results:

grep -A 1 mountpoint foo.json | grep "/"
        "mountpoint": "/snap/core18/2246",
        "mountpoint": "/snap/core18/2253",
        "mountpoint": "/snap/core20/1169",
        "mountpoint": "/snap/lxd/21803",
        "mountpoint": "/snap/core20/1242",
        "mountpoint": "/snap/lxd/21835",
        "mountpoint": "/snap/snapd/14066",
        "mountpoint": "/snap/snapd/13640",
            "mountpoint": "/boot/efi",
            "mountpoint": "/boot",
                    "mountpoint": "/",

grep -A 1 mountpoint foo.json | grep -A 1 '"/"'
                    "mountpoint": "/",
                    "name": "ubuntu--vg-ubuntu--lv"

Not sure if this is the right place, but seems that you are experts.
I have this kind of json document:
jq '.' test.json

{
  "node1": {
    "name": "host001",
    "free": true
  },
  "node2": {
    "name": "host002",
    "free": false
  },
  "node3": {
    "name": "host003",
    "free": false
  },
  "node4": {
    "name": "host004",
    "free": true
  }
}

I want to obtain the main keys which nested object owns "free" field as true.
So, this is the way to get the keys I'm talking about:
jq 'keys' test.json

[
  "node1",
  "node2",
  "node3",
  "node4"
]

And this is the most close I've been:
jq '.[] | select (.free == true)' test.json

{
  "name": "host001",
  "free": true
}
{
  "name": "host004",
  "free": true
}

I don't know how to get "node1" and "node4" from selection.
Thank you in advance !

@testillano Just add

| map(.name)

🙂

jq '.[] | select (.free == true) | map(.name)' test.json
jq: error (at test.json:18): Cannot index string with string "name"

Not sure if I understood, indeed it is not the name but the upper key what I need (node1, node4).

@testillano Ah, I'm very sorry -- I was on my phone having a lunch apparently not paying enough attention. This should work:

to_entries | map(select(.value.free == true)) | map(.key)

https://jqplay.org/s/ouOA-ttrYuM

You don't need the last call to map() since it's not an array.

jq '.[] | select(.free == true).name' test.json

Alternatively you could just use map() if you want an array:

jq 'map(select(.free == true).name)' test.json

I wanted the parents (node1, node4), so @jsmucr hint is enough:

jq 'to_entries | map(select(.value.free == true)) | map(.key)' test.json 
[
  "node1",
  "node4"
]

It is difficult to tune the jq philosophy ...
Thank you all !

@testillano FYI you don't need to call map() twice:

jq 'to_entries | map(select(.value.free == true).key)'

Thank you @fearphage , I will simplify it !

Guys... how I can from json like that

[
  {
    "Key": "{ some_map_1 }",
    "Value": "{ another_map_1 }",
    "Timestamp": "date"
  },
  {
    "Key": "{ some_map_2 }",
    "Value": "{ another_map_2 }",
    "Timestamp": "date"
  },
]

Get raw data like that?

some_map_1, another_map_1
some_map_2, another_map_2

I tried something like that but I get wrong result((

$ cat test.txt | jq '.[] | .Key, ",",  .Value' -r
{ some_map_1 }
,
{ another_map_1 }
{ some_map_2 }
,
{ another_map_2 }

upd:
I used that finally:

cat test.txt |jq '.[] | {key : .Key, value: .Value}| .key+", "+.value' -r

But maybe yet another solution better?

@ksemele How about this? :-)

map([.Key,.Value] | join(", ")) | .[]

jq --raw-output '.[] | "\(.Key | gsub("^{ | }$"; "")), \(.Value | gsub("^{ | }$"; ""))"'

Hi,
I have an object and I would like to select from input_chunks all object where status.mem_size != "0b"
I tried like this .[] | select(.status.mem_size!="0b") or .input_chunks | select(.status.mem_size!="0b") but it didn't work :(
Maybe someone can help me?

{
  "storage_layer": {
    "chunks": {
      "total_chunks": 28,
      "mem_chunks": 27,
      "fs_chunks": 1,
      "fs_chunks_up": 0,
      "fs_chunks_down": 1
    }
  },
  "input_chunks": {
    "kube": {
      "status": {
        "overlimit": false,
        "mem_size": "0b",
        "mem_limit": "28.6M"
      },
      "chunks": {
        "total": 0,
        "up": 0,
        "down": 0,
        "busy": 0,
        "busy_size": "0b"
      }
    },
    "storage_backlog.1": {
      "status": {
        "overlimit": false,
        "mem_size": "0b",
        "mem_limit": "0b"
      },
      "chunks": {
        "total": 0,
        "up": 0,
        "down": 0,
        "busy": 0,
        "busy_size": "0b"
      }
    },
    "emitter_for_detect-nginx-ingress": {
      "status": {
        "overlimit": false,
        "mem_size": "0b",
        "mem_limit": "9.5M"
      },
      "chunks": {
        "total": 0,
        "up": 0,
        "down": 0,
        "busy": 0,
        "busy_size": "0b"
      }
    },
    "emitter_for_detect-aaaa-logger": {
      "status": {
        "overlimit": false,
        "mem_size": "3.0M",
        "mem_limit": "9.5M"
      },
      "chunks": {
        "total": 23,
        "up": 23,
        "down": 0,
        "busy": 23,
        "busy_size": "3.0M"
      }
    },
    "emitter_for_detect-kong-proxy-logger": {
      "status": {
        "overlimit": false,
        "mem_size": "99.6K",
        "mem_limit": "9.5M"
      },
      "chunks": {
        "total": 4,
        "up": 4,
        "down": 0,
        "busy": 4,
        "busy_size": "99.6K"
      }
    },
    "emitter_for_detect-elastic-logs": {
      "status": {
        "overlimit": false,
        "mem_size": "0b",
        "mem_limit": "9.5M"
      },
      "chunks": {
        "total": 0,
        "up": 0,
        "down": 0,
        "busy": 0,
        "busy_size": "0b"
      }
    },
    "emitter_for_detect-audit-logs": {
      "status": {
        "overlimit": false,
        "mem_size": "0b",
        "mem_limit": "9.5M"
      },
      "chunks": {
        "total": 0,
        "up": 0,
        "down": 0,
        "busy": 0,
        "busy_size": "0b"
      }
    },
    "emitter_for_detect-other": {
      "status": {
        "overlimit": false,
        "mem_size": "0b",
        "mem_limit": "9.5M"
      },
      "chunks": {
        "total": 0,
        "up": 0,
        "down": 0,
        "busy": 0,
        "busy_size": "0b"
      }
    }
  }
}

.input_chunks | to_entries | map(.value | select(.status.mem_size != "0b") | .chunks)

.input_chunks | to_entries | map(.value | select(.status.mem_size != "0b") | .chunks)

what if I would like to have keys emitter_for_detect... for everyone where mem_size != "0b" ?

@mirisu2 I'm not sure if I understand the question but maybe this...? :)

.input_chunks | to_entries | map(select(.key | startswith("emitter_for_detect")) | select(.value.status | .mem_size != "0b")) | from_entries

{
  "emitter_for_detect-aaaa-logger": {
    "status": {
      "overlimit": false,
      "mem_size": "3.0M",
      "mem_limit": "9.5M"
    },
    "chunks": {
      "total": 23,
      "up": 23,
      "down": 0,
      "busy": 23,
      "busy_size": "3.0M"
    }
  },
  "emitter_for_detect-kong-proxy-logger": {
    "status": {
      "overlimit": false,
      "mem_size": "99.6K",
      "mem_limit": "9.5M"
    },
    "chunks": {
      "total": 4,
      "up": 4,
      "down": 0,
      "busy": 4,
      "busy_size": "99.6K"
    }
  }
}

Yes. You got it right!
But I got {}

Maybe old jq version? Try https://jqplay.org/.

I already tried on jqplay. The result is the same. :(

That's weird then.

https://jqplay.org/s/rk2-UauouuV

Puzzle: how do I go from this:

{
  "widgets": [
    {
      "name": "foo",
      "properties": [
        "baz"
      ]
    },
    {
      "name": "bar"
    }
  ]
}

to this

{
  "widgets": [
    {
      "name": "foo",
      "properties": [
        "baz",
        "boo"
      ]
    },
    {
      "name": "bar"
    }
  ]
}

Or, is it possible to I add an item into an internal array?

@notorand-it Try this:

. as $root |
($root.widgets[] | select(.name != "foo")) as $others |
($root.widgets[] | select(.name == "foo") * { properties: (.properties + ["boo"])}) as $newfoo |
$root * { widgets: ([$newfoo] + [$others]) }

I'm pretty sure it can be done some more sophisticated way, but this works too and is easy to understand.

Puzzle: how do I go from this:

{ "widgets": [ ...
  "properties":["baz"]
  ...

to this

{ "widgets": [...
   "properties": ["baz","boo"]
...

Or, is it possible to I add an item into an internal array?

This is a good one to understand how jq can manipulate an object directly:

jq '.widgets[0].properties += ["boo"]'

This works because adding an array to another array concatenates the elements:

jq -cn '[1]+[2]'
[1,2]
jq -cn '["baz"]+["boo"]'
["baz","boo"]

Hoping I found the right place... I've been trying to figure out if there's a way to use jq in order to "flatten" nested json into simple dot-delimited "path.to.key=value" lines.

For example, given the following input:

user@shell:~$ cat example.json
{
  "data": {
    "object": { 
      "user": {
        "id":1,
        "range": [-255,0,255],
        "notation":"big-O",
        "details": {
          "lat":0.000,"long":0.000,"time":42
        }
      },
      "groups":[
        {"id":2,"name": "foo"},
        {"id":3,"name": "bar"}
      ]
    },
    "metdata": {
      "list": [
        [ [1,42],[3.14, 98.6] ], 
        [ 3, 6, 9, "low" ],
        [{"x":1,"y":-1}]
      ],
      "ugly_nest": {"depth":{"test": true} }
    }
  },
  "log":"123abc"
}

Is there a jq one-liner I could use to transform into this output?

user@shell:~$ python flatten.py $(cat example.json)
data.object.user.id=1
data.object.user.range[0]=-255
data.object.user.range[1]=0
data.object.user.range[2]=255
data.object.user.notation=big-O
data.object.user.details.lat=0.0
data.object.user.details.long=0.0
data.object.user.details.time=42
data.object.groups[0].id=2
data.object.groups[0].name=foo
data.object.groups[1].id=3
data.object.groups[1].name=bar
data.metdata.list[0][0][0]=1
data.metdata.list[0][0][1]=42
data.metdata.list[0][0][1][0]=3.14
data.metdata.list[0][0][1][1]=98.6
data.metdata.list[1][0]=3
data.metdata.list[1][1]=6
data.metdata.list[1][2]=9
data.metdata.list[1][3]=low
data.metdata.list[2][].x=1
data.metdata.list[2][].y=-1
data.metdata.ugly_nest.depth.test=True
log=123abc

I wrote a python script to do this, but figured this could be done with jq and/or other bash built-ins.

Tried a ton of different things similar to the example below but can't seem to figure out how to join each nested instance properly:

cat example.json | jq '.. | to_entries? | [.key , .value] | join("=")'

Thanks for any help!

Did you check this StackOverflow question? It seems to match what you want.

@bfontaine - thank you! This helps tremendously. This accomplishes the flattening aspect quite well, I can tweak to get formatting/output from here. Thank you very much!

Hello! With the help of some of the tricks shown here, I came up with a way to transform my JSON. The result is on JqPlay.

But the resulting Jq command

jq 'to_entries | .[] | {"rs": .key, "chrom":.value.chrom, "pos":.value.pos, "a1": .value.alleles | keys.[0], "a2": .value.alleles | keys | .[1:] | join(",") } | [.chrom, .pos, .a1, .a2] | @tsv'

will not run on the commandline

jq: error: syntax error, unexpected '[', expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>

Any help appreciated.

@mfasold I have no trouble running this command with your input from jqPlay.

$ cat test | jq 'to_entries | .[] | {"rs": .key, "chrom":.value.chrom, "pos":.value.pos, "a1": .value.alleles | keys.[0], "a2": .value.alleles | keys | .[1:] | join(",") } | [.chrom, .pos, .a1, .a2] | @tsv'
"1\t209721440\tC\tT"
"1\t111734975\tC\tG,T"
$ jq --version
jq-1.7.1

Thank you! I had jq 1.6 installed on two machines, leading me to believe this would be the newest version. Indeed, it works in jq version 1.7.1. Help much appreciated!

@mfasold FYI this could be simplified by removing the intermediate object:

to_entries[]
| [
  .value.chrom, 
  .value.pos, 
  (.value.alleles | keys[0]),
  (.value.alleles | keys[1:] | join(","))
]
| @tsv

@fearphage Thank you! TIL about the () operator.