Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
jq Cheet Sheet

Processing JSON using jq

jq is useful to slice, filter, map and transform structured json data.

Installing jq

On Mac OS

brew install jq

On AWS Linux

Not available as yum install on our current AMI. It should be on the latest AMI though: https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/

Installing from the source proved to be tricky.

Useful arguments

When running jq, the following arguments may become handy:

Argument Description
--version Output the jq version and exit with zero.
--sort-keys Output the fields of each object with the keys in sorted order.

Basic concepts

The syntax for jq is pretty coherent:

Syntax Description
, Filters separated by a comma will produce multiple independent outputs
? Will ignores error if the type is unexpected
[] Array construction
{} Object construction
+ Concatenate or Add
- Difference of sets or Substract
length Size of selected element
| Pipes are used to chain commands in a similar fashion than bash

Dealing with json objects

Description Command
Display all keys jq 'keys'
Adds + 1 to all items jq 'map_values(.+1)'
Delete a key jq 'del(.foo)'
Convert an object to array to_entries | map([.key, .value])

Dealing with fields

Description Command
Concatenate two fields fieldNew=.field1+' '+.field2

Dealing with json arrays

Slicing and Filtering

Description Command
All jq .[]
First jq '.[0]'
Range jq '.[2:4]'
First 3 jq '.[:3]'
Last 2 jq '.[-2:]'
Before Last jq '.[-2]'
Select array of int by value jq 'map(select(. >= 2))'
Select array of objects by value ** jq '.[] | select(.id == "second")'**
Select by type ** jq '.[] | numbers' ** with type been arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars

Mapping and Transforming

Description Command
Add + 1 to all items jq 'map(.+1)'
Delete 2 items jq 'del(.[1, 2])'
Concatenate arrays jq 'add'
Flatten an array jq 'flatten'
Create a range of numbers jq '[range(2;4)]'
Display the type of each item jq 'map(type)'
Sort an array of basic type jq 'sort'
Sort an array of objects jq 'sort_by(.foo)'
Group by a key - opposite to flatten jq 'group_by(.foo)'
Minimun value of an array jq 'min' .See also min, max, min_by(path_exp), max_by(path_exp)
Remove duplicates jq 'unique' or jq 'unique_by(.foo)' or jq 'unique_by(length)'
Reverse an array jq 'reverse'
@TECH7Fox
Copy link

TECH7Fox commented Oct 21, 2021

Thank you, didn't know about the startwith. Its working now.

@pandiloko
Copy link

pandiloko commented Nov 9, 2021

Selecting multiple fields and output them in one line per element. Additionally:

  • round to two decimal precision
  • convert size field from bytes to megabytes and add the literal string "MB"
  • convert UNIX timestamp to date

jq 'def roundit: .*100.0|round/100.0;.[]|[.name,(.size/1024/1024|roundit|tostring)+" MB",(.mtime|todate)]|join(" - ")'

Sample input:

[
 {
   "size": 50176,
   "type": "file",
   "name": "visio-test.vsdx",
   "mtime": 1614363948
 },
 {
   "size": 6990464,
   "type": "file",
   "name": "zoc8023_x64.exe",
   "mtime": 1620808178
 }
]

Sample output:

"visio-test.vsdx - 0.05 MB - 2021-02-26T18:25:48Z"
"zoc8023_x64.exe - 6.67 MB - 2021-05-12T08:29:38Z"

@hgovindh86
Copy link

hgovindh86 commented Nov 26, 2021

how do I search for a string inside a list of dictionaries of lists?

"msg": {
"blockdevices": [
{
"mountpoint": "/snap/core18/2246",
"name": "loop0"
},
{
"mountpoint": "/snap/core18/2253",
"name": "loop1"
},
{
"mountpoint": "/snap/core20/1169",
"name": "loop2"
},
{
"mountpoint": "/snap/lxd/21803",
"name": "loop3"
},
{
"mountpoint": "/snap/core20/1242",
"name": "loop4"
},
{
"mountpoint": "/snap/lxd/21835",
"name": "loop5"
},
{
"mountpoint": "/snap/snapd/14066",
"name": "loop6"
},
{
"mountpoint": "/snap/snapd/13640",
"name": "loop7"
},
{
"children": [
{
"mountpoint": "/boot/efi",
"name": "sda1"
},
{
"mountpoint": "/boot",
"name": "sda2"
},
{
"children": [
{
"children": [
{
"mountpoint": "/",
"name": "ubuntu--vg-ubuntu--lv"
},
{
"mountpoint": null,
"name": "ubuntu--vg-clean"
}
],
"mountpoint": null,
"name": "ubuntu--vg-ubuntu--lv-real"
},
{
"children": [
{
"mountpoint": null,
"name": "ubuntu--vg-clean"
}
],
"mountpoint": null,
"name": "ubuntu--vg-clean-cow"
}
],
"mountpoint": null,
"name": "sda3"
}
],
"mountpoint": null,
"name": "sda"
},
{
"children": [
{
"mountpoint": null,
"name": "datavg-datavol1"
}
],
"mountpoint": null,
"name": "sdb"
}
]
}
}

I want to search for the element "mountpoint": "/" and get the name associated with that which is sda

@jsmucr
Copy link

jsmucr commented Nov 26, 2021

@hgovindh86 This is as close as I could get:

# Save the whole document for later
. as $doc |
# Get all paths there are (each path turns into an array of path elements) as $p1
paths(scalars) as $p1 |
# Strip the last element off every path and store the result as $p2
$p1[0:($p1 | length-1)] as $p2 |
# With the whole document
$doc |
# Get the object at each of the paths (for example {"mountpoint": null,"name": "datavg-datavol1"})
getpath($p2) |
# Select objects with .mountpoint being "/"
select(.mountpoint == "/") |
# Get the name of each such mountpoint
.name
"ubuntu--vg-ubuntu--lv"
"ubuntu--vg-ubuntu--lv"

@hgovindh86
Copy link

hgovindh86 commented Nov 26, 2021

@jsmucr, yeah.. I too could not make the right query to extract the drive name like sda, sdb etc..

@alexsimonsays
Copy link

alexsimonsays commented Dec 25, 2021

Hello,

I have a JSON code in the following format, and I'm having trouble figuring out how to use JQ to call information from the nested dictionaries within the objects within the list - if that makes sense. The list is the entire Value of the highest dictionary. Here is a sample of the structure:

{
"product" :
[
{
"A_key1" : "valueA1",
"A_key2": "valueA2",
"A_key3": {"keyA3x_nested" : "valueA3x_nested", "keyA3y_nested" : "valueA3y_nested"}
},
{
"B_key1" : "valueB1",
"B_key2" : "valueB2",
"B_key3" : {"keyB3x_nested" : "valueB3x_nested", "keyB3y_nested" : "valueB3y_nested"}
},
]
}

What I'm trying to do is select only some items from different positions within the hierarchy as follows:

{ "product": { "A_key1": "valueA1", "A_key3": {"keyA3y_nested": "valueA3y_nested"} "B_key1": "valueB1", "B_key3": {"keyB3y_nested": "valueB3y_nested"} } }

So far, I am able to do this on the command prompt:
curl https://sample_api | jq '{product : .product[0]}'

^ That returns the entire object for "A" which includes all 3 of A's keys, but how do I just pull information selectively from lower in the hierarchy?

Thank you.

@a-h
Copy link

a-h commented Jan 4, 2022

List the names of all CloudFormation templates in your AWS Account's region in plaintext.

aws cloudformation list-stacks | jq --raw-output '.StackSummaries | map(.StackName) | .[]'

And get the template.

aws cloudformation get-template --stack-name=name | jq --raw-output ".TemplateBody | ."

@GlebTasks1
Copy link

GlebTasks1 commented May 13, 2022

Hi guys!

Please help me to figure that one out...

I want to get id where name is - terraform-02

jq '.[] | {"name" : "terraform-02"}'
jq '.[] | map(select(.name == "terraform-02"))' - doesn't work

"ssh_keys": [
{
"id": ...,
"public_key": "...",
"name": "terraform-02",
"fingerprint": "."
},
{
"id": ...,
"public_key": "...",
"name": "terraform-02",
"fingerprint": "..."
},
]

@chb0github
Copy link

chb0github commented May 13, 2022

Remove the .[] |

The input to map must be an array. When you do .[] you're dearraying it, or whatever the term is

@AYehia0
Copy link

AYehia0 commented May 16, 2022

I wonder how can I convert this to array of objects,
I have this :

{
   "code":200,
   "status":"OK",
   "data":[
      {
         "timings":{
            "Fajr":"03:37 (EET)",
            "Sunrise":"05:13 (EET)",
            "Dhuhr":"11:53 (EET)",
            "Asr":"15:30 (EET)",
            "Sunset":"18:34 (EET)",
            "Maghrib":"18:34 (EET)",
            "Isha":"19:59 (EET)",
            "Imsak":"03:27 (EET)",
            "Midnight":"23:53 (EET)"
         },
         "date":{
            "readable":"01 May 2022",
            "timestamp":"1651388461",
            "gregorian":{
               "date":"01-05-2022",
               "format":"DD-MM-YYYY",
               "day":"01",
               "weekday":{
                  "en":"Sunday"
               },
               "month":{
                  "number":5,
                  "en":"May"
               },
               "year":"2022",
               "designation":{
                  "abbreviated":"AD",
                  "expanded":"Anno Domini"
               }
            },
            "hijri":{
               "date":"29-09-1443",
               "format":"DD-MM-YYYY",
               "day":"29",
               "weekday":{
                  "en":"Al Ahad",
                  "ar":"الاحد"
               },
               "month":{
                  "number":9,
                  "en":"Ramaḍān",
                  "ar":"رَمَضان"
               },
               "year":"1443",
               "designation":{
                  "abbreviated":"AH",
                  "expanded":"Anno Hegirae"
               },
               "holidays":[
                  "Lailat-ul-Qadr"
               ]
            }
         },
         "meta":{
            "latitude":30.5503,
            "longitude":31.0106,
            "timezone":"Africa/Cairo",
            "method":{
               "id":5,
               "name":"Egyptian General Authority of Survey",
               "params":{
                  "Fajr":19.5,
                  "Isha":17.5
               },
               "location":{
                  "latitude":30.0444196,
                  "longitude":31.2357116
               }
            },
            "latitudeAdjustmentMethod":"ANGLE_BASED",
            "midnightMode":"STANDARD",
            "school":"STANDARD",
            "offset":{
               "Imsak":0,
               "Fajr":0,
               "Sunrise":0,
               "Dhuhr":0,
               "Asr":0,
               "Maghrib":0,
               "Sunset":0,
               "Isha":0,
               "Midnight":0
            }
         }
      },
      {
         
      }
   ]
}

I am trying to group as :

[
{
  "Fajr": "03:11 (EET)",
  "Sunrise": "04:55 (EET)",
  "Dhuhr": "11:53 (EET)",
  "Asr": "15:31 (EET)",
  "Sunset": "18:52 (EET)",
  "Maghrib": "18:52 (EET)",
  "Isha": "20:24 (EET)",
  "Imsak": "03:01 (EET)",
  "Midnight": "23:53 (EET)"
},
{
  "Fajr": "03:10 (EET)",
  "Sunrise": "04:55 (EET)",
  "Dhuhr": "11:54 (EET)",
  "Asr": "15:31 (EET)",
  "Sunset": "18:53 (EET)",
  "Maghrib": "18:53 (EET)",
  "Isha": "20:25 (EET)",
  "Imsak": "03:00 (EET)",
  "Midnight": "23:54 (EET)"
},
{
  "Fajr": "03:10 (EET)",
  "Sunrise": "04:54 (EET)",
  "Dhuhr": "11:54 (EET)",
  "Asr": "15:31 (EET)",
  "Sunset": "18:53 (EET)",
  "Maghrib": "18:53 (EET)",
  "Isha": "20:25 (EET)",
  "Imsak": "03:00 (EET)",
  "Midnight": "23:54 (EET)"
}
]

I am doing like this : jq '.data | .[] | .timings | .' file.json, it gives me objects how can i remap into a list of objects ?

@fearphage
Copy link

fearphage commented May 16, 2022

I am doing like this : jq '.data | .[] | .timings | .' file.json, it gives me objects how can i remap into a list of objects ?

jq '.data | map(.timings)' file.json

@chb0github
Copy link

chb0github commented May 16, 2022

jq '.data | map(.timings) | map(select(.))' file.json

Because you have an empty object in your array, this would produce a null and so you need to filter it out. With your input data (and you only have 1 matching element) I get

[
  {
    "Asr": "15:30 (EET)",
    "Dhuhr": "11:53 (EET)",
    "Fajr": "03:37 (EET)",
    "Imsak": "03:27 (EET)",
    "Isha": "19:59 (EET)",
    "Maghrib": "18:34 (EET)",
    "Midnight": "23:53 (EET)",
    "Sunrise": "05:13 (EET)",
    "Sunset": "18:34 (EET)"
  }
]

@AYehia0
Copy link

AYehia0 commented May 16, 2022

thank you all @chb0github and @fearphage 😄
I was playing around and did something similar (added the day too) : jq '.data | to_entries | map_values(.value + {day: .key}) | map(.timings + {day : .day})' exp.json

  
  [{
    "Fajr": "03:10 (EET)",
    "Sunrise": "04:55 (EET)",
    "Dhuhr": "11:54 (EET)",
    "Asr": "15:31 (EET)",
    "Sunset": "18:53 (EET)",
    "Maghrib": "18:53 (EET)",
    "Isha": "20:25 (EET)",
    "Imsak": "03:00 (EET)",
    "Midnight": "23:54 (EET)",
    "day": 29
  },
  {
    "Fajr": "03:10 (EET)",
    "Sunrise": "04:54 (EET)",
    "Dhuhr": "11:54 (EET)",
    "Asr": "15:31 (EET)",
    "Sunset": "18:53 (EET)",
    "Maghrib": "18:53 (EET)",
    "Isha": "20:25 (EET)",
    "Imsak": "03:00 (EET)",
    "Midnight": "23:54 (EET)",
    "day": 30
  }
]

@AYehia0
Copy link

AYehia0 commented May 17, 2022

jq '.data | map(.timings) | map(select(.))' file.json

Because you have an empty object in your array, this would produce a null and so you need to filter it out. With your input data (and you only have 1 matching element) I get

[
  {
    "Asr": "15:30 (EET)",
    "Dhuhr": "11:53 (EET)",
    "Fajr": "03:37 (EET)",
    "Imsak": "03:27 (EET)",
    "Isha": "19:59 (EET)",
    "Maghrib": "18:34 (EET)",
    "Midnight": "23:53 (EET)",
    "Sunrise": "05:13 (EET)",
    "Sunset": "18:34 (EET)"
  }
]

I wonder how can i remove the (EET) from each value of all the objects inside the list ?
can i do something like this ?

  [{
    "Fajr": "03:10",
    "Sunrise": "04:54",
    "Dhuhr": "11:54",
    "Asr": "15:31",
    "Sunset": "18:53",
    "Maghrib": "18:53",
    "Isha": "20:25",
    "Imsak": "03:00",
    "Midnight": "23:54",
    "day": 30
  }]

here is what i tried (but i lose the key lol) : jq '.data | map(.timings) | map(.[] | split(" (")[0])' :

[
  "20:25",
  "03:00",
  "23:54",
  "03:10",
  "04:54",
  "11:54",
  "15:31",
  "18:53",
  "18:53",
  "20:25",
  "03:00",
  "23:54"
]

@jsmucr
Copy link

jsmucr commented May 17, 2022

@AYehia0

jq '.[] | to_entries | map({key, value: (.value[:5])}) | [ from_entries ]'

@chb0github
Copy link

chb0github commented May 17, 2022

@AYehia0 If you assume fixed length time format:

jq '.data | 
    map(.timings) | 
    map(select(.)) | 
    map(
        to_entries | 
            map({
                (.key): .value[:5]
            }) | add
    )' foo.json
[
  {
    "Asr": "15:30",
    "Dhuhr": "11:53",
    "Fajr": "03:37",
    "Imsak": "03:27",
    "Isha": "19:59",
    "Maghrib": "18:34",
    "Midnight": "23:53",
    "Sunrise": "05:13",
    "Sunset": "18:34"
  }
]

@amitj-humanic
Copy link

amitj-humanic commented May 27, 2022

Guys, I have this json,

{ "k1": "rv1", "k2": "rv2", "k3": [ { "k31": "rv311", "k32": "rv312" }, { "k31": "rv321", "k32": "rv322" } ] }
I can make it flat by using following jq command:

jq '. as $parent | .k3[] | [$parent.k1, $parent.k2, .k31, .k32]| join(" ")'

this returns

"rv1 rv2 rv311 rv312"
"rv1 rv2 rv321 rv322"

I am trying to figure out how to get jq to output the following:

"rv1 rv2 0 rv311 rv312"
"rv1 rv2 1 rv321 rv322"

where 0 and 1 are the indexes.
So, How can I get the index of the k3 array for each of the sub-dictionaries,
or if is there any better way to get this result.

@chb0github
Copy link

chb0github commented May 27, 2022

I wish these questions would get posted to SoF so I get credit :)

echo '{
  "k1": "rv1",
  "k2": "rv2",
  "k3": [
    {
      "k31": "rv311",
      "k32": "rv312"
    },
    {
      "k31": "rv321",
      "k32": "rv322"
    }
  ]
}
' | jq ' . as $parent | (
    .k3 | (
        to_entries |
            map(
                "\"\($parent.k1) \($parent.k2) \(.key) \(.value.k31) \(.value.k32)\""
            )
    )
) | join("\n")
'
"rv1 rv2 0 rv311 rv312"
"rv1 rv2 1 rv321 rv322"

@chb0github
Copy link

chb0github commented May 27, 2022

@hgovindh86 how's this:

{
  "msg": {
    "blockdevices": [
      {
        "mountpoint": "/snap/core18/2246",
        "name": "loop0"
      },
      {
        "mountpoint": "/snap/core18/2253",
        "name": "loop1"
      },
      {
        "mountpoint": "/snap/core20/1169",
        "name": "loop2"
      },
      {
        "mountpoint": "/snap/lxd/21803",
        "name": "loop3"
      },
      {
        "mountpoint": "/snap/core20/1242",
        "name": "loop4"
      },
      {
        "mountpoint": "/snap/lxd/21835",
        "name": "loop5"
      },
      {
        "mountpoint": "/snap/snapd/14066",
        "name": "loop6"
      },
      {
        "mountpoint": "/snap/snapd/13640",
        "name": "loop7"
      },
      {
        "children": [
          {
            "mountpoint": "/boot/efi",
            "name": "sda1"
          },
          {
            "mountpoint": "/boot",
            "name": "sda2"
          },
          {
            "children": [
              {
                "children": [
                  {
                    "mountpoint": "/",
                    "name": "ubuntu--vg-ubuntu--lv"
                  },
                  {
                    "mountpoint": null,
                    "name": "ubuntu--vg-clean"
                  }
                ],
                "mountpoint": null,
                "name": "ubuntu--vg-ubuntu--lv-real"
              },
              {
                "children": [
                  {
                    "mountpoint": null,
                    "name": "ubuntu--vg-clean"
                  }
                ],
                "mountpoint": null,
                "name": "ubuntu--vg-clean-cow"
              }
            ],
            "mountpoint": null,
            "name": "sda3"
          }
        ],
        "mountpoint": null,
        "name": "sda"
      },
      {
        "children": [
          {
            "mountpoint": null,
            "name": "datavg-datavol1"
          }
        ],
        "mountpoint": null,
        "name": "sdb"
      }
    ]
  }
}

as foo.json

jq '.. | {m: .mountpoint?,name} | select(.m == "/") | .name' foo.json
ubuntu--vg-ubuntu--lv

The result isn't what you say it should be, but based on independent inspection it's correct. If I grep for your results:

grep -A 1 mountpoint foo.json | grep "/"
        "mountpoint": "/snap/core18/2246",
        "mountpoint": "/snap/core18/2253",
        "mountpoint": "/snap/core20/1169",
        "mountpoint": "/snap/lxd/21803",
        "mountpoint": "/snap/core20/1242",
        "mountpoint": "/snap/lxd/21835",
        "mountpoint": "/snap/snapd/14066",
        "mountpoint": "/snap/snapd/13640",
            "mountpoint": "/boot/efi",
            "mountpoint": "/boot",
                    "mountpoint": "/",

grep -A 1 mountpoint foo.json | grep -A 1 '"/"'
                    "mountpoint": "/",
                    "name": "ubuntu--vg-ubuntu--lv"

@testillano
Copy link

testillano commented Jun 6, 2022

Not sure if this is the right place, but seems that you are experts.
I have this kind of json document:
jq '.' test.json

{
  "node1": {
    "name": "host001",
    "free": true
  },
  "node2": {
    "name": "host002",
    "free": false
  },
  "node3": {
    "name": "host003",
    "free": false
  },
  "node4": {
    "name": "host004",
    "free": true
  }
}

I want to obtain the main keys which nested object owns "free" field as true.
So, this is the way to get the keys I'm talking about:
jq 'keys' test.json

[
  "node1",
  "node2",
  "node3",
  "node4"
]

And this is the most close I've been:
jq '.[] | select (.free == true)' test.json

{
  "name": "host001",
  "free": true
}
{
  "name": "host004",
  "free": true
}

I don't know how to get "node1" and "node4" from selection.
Thank you in advance !

@jsmucr
Copy link

jsmucr commented Jun 6, 2022

@testillano Just add

| map(.name)

🙂

@testillano
Copy link

testillano commented Jun 6, 2022

jq '.[] | select (.free == true) | map(.name)' test.json
jq: error (at test.json:18): Cannot index string with string "name"

Not sure if I understood, indeed it is not the name but the upper key what I need (node1, node4).

@jsmucr
Copy link

jsmucr commented Jun 6, 2022

@testillano Ah, I'm very sorry -- I was on my phone having a lunch apparently not paying enough attention. This should work:

to_entries | map(select(.value.free == true)) | map(.key)

https://jqplay.org/s/ouOA-ttrYuM

@fearphage
Copy link

fearphage commented Jun 6, 2022

You don't need the last call to map() since it's not an array.

jq '.[] | select(.free == true).name' test.json

Alternatively you could just use map() if you want an array:

jq 'map(select(.free == true).name)' test.json

@testillano
Copy link

testillano commented Jun 6, 2022

I wanted the parents (node1, node4), so @jsmucr hint is enough:

jq 'to_entries | map(select(.value.free == true)) | map(.key)' test.json 
[
  "node1",
  "node4"
]

It is difficult to tune the jq philosophy ...
Thank you all !

@fearphage
Copy link

fearphage commented Jun 6, 2022

@testillano FYI you don't need to call map() twice:

jq 'to_entries | map(select(.value.free == true).key)'

@testillano
Copy link

testillano commented Jun 7, 2022

Thank you @fearphage , I will simplify it !

@ksemele
Copy link

ksemele commented Jun 10, 2022

Guys... how I can from json like that

[
  {
    "Key": "{ some_map_1 }",
    "Value": "{ another_map_1 }",
    "Timestamp": "date"
  },
  {
    "Key": "{ some_map_2 }",
    "Value": "{ another_map_2 }",
    "Timestamp": "date"
  },
]

Get raw data like that?

some_map_1, another_map_1
some_map_2, another_map_2

I tried something like that but I get wrong result((

$ cat test.txt | jq '.[] | .Key, ",",  .Value' -r
{ some_map_1 }
,
{ another_map_1 }
{ some_map_2 }
,
{ another_map_2 }

upd:
I used that finally:

cat test.txt |jq '.[] | {key : .Key, value: .Value}| .key+", "+.value' -r

But maybe yet another solution better?

@jsmucr
Copy link

jsmucr commented Jun 10, 2022

@ksemele How about this? :-)

map([.Key,.Value] | join(", ")) | .[]

@fearphage
Copy link

fearphage commented Jun 10, 2022

jq --raw-output '.[] | "\(.Key | gsub("^{ | }$"; "")), \(.Value | gsub("^{ | }$"; ""))"'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment