Skip to content

Instantly share code, notes, and snippets.

@olih
Last active July 13, 2024 12:46
Show Gist options
  • Save olih/f7437fb6962fb3ee9fe95bda8d2c8fa4 to your computer and use it in GitHub Desktop.
Save olih/f7437fb6962fb3ee9fe95bda8d2c8fa4 to your computer and use it in GitHub Desktop.
jq Cheet Sheet

Processing JSON using jq

jq is useful to slice, filter, map and transform structured json data.

Installing jq

On Mac OS

brew install jq

On AWS Linux

Not available as yum install on our current AMI. It should be on the latest AMI though: https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/

Installing from the source proved to be tricky.

Useful arguments

When running jq, the following arguments may become handy:

Argument Description
--version Output the jq version and exit with zero.
--sort-keys Output the fields of each object with the keys in sorted order.

Basic concepts

The syntax for jq is pretty coherent:

Syntax Description
, Filters separated by a comma will produce multiple independent outputs
? Will ignores error if the type is unexpected
[] Array construction
{} Object construction
+ Concatenate or Add
- Difference of sets or Substract
length Size of selected element
| Pipes are used to chain commands in a similar fashion than bash

Dealing with json objects

Description Command
Display all keys jq 'keys'
Adds + 1 to all items jq 'map_values(.+1)'
Delete a key jq 'del(.foo)'
Convert an object to array to_entries | map([.key, .value])

Dealing with fields

Description Command
Concatenate two fields fieldNew=.field1+' '+.field2

Dealing with json arrays

Slicing and Filtering

Description Command
All jq .[]
First jq '.[0]'
Range jq '.[2:4]'
First 3 jq '.[:3]'
Last 2 jq '.[-2:]'
Before Last jq '.[-2]'
Select array of int by value jq 'map(select(. >= 2))'
Select array of objects by value ** jq '.[] | select(.id == "second")'**
Select by type ** jq '.[] | numbers' ** with type been arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars

Mapping and Transforming

Description Command
Add + 1 to all items jq 'map(.+1)'
Delete 2 items jq 'del(.[1, 2])'
Concatenate arrays jq 'add'
Flatten an array jq 'flatten'
Create a range of numbers jq '[range(2;4)]'
Display the type of each item jq 'map(type)'
Sort an array of basic type jq 'sort'
Sort an array of objects jq 'sort_by(.foo)'
Group by a key - opposite to flatten jq 'group_by(.foo)'
Minimun value of an array jq 'min' .See also min, max, min_by(path_exp), max_by(path_exp)
Remove duplicates jq 'unique' or jq 'unique_by(.foo)' or jq 'unique_by(length)'
Reverse an array jq 'reverse'
@jsmucr
Copy link

jsmucr commented Jun 10, 2022

@ksemele How about this? :-)

map([.Key,.Value] | join(", ")) | .[]

@fearphage
Copy link

jq --raw-output '.[] | "\(.Key | gsub("^{ | }$"; "")), \(.Value | gsub("^{ | }$"; ""))"'

@mirisu2
Copy link

mirisu2 commented Jul 15, 2022

Hi,
I have an object and I would like to select from input_chunks all object where status.mem_size != "0b"
I tried like this .[] | select(.status.mem_size!="0b") or .input_chunks | select(.status.mem_size!="0b") but it didn't work :(
Maybe someone can help me?

{
  "storage_layer": {
    "chunks": {
      "total_chunks": 28,
      "mem_chunks": 27,
      "fs_chunks": 1,
      "fs_chunks_up": 0,
      "fs_chunks_down": 1
    }
  },
  "input_chunks": {
    "kube": {
      "status": {
        "overlimit": false,
        "mem_size": "0b",
        "mem_limit": "28.6M"
      },
      "chunks": {
        "total": 0,
        "up": 0,
        "down": 0,
        "busy": 0,
        "busy_size": "0b"
      }
    },
    "storage_backlog.1": {
      "status": {
        "overlimit": false,
        "mem_size": "0b",
        "mem_limit": "0b"
      },
      "chunks": {
        "total": 0,
        "up": 0,
        "down": 0,
        "busy": 0,
        "busy_size": "0b"
      }
    },
    "emitter_for_detect-nginx-ingress": {
      "status": {
        "overlimit": false,
        "mem_size": "0b",
        "mem_limit": "9.5M"
      },
      "chunks": {
        "total": 0,
        "up": 0,
        "down": 0,
        "busy": 0,
        "busy_size": "0b"
      }
    },
    "emitter_for_detect-aaaa-logger": {
      "status": {
        "overlimit": false,
        "mem_size": "3.0M",
        "mem_limit": "9.5M"
      },
      "chunks": {
        "total": 23,
        "up": 23,
        "down": 0,
        "busy": 23,
        "busy_size": "3.0M"
      }
    },
    "emitter_for_detect-kong-proxy-logger": {
      "status": {
        "overlimit": false,
        "mem_size": "99.6K",
        "mem_limit": "9.5M"
      },
      "chunks": {
        "total": 4,
        "up": 4,
        "down": 0,
        "busy": 4,
        "busy_size": "99.6K"
      }
    },
    "emitter_for_detect-elastic-logs": {
      "status": {
        "overlimit": false,
        "mem_size": "0b",
        "mem_limit": "9.5M"
      },
      "chunks": {
        "total": 0,
        "up": 0,
        "down": 0,
        "busy": 0,
        "busy_size": "0b"
      }
    },
    "emitter_for_detect-audit-logs": {
      "status": {
        "overlimit": false,
        "mem_size": "0b",
        "mem_limit": "9.5M"
      },
      "chunks": {
        "total": 0,
        "up": 0,
        "down": 0,
        "busy": 0,
        "busy_size": "0b"
      }
    },
    "emitter_for_detect-other": {
      "status": {
        "overlimit": false,
        "mem_size": "0b",
        "mem_limit": "9.5M"
      },
      "chunks": {
        "total": 0,
        "up": 0,
        "down": 0,
        "busy": 0,
        "busy_size": "0b"
      }
    }
  }
}

@jsmucr
Copy link

jsmucr commented Jul 16, 2022

.input_chunks | to_entries | map(.value | select(.status.mem_size != "0b") | .chunks)

@mirisu2
Copy link

mirisu2 commented Jul 16, 2022

.input_chunks | to_entries | map(.value | select(.status.mem_size != "0b") | .chunks)

what if I would like to have keys emitter_for_detect... for everyone where mem_size != "0b" ?

@jsmucr
Copy link

jsmucr commented Jul 16, 2022

@mirisu2 I'm not sure if I understand the question but maybe this...? :)

.input_chunks | to_entries | map(select(.key | startswith("emitter_for_detect")) | select(.value.status | .mem_size != "0b")) | from_entries
{
  "emitter_for_detect-aaaa-logger": {
    "status": {
      "overlimit": false,
      "mem_size": "3.0M",
      "mem_limit": "9.5M"
    },
    "chunks": {
      "total": 23,
      "up": 23,
      "down": 0,
      "busy": 23,
      "busy_size": "3.0M"
    }
  },
  "emitter_for_detect-kong-proxy-logger": {
    "status": {
      "overlimit": false,
      "mem_size": "99.6K",
      "mem_limit": "9.5M"
    },
    "chunks": {
      "total": 4,
      "up": 4,
      "down": 0,
      "busy": 4,
      "busy_size": "99.6K"
    }
  }
}

@mirisu2
Copy link

mirisu2 commented Jul 16, 2022

Yes. You got it right!
But I got {}

@jsmucr
Copy link

jsmucr commented Jul 17, 2022

Maybe old jq version? Try https://jqplay.org/.

@mirisu2
Copy link

mirisu2 commented Jul 17, 2022

I already tried on jqplay. The result is the same. :(

@jsmucr
Copy link

jsmucr commented Jul 17, 2022

@notorand-it
Copy link

notorand-it commented Oct 26, 2022

Puzzle: how do I go from this:

{
  "widgets": [
    {
      "name": "foo",
      "properties": [
        "baz"
      ]
    },
    {
      "name": "bar"
    }
  ]
}

to this

{
  "widgets": [
    {
      "name": "foo",
      "properties": [
        "baz",
        "boo"
      ]
    },
    {
      "name": "bar"
    }
  ]
}

Or, is it possible to I add an item into an internal array?

@jsmucr
Copy link

jsmucr commented Oct 31, 2022

@notorand-it Try this:

. as $root |
($root.widgets[] | select(.name != "foo")) as $others |
($root.widgets[] | select(.name == "foo") * { properties: (.properties + ["boo"])}) as $newfoo |
$root * { widgets: ([$newfoo] + [$others]) }

I'm pretty sure it can be done some more sophisticated way, but this works too and is easy to understand.

@twosider
Copy link

twosider commented Apr 13, 2023

Puzzle: how do I go from this:

{ "widgets": [ ...
  "properties":["baz"]
  ...

to this

{ "widgets": [...
   "properties": ["baz","boo"]
...

Or, is it possible to I add an item into an internal array?

This is a good one to understand how jq can manipulate an object directly:

jq '.widgets[0].properties += ["boo"]'

This works because adding an array to another array concatenates the elements:

jq -cn '[1]+[2]'
[1,2]
jq -cn '["baz"]+["boo"]'
["baz","boo"]

@tmprender
Copy link

Hoping I found the right place... I've been trying to figure out if there's a way to use jq in order to "flatten" nested json into simple dot-delimited "path.to.key=value" lines.

For example, given the following input:

user@shell:~$ cat example.json
{
  "data": {
    "object": { 
      "user": {
        "id":1,
        "range": [-255,0,255],
        "notation":"big-O",
        "details": {
          "lat":0.000,"long":0.000,"time":42
        }
      },
      "groups":[
        {"id":2,"name": "foo"},
        {"id":3,"name": "bar"}
      ]
    },
    "metdata": {
      "list": [
        [ [1,42],[3.14, 98.6] ], 
        [ 3, 6, 9, "low" ],
        [{"x":1,"y":-1}]
      ],
      "ugly_nest": {"depth":{"test": true} }
    }
  },
  "log":"123abc"
}

Is there a jq one-liner I could use to transform into this output?

user@shell:~$ python flatten.py $(cat example.json)
data.object.user.id=1
data.object.user.range[0]=-255
data.object.user.range[1]=0
data.object.user.range[2]=255
data.object.user.notation=big-O
data.object.user.details.lat=0.0
data.object.user.details.long=0.0
data.object.user.details.time=42
data.object.groups[0].id=2
data.object.groups[0].name=foo
data.object.groups[1].id=3
data.object.groups[1].name=bar
data.metdata.list[0][0][0]=1
data.metdata.list[0][0][1]=42
data.metdata.list[0][0][1][0]=3.14
data.metdata.list[0][0][1][1]=98.6
data.metdata.list[1][0]=3
data.metdata.list[1][1]=6
data.metdata.list[1][2]=9
data.metdata.list[1][3]=low
data.metdata.list[2][].x=1
data.metdata.list[2][].y=-1
data.metdata.ugly_nest.depth.test=True
log=123abc

I wrote a python script to do this, but figured this could be done with jq and/or other bash built-ins.

Tried a ton of different things similar to the example below but can't seem to figure out how to join each nested instance properly:

cat example.json | jq '.. | to_entries? | [.key , .value] | join("=")'

Thanks for any help!

@bfontaine
Copy link

Did you check this StackOverflow question? It seems to match what you want.

@tmprender
Copy link

@bfontaine - thank you! This helps tremendously. This accomplishes the flattening aspect quite well, I can tweak to get formatting/output from here. Thank you very much!

@mfasold
Copy link

mfasold commented Apr 16, 2024

Hello! With the help of some of the tricks shown here, I came up with a way to transform my JSON. The result is on JqPlay.

But the resulting Jq command

jq 'to_entries | .[] | {"rs": .key, "chrom":.value.chrom, "pos":.value.pos, "a1": .value.alleles | keys.[0], "a2": .value.alleles | keys | .[1:] | join(",") } | [.chrom, .pos, .a1, .a2] | @tsv'

will not run on the commandline

jq: error: syntax error, unexpected '[', expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>

Any help appreciated.

@bfontaine
Copy link

@mfasold I have no trouble running this command with your input from jqPlay.

$ cat test | jq 'to_entries | .[] | {"rs": .key, "chrom":.value.chrom, "pos":.value.pos, "a1": .value.alleles | keys.[0], "a2": .value.alleles | keys | .[1:] | join(",") } | [.chrom, .pos, .a1, .a2] | @tsv'
"1\t209721440\tC\tT"
"1\t111734975\tC\tG,T"
$ jq --version
jq-1.7.1

@mfasold
Copy link

mfasold commented Apr 16, 2024

Thank you! I had jq 1.6 installed on two machines, leading me to believe this would be the newest version. Indeed, it works in jq version 1.7.1. Help much appreciated!

@fearphage
Copy link

@mfasold FYI this could be simplified by removing the intermediate object:

to_entries[]
| [
  .value.chrom, 
  .value.pos, 
  (.value.alleles | keys[0]),
  (.value.alleles | keys[1:] | join(","))
]
| @tsv

@mfasold
Copy link

mfasold commented Apr 19, 2024

@fearphage Thank you! TIL about the () operator.

@bouchezi
Copy link

bouchezi commented May 3, 2024

Hey there,

Seems you guys are expert. I can't find my way to do what I want. I have this json array:

[
  {
    "tableReference": {
      "tableId": "Applicant"
    }
  },
  {
    "tableReference": {
      "tableId": "ApplicantBureau"
    }
  }
]

I want to transform it to a map:

{
  "tableId" : [
    "Applicant",
    "ApplicantBureau"
  ]
}

I've tried lot of things, but none is giving me the right result 😓

Any help highly appreciated 🙏

@chb0github
Copy link

chb0github commented May 3, 2024 via email

@bouchezi
Copy link

bouchezi commented May 3, 2024

Yes, I tried to use that one, but I can't figure out how to go from an array to a map and remove "tableReference"
Cause here I want to group on the key, not the value

@bouchezi
Copy link

bouchezi commented May 3, 2024

Got it working using this: jq '[{tableId: map(.tableReference.tableId)}] | add'

@chb0github
Copy link

Having looked at your original answer request (tableIds: note the plural) and the solution you have accepted for yourself, group_by would never work. It's general purpose. I mean, technically, if you're gonna skip the key and hard code it, you don't need a map - you know the array values represent what you need.

group_by is definitely a difficult function to work with as, IMO, it doesn't produce and intuitive output

JSON for how it's meant to work, along with output. I added an id field so we could clearly distinguish each object.

[
  {
    "id" : 1,
    "tableReference": {
      "tableId": "Applicant"
    }
  },
  {
    "id" : 3,
    "tableReference": {
      "tableId": "ApplicantBureau"
    }
  },
  {
    "id" : 2,
    "tableReference": {
      "tableId": "Applicant"
    }
  },
  {
    "id" : 4,
    "tableReference": {
      "tableId": "Foo"
    }
  }
]

jq:

group_by(.tableReference.tableId) | 
    map({ 
        key: .[0].tableReference.tableId, value: [.[] | .] 
    }) | 
from_entries

Output:

{
  "Applicant": [
    {
      "id": 1,
      "tableReference": {
        "tableId": "Applicant"
      }
    },
    {
      "id": 2,
      "tableReference": {
        "tableId": "Applicant"
      }
    }
  ],
  "ApplicantBureau": [
    {
      "id": 3,
      "tableReference": {
        "tableId": "ApplicantBureau"
      }
    }
  ],
  "Foo": [
    {
      "id": 4,
      "tableReference": {
        "tableId": "Foo"
      }
    }
  ]
}

You'll notice you now have 1 map with the key being the tableId? You should actually post to stack over flow - you're more likely to get help since people can get credit

@chb0github
Copy link

@tmprender - can you post your working example? The one on stack overflow doesn't go full depth

@chb0github
Copy link

chb0github commented May 9, 2024

So, I am trying to do the above @bouchezi , and it should be about this simple:

jq '[path(..)] | [map(join(".")), map( getpath(.))] ' example.json

When I do this:
jq [path(..)] | .[1:3]
I get

[
  [
    "data"
  ],
  [
    "data",
    "object"
  ]
]

and when I fetch the second element, as an example, I get:

jq:

jq 'getpath(  [     
    "data",
    "object"
  ])' example.json

produces:

{
  "user": {
    "id": 1,
    "range": [
      -255,
      0,
      255
    ],
    "notation": "big-O",
    "details": {
      "lat": 0.000,
      "long": 0.000,
      "time": 42
    }
  },
  "groups": [
    {
      "id": 2,
      "name": "foo"
    },
    {
      "id": 3,
      "name": "bar"
    }
  ]
}

So, the output from [path(..)] looks just fine.

When I do:

jq '[path(..)] | map(join(".")) example.json'

I get what I would expect (so far, so good) (sample):

[
 "data.object.user",
  "data.object.user.id",
  "data.object.user.range",
  "data.object.user.range.0",
  "data.object.user.range.1",
  "data.object.user.range.2",
  "data.object.user.notation"
]

But when I do this:

jq '[path(..)] | map(getpath(.))' example.json

I get the error:

jq: error (at example.json:27): Cannot index array with string "data"

This should work. Because if I can get this to work, I think the whole problem can be reduced to:

jq -re '[path(..)] | [map(join(".")),map(getpath(.))] | map(join("=")) | .[]'

Simplest example:

-> % jq '[path(..)] | .[3] ' example.json 
[
  "data",
  "object",
  "user"
]
(ehsm-py3.9) cbongior@cbongior-mac [09:54:38] [~/dev/ehsm] [main *]
-> % jq 'getpath([               
  "data",
  "object",
  "user"
])' example.json
{
  "id": 1,
  "range": [
    -255,
    0,
    255
  ],
  "notation": "big-O",
  "details": {
    "lat": 0.000,
    "long": 0.000,
    "time": 42
  }
}
(ehsm-py3.9) cbongior@cbongior-mac [09:55:03] [~/dev/ehsm] [main *]
-> % jq '[path(..)] | .[3] | getpath' example.json 
jq: error: getpath/0 is not defined at <top-level>, line 1:
[path(..)] | .[3] | getpath                    
jq: 1 compile error
(ehsm-py3.9) cbongior@cbongior-mac [09:55:25] [~/dev/ehsm] [main *]
-> % jq '[path(..)] | .[3] | getpath(.)' example.json
jq: error (at example.json:27): Cannot index array with string "data"

@wader
Copy link

wader commented May 9, 2024

Something like this? https://jqplay.org/s/lDs_huUTqNi

$ jq -r 'path(.. | scalars) as $p | getpath($p) as $v | "\($p|join("."))=\($v)"' example.json
data.object.user.id=1
data.object.user.range.0=-255
data.object.user.range.1=0
data.object.user.range.2=255
data.object.user.notation=big-O
data.object.user.details.lat=0.000
data.object.user.details.long=0.000
data.object.user.details.time=42
data.object.groups.0.id=2
data.object.groups.0.name=foo
data.object.groups.1.id=3
data.object.groups.1.name=bar
data.metdata.list.0.0.0=1
data.metdata.list.0.0.1=42
data.metdata.list.0.1.0=3.14
data.metdata.list.0.1.1=98.6
data.metdata.list.1.0=3
data.metdata.list.1.1=6
data.metdata.list.1.2=9
data.metdata.list.1.3=low
data.metdata.list.2.0.x=1
data.metdata.list.2.0.y=-1
data.metdata.ugly_nest.depth.test=true
log=123abc

@wader
Copy link

wader commented May 9, 2024

@chb0github what might confuse you is jq's use of implicit input argument.

# here getpath/1 will get ["log"] both as implicit input and as it's first "explicit" input (the path to get)
$ jq '["log"] | getpath(.)' example.json
jq: error (at example.json:27): Cannot index array with string "log"

# by using "... as $something" you will "bind" the value ["log"] to $p, this also make the input passthru so that
# getpath will get json from example.json as input and then $p as the path to get
$ jq '["log"] as $p | getpath($p)' example.json
"123abc"

and in the comment above path(..) as $p will cause each output of path(..) to be bound to $p and the rest of the pipeline is evaluated each time

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment