jq Cheet Sheet

jq-cheetsheet.md

Processing JSON using jq

jq is useful to slice, filter, map and transform structured json data.

Installing jq

On Ubuntu

sudo apt-get install jq

On Mac OS

brew install jq

On AWS Linux

Not available as yum install on our current AMI. It should be on the latest AMI though: https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/

Installing from the source proved to be tricky.

Useful arguments

When running jq, the following arguments may become handy:

Argument	Description
--version	Output the jq version and exit with zero.
--sort-keys	Output the fields of each object with the keys in sorted order.

Basic concepts

The syntax for jq is pretty coherent:

Syntax	Description
,	Filters separated by a comma will produce multiple independent outputs
?	Will ignore error if the type is unexpected
[]	Array construction
{}	Object construction
+	Concatenate or Add
-	Difference of sets or Subtract
length	Size of selected element
|	Pipes are used to chain inpiuts and outputs in a similar fashion to bash

Dealing with json objects

Description	Command
Display all keys	jq 'keys'
Adds + 1 to all items	jq 'map_values(.+1)'
Delete a key	jq 'del(.foo)'
Convert an object to array	to_entries | map([.key, .value])

Dealing with fields

Description	Command
Concatenate two fields	fieldNew=.field1+' '+.field2

Dealing with json arrays

Slicing and Filtering

Description	Command
All	jq .[]
First	jq '.[0]'
Range	jq '.[2:4]'
First 3	jq '.[:3]'
Last 2	jq '.[-2:]'
Before Last	jq '.[-2]'
Select where value is foo or bar	q 'select(.value == "foo" or .value == "bar")'
Select array of int by value	jq 'map(select(. >= 2))'
Select array of objects by value	jq '.[] | select(.id == "second")'
Select by type	jq '.[] | numbers' with type being arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars

Mapping and Transforming

Description	Command
Add + 1 to all items	jq 'map(.+1)'
Delete 2 items	jq 'del(.[1, 2])'
Concatenate arrays	jq 'add'
Flatten an array	jq 'flatten'
Create a range of numbers	jq '[range(2;4)]'
Display the type of each item	jq 'map(type)'
Sort an array of basic type	jq 'sort'
Sort an array of objects	jq 'sort_by(.foo)'
Group by a key - opposite to flatten	jq 'group_by(.foo)'
Minimun value of an array	jq 'min' .See also min, max, min_by(path_exp), max_by(path_exp)
Remove duplicates	jq 'unique' or jq 'unique_by(.foo)' or jq 'unique_by(length)'
Reverse an array	jq 'reverse'
Convert {id: id-val, value: val} to a json	'[ .path | { (.id|tostring): .value }]'

Use select(boolean_expression)

          jq '.[] | select(.id == "second")'
Input:	  [{"id": "first", "val": 1}, {"id": "second", "val": 2}]
Output: 	{"id": "second", "val": 2}

        jq 'map(select(. >= 2))'
Input	  [1,5,3,0,7]
Output 	[5,3,7]

Create csv file

echo "ID,NAME,TYPE" > output.csv cat input.json | jq -r '.resource_list[] | [.id, .name, .type] | @csv' >> output.csv

Examples

1. Find length of a list cat users-list.json | jq -r '.results | length'

2. To extract top level attributes "timestamp" and "report" jq '. | {timestamp,report}'

3. To extract name and age of each "results" item jq '.results[] | {name, age}'

4. To extract name and age as text values instead of JSON jq -r '.results[] | {name, age} | join(" ")'

5. Filter this by attribute

      jq '.results[] | select(.name == "John") | {age}'          # Get age for 'John'
      jq '.results[] | select((.name == "Joe") and (.age = 10))' # Get complete records for all 'Joe' aged 10
      jq '.results[] | select(.name | contains("Jo"))'           # Get complete records for all names with 'Jo'
      jq '.results[] | select(.name | test("Joe\s+Smith"))'      # Get complete records for all names matching PCRE regex 'Joe\+Smith'
   

6. Avoid null output when accessing non-existing keys jq '.mykey | select(. != null)'

7. If you want to combine subkeys at different levels jq '.items[] | { "created" : .metadata["created"], name }' OR jq '.items[] | .metadata["created"], .name'

8. If you want to extract all properties into json jq '. | to_entries[] | select( .key | contains("prop"))' This will output each key value pair as: {"key": "Keyname", "value": "Value"}

9. When you just want the keys jq '.animals | keys'

10. If you want to convert JSON of format from (8) to a proper key value json '[ .path | { (.key|tostring): .value }]'

11. Match a regular expression .path | match("([a-zA-Z0-9+._-]+@[a-zA-Z0-9._-]+.[a-zA-Z0-9_-]+)"; "i") | .string will search for emails .path | match("\\d+") | .string will search for digits

12. Replace strings .email | sub("john";"richard";"i")

Changing values with jq

Merging/overwriting keys

echo '{ "a": 1, "b": 2 }' |\
jq '. |= . + {
  "c": 3
}'

Adding elements to lists

echo '{ "names": ["Marie", "Sophie"] }' |\
jq '.names |= .+ [
   "Natalie"
]'   

Delete values with jq

jq 'del(.somekey)' input.json

Merge JSON strings

For example merge three object lists:

echo '[ {"a":1}, {"b":2} ]' | \
jq --argjson input1 '[ { "c":3 } ]' \
   --argjson input2 '[ { "d":4 }, { "e": 5} ]' \
   '. = $input1 + . +  $input2'

Merge files (since jq 1.4)

The following command will merge "somekey" from both passed files

jq -s '.[0] * .[1] | {somekey: .somekey}' <file1> <file2>

Handle Empty Arrays

When you want to iterate over an array, and the array you access is empty you get something like

jq: error (at <stdin>:3): Cannot iterate over null (null)

To workaround the optional array protect the access with

select(.my_array | length > 0)

Testing Types

$ echo '[true, null, 42, "hello", []]' | ./jq 'map(type)'
["boolean","null","number","string","array"]

Using it in shell scripts

Parsing JSON into env vars

To fill environment variables from JSON object keys (e.g. $FOO from jq query ".foo")

export $(jq -r '@sh "FOO=\(.foo) BAZ=\(.baz)"')

To make a bash array

read -a bash_array < <(jq -r .|arrays|select(.!=null)|@tsv)

JSON template using env vars

To create proper JSON from a shell script and properly escape variables:

jq -n --arg foobaz "$FOOBAZ" '{"foobaz":$foobaz}'

URL Encode

Quick easy way to url encode something

date | jq -sRr @uri

String Concat

Concatenation like this:

echo '{ "object" : { "name": "banana", "color": "yellow" }}' |\
jq -r '.object | (.name)+" is "+(.color)'

will print banana is yellow.

String Interpolation

Or using Interpolation:

echo '{ "object" : { "name": "banana", "color": "yellow" }}' |\
jq -r '.object | "\(.name) is \(.color)"'

will also print banana is yellow.

Turn A List From A Command Into JSON

There are a lot of command-line utilities that produce a list of things. Since JSON is a universal data format, it would be useful to be able to quickly turn some items from stdout into a JSON list.

The jq utility can help with this.

Let's say I'm working with the following git command that lists changed files in a specific directory.

$ git diff --name-only | grep some/dir

I can then pipe that list of files to jq with a few flags.

$ git diff --name-only \
  | grep some/dir \
  | jq -R -s 'split("\n")[:-1]'

Here's what is going on:

• The -R flag tells jq to accept raw input, rather than looking for JSON.
• The -s flag is short for --slurp and tells jq to read in the entire input before applying the filter.
• The string argument is the filter to be applied to the output. It splits on newlines and then takes the entire array except for the last item ([:-1]) which would be an empty string for the trailing newline.
• jq automatically turns the whole thing into a formatted JSON list.

References

Reference - https://stedolan.github.io/jq/manual/