looping through JSON array in shell script

loop through json.md

Assume we have following json:

{
  "path": {
    "components": [],
    "name": "",
    "toString": ""
  },
  "revision": "master",
  "children": {
    "size": 5,
    "limit": 500,
    "isLastPage": true,
    "values": [
      {
        "path": {
          "components": [
            ".gitignore"
          ],
          "parent": "",
          "name": ".gitignore",
          "extension": "gitignore",
          "toString": ".gitignore"
        },
        "contentId": "c9e472ef4e603480cdd85012b01bd5f4eddc86c6",
        "type": "FILE",
        "size": 224
      },
      {
        "path": {
          "components": [
            "Jenkinsfile"
          ],
          "parent": "",
          "name": "Jenkinsfile",
          "toString": "Jenkinsfile"
        },
        "contentId": "e878a88eed6b19b2eb0852c39bfd290151b865a4",
        "type": "FILE",
        "size": 1396
      },
      {
        "path": {
          "components": [
            "README.md"
          ],
          "parent": "",
          "name": "README.md",
          "extension": "md",
          "toString": "README.md"
        },
        "contentId": "05782ad495bfe11e00a77c30ea3ce17c7fa39606",
        "type": "FILE",
        "size": 237
      },
      {
        "path": {
          "components": [
            "pom.xml"
          ],
          "parent": "",
          "name": "pom.xml",
          "extension": "xml",
          "toString": "pom.xml"
        },
        "contentId": "9cd4887f8fc8c2ecc69ca08508b0f5d7b019dafd",
        "type": "FILE",
        "size": 2548
      },
      {
        "path": {
          "components": [
            "src"
          ],
          "parent": "",
          "name": "src",
          "toString": "src"
        },
        "node": "395c71003030308d1e4148b7786e9f331c269bdf",
        "type": "DIRECTORY"
      }
    ],
    "start": 0
  }
}

Extracting the members

jq -c '.children.values[]|[.path.components[0],.type,.size]'

• .children.values[] outputs every member of the array .values.
• | pipes the previous result through the next filter, rather like a shell pipe
• [...,...,...] makes all the terms inside appear in a single array
• The -c option produces "compact" format ie. one object per line

Result:

[".gitignore","FILE",224]
["Jenkinsfile","FILE",1396]
["README.md","FILE",237]
...

Formatting the result

If you want to output a neatly-aligned table, that's a task better handled by other tools, such as column or paste.

jq -c '.children.values[]|[.path.components[0],.type,.size]' | column -t -s'[],"'

• t tells column to guess the number of columns based on the input
• s... specifies the delimiter character(s)

Result:

.gitignore   FILE       224
Jenkinsfile  FILE       1396
README.md    FILE       237

This relies on the characters [, ], , and " not appearing in your filenames, which is not a safe assumption.

paste can also arrange multiple inputs side-by-side. For this, we can remove the JSON structures altogether, and output raw lines (hat-tip to @muru):

jq -r '.children.values[]|.path.components[0],.type,.size' | paste - - -

paste - - - means 3 columns, all read from the the same source. This time, the only assumption is that the filenames don't contain newlines.

Another way...

from file:

for k in $(jq '.children.values | keys | .[]' file); do
    ...
done

or from string:

for k in $(jq '.children.values | keys | .[]' <<< "$MYJSONSTRING"); do
    ...
done

So e.g. one might use:

for k in $(jq '.children.values | keys | .[]' file); do
    value=$(jq -r ".children.values[$k]" file);
    name=$(jq -r '.path.name' <<< "$value");
    type=$(jq -r '.type' <<< "$value");
    size=$(jq -r '.size' <<< "$value");
    printf '%s\t%s\t%s\n' "$name" "$type" "$size";
done | column -t -s$'\t'

if you have no newlines for the values, you can make it with a single jq call inside the loop which makes it much faster:

for k in $(jq '.children.values | keys | .[]' file); do
    IFS=$'\n' read -r -d '' name type size \
        <<< "$(jq -r ".children.values[$k] | .path.name,.type,.size" file)"
    printf '%s\t%s\t%s\n' "$name" "$type" "$size";
done | column -t -s$'\t'