learn_jq.ipynb

{
"cells": [
{
"cell_type": "markdown",
"id": "aa2988ce",
"metadata": {},
"source": [
"# JQ Interactive Learning\n",
"\n",
"Please first make sure that you have jq installed on your machine. You can check by running `jq --help | head -n 1` and seeing if the command is recognized. \n",
"\n",
"If you do not have it installed yet, you can install it with `brew install jq` on Mac. Please view the jq docs for installation instructions for other operating systems."
]
},
{
"cell_type": "markdown",
"id": "9a4b3f37",
"metadata": {},
"source": [
"Check if jq is available in the notebook:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4ef8ac56",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"jq --help | head -n 1"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8bfa3b0b",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"echo '{\"fruit\":{\"name\":\"apple\",\"color\":\"green\",\"price\":1.20}}' | jq '.'"
]
},
{
"cell_type": "markdown",
"id": "2a75adac",
"metadata": {},
"source": [
"Let's look at a basic curl response without jq:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f279927c",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"curl -s http://api.open-notify.org/iss-now.json"
]
},
{
"cell_type": "markdown",
"id": "1826b5e4",
"metadata": {},
"source": [
"Apply jq to an API response - a common technique you will find in CICD scripts:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6b3b761f",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"curl -s http://api.open-notify.org/iss-now.json | jq '.'"
]
},
{
"cell_type": "markdown",
"id": "728c5ff5",
"metadata": {},
"source": [
"Return value based on provided key:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d2c4256d",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"json='{\"name\": \"apple\",\"color\": \"green\",\"price\": 1.2}'\n",
"echo \"$json\" | jq '.color'"
]
},
{
"cell_type": "markdown",
"id": "74d4b824",
"metadata": {},
"source": [
"Basic json array example - this comes up with many APIs such as GitHub that returns the response as an array:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b4052ee2",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"json='[\"Mike\", \"Serge\", \"Rachel\", \"Thomas\", \"David\", \"Adelaide\", \"Mika\", \"Julio\", \"Arvind\"]'\n",
"echo \"$json\" | jq '.[]'"
]
},
{
"cell_type": "markdown",
"id": "8c9009fe",
"metadata": {},
"source": [
"Let's look at how compact output can pretify arrays with larger nested objects - this can be useful for writing scripts that loop through arrays and apply changes based on the position of output:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dcfe413b",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"json='[{\"name\": \"apple\",\"color\": \"green\",\"price\": 1.2},{\"name\": \"banana\",\"color\": \"yellow\",\"price\": 0.5},{\"name\": \"kiwi\",\"color\": \"green\",\"price\": 1.25}]'\n",
"echo \"$json\" | jq '.[]'"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b4b11254",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"json='[{\"name\": \"apple\",\"color\": \"green\",\"price\": 1.2},{\"name\": \"banana\",\"color\": \"yellow\",\"price\": 0.5},{\"name\": \"kiwi\",\"color\": \"green\",\"price\": 1.25}]'\n",
"echo \"$json\" | jq -c '.[]'"
]
},
{
"cell_type": "markdown",
"id": "fc948924",
"metadata": {},
"source": [
"Perhaps we would like to loop through an array of team members and sort the items alphabetically - notice the `-r` for raw, unquoted output:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c30d2d33",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"json='[{\"team1\": \"Mike\"}, {\"team1\": \"Serge\"}, {\"team1\": \"Rachel\"}, {\"team1\": \"Thomas\"}, {\"team1\": \"David\"}, {\"team1\": \"Adelaide\"}, {\"team1\": \"Mika\"}, {\"team1\": \"Julio\"}, {\"team1\": \"Arvind\"}]'\n",
"echo \"$json\" | jq -r '.[] | .team1' | sort"
]
},
{
"cell_type": "markdown",
"id": "d1528bd3",
"metadata": {},
"source": [
"How about indices with json arrays?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8d59867b",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"json='[{\"team1\": \"Mike\"}, {\"team1\": \"Serge\"}, {\"team1\": \"Rachel\"}, {\"team1\": \"Thomas\"}, {\"team1\": \"David\"}, {\"team1\": \"Adelaide\"}, {\"team1\": \"Mika\"}, {\"team1\": \"Julio\"}, {\"team1\": \"Arvind\"}]'\n",
"index=0\n",
"for row in $(echo \"${json}\" | jq -r '.[] | .team1'); do\n",
"    temp=$(echo \"${json}\" | jq -r '.['\"$index\"'] | .team1')\n",
"    echo \"Row: ${index}, the team member at index ${index} is $temp\"\n",
"    ((++index))\n",
"done"
]
},
{
"cell_type": "markdown",
"id": "7cd8a926",
"metadata": {},
"source": [
"Finally, I will leave you with a practical example that checks that the build is running that latest commit.\n",
"\n",
"We have the following bash script:\n",
"```\n",
"#!/bin/bash\n",
"\n",
"OWNER=\"owner\"\n",
"REPO=\"repo\"\n",
"\n",
"if [[ \"$TRAVIS_EVENT_TYPE\" == \"push\" && \"$TRAVIS_BRANCH\" == \"master\" ]]; then\n",
"    latest_master_commit=$(curl --silent -X GET \\\n",
"        -H \"Authorization: token $GH_TOKEN\" \\\n",
"        -H \"Accept: application/vnd.github+json\" \\\n",
"        https://api.github.com/repos/\"${OWNER}\"/\"${REPO}\"/commits | jq -r '.[0] | .sha') \n",
"    if [[ \"$latest_master_commit\" == \"$TRAVIS_COMMIT\" ]]; then\n",
"        echo \"This build is running the latest commit, continuing to terraform apply...\" \n",
"        terraform apply -auto-approve\n",
"    else\n",
"        echo \"This is not the latest commit, please merge master into your feature branch, resolve any conflicts, and re-run the build on the latest commit. The terraform state is remote, so if you re-run an old build it may cause infrastructure to be deleted because terraform will see infra in the state file that is not reflected in the IaC.\"\n",
"        exit 1  \n",
"    fi                                            \n",
"else\n",
"    echo \"Not running protected-terraform-apply because this is not a push to master.\"\n",
"fi\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "a27475bf",
"metadata": {},
"source": [
"Notice \n",
"\n",
"```\n",
"latest_master_commit=$(curl --silent -X GET \\\n",
"        -H \"Authorization: token $GH_TOKEN\" \\\n",
"        -H \"Accept: application/vnd.github+json\" \\\n",
"        https://api.github.com/repos/\"${OWNER}\"/\"${REPO}\"/commits | jq -r '.[0] | .sha')\n",
"```\n",
"You can try it in the block below - make sure to generate a personal access token under your GitHub Developer Settings that has repo access and plug it in the variable along with the repo:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1bf5995f",
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"OWNER=\"owner\"\n",
"REPO=\"repo\"\n",
"\n",
"GITHUB_PERSONAL_ACCESS_TOKEN=\"<insert>\"\n",
"\n",
"curl --silent -X GET \\\n",
"        -H \"Authorization: token $GITHUB_PERSONAL_ACCESS_TOKEN\" \\\n",
"        -H \"Accept: application/vnd.github+json\" \\\n",
"        https://api.github.com/repos/\"${OWNER}\"/\"${REPO}\"/commits | jq -r '.[0] | .sha'"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}