Welcome! If you're here, chances are you've made an API call, and it's returning JSON data that you want to parse using a jq script. Let's work through it together.
Whether you're a seasoned developer or just starting out, this blog is your go-to resource for mastering the art of JSON parsing with jq. Specifically, if you're searching for insights on how to efficiently filter and process date ranges, you've come to the right place.
Explore topics such as 'jq filter date range', 'jq how to select date range', 'jq filter selected date range', "How to select a date range from a JSON string by using jq", "jq between two dates" and more. Whether you're looking to filter records in the future, select last week's records, or retrieve emails from a specific timeframe using jq, this blog provides a comprehensive guide to help you navigate the intricacies of JSON parsing with precision.
Let's dive into the world of jq and together, master the art of effortlessly filtering and selecting date ranges!
example.json
{
"count": 10,
"next": "https://example.com/api/v1/users_api/",
"extras": "api-options",
"results": [
{
"date_joined": "2020-01-01T09:32:39",
"username": "user1",
"email": "user1@example.com"
},
{
"date_joined": "2021-01-01T10:35:46",
"username": "user2",
"email": "user2@example.com"
},
{
"date_joined": "2022-01-01T17:56:22",
"username": "user3",
"email": "user3@example.com"
},
{
"date_joined": "2023-01-02T17:32:10",
"username": "user4",
"email": "user4@example.com"
},
{
"date_joined": "2023-01-04T17:34:21",
"username": "user5",
"email": "user5@example.com"
},
{
"date_joined": "2023-01-06T01:05:29",
"username": "user6",
"email": "user6@example.com"
}
]
}example2.json
{
"menu": {
"id": "file",
"value": "File",
"popup": {
"menuitem": [
{
"date_joined": "2020-01-01T09:32:39",
"username": "user1",
"email": "user1@example.com"
},
{
"date_joined": "2021-01-01T10:35:46",
"username": "user2",
"email": "user2@example.com"
},
{
"date_joined": "2022-01-01T17:56:22",
"username": "user3",
"email": "user3@example.com"
},
{
"date_joined": "2026-01-01T17:56:22",
"username": "user33",
"email": "user33@example.com"
}
]
}
}
}For your situation, take a moment to inspect how your JSON is structured. Since my JSON data looked like the example below.
{
"count": 10,
"next": "https://example.com/api/v1/users_api/",
"previous": null,
"extras": "api-options",
"results": [
{
"key": "value"
},
{
"key": "value"
}
]
}In my case, to extract the desired results, I had to select '.results' using the following command:
cat example.json | jq -r '.results'Ensure that you can successfully parse your JSON file to achieve the final result demonstrated in the example below:
[
{
"date_joined": "2020-01-01T09:32:39",
"username": "user1",
"email": "user1@example.com"
},
{
"date_joined": "2021-01-01T10:35:46",
"username": "user2",
"email": "user2@example.com"
},
{
"date_joined": "2022-01-01T17:56:22",
"username": "user3",
"email": "user3@example.com"
}
]Familiarizing yourself with these examples will provide a clearer understanding of the subsequent information.
If you have a JSON file structured similarly to example2.json, you can use the following command as a reference:
cat example2.json | jq -r '.menu.popup.menuitem'And the expected result will be:
[
{
"date_joined": "2020-01-01T09:32:39",
"username": "user1",
"email": "user1@example.com"
},
{
"date_joined": "2021-01-01T10:35:46",
"username": "user2",
"email": "user2@example.com"
},
{
"date_joined": "2022-01-01T17:56:22",
"username": "user3",
"email": "user3@example.com"
}
]Now that we've successfully obtained the desired results, as illustrated in the example below
[
{
"key": "value"
},
{
"key": "value"
}
]Let's start with advanced setup.
Now, let's proceed to parse the obtained data.
We'll start by retrieving all records with a date later than 2022-01-01T14:06:34
cat example.json | jq -r '.results | map(select(.date_joined > "2022-01-01T14:06:34" ))'Alternatively, for example2.json
cat example2.json | jq -r '.menu.popup.menuitem | map(select(.date_joined > "2022-01-01T14:06:34" ))'However, I want to select specific dates, not just those after a certain point
- example.json
cat example.json | jq -r '.results | map(select(.date_joined | . >= "2022-01-01T14:06:34" and . <= "2024-01-01T14:06:34" ))'
#or
cat example.json | jq --arg s '2022-01-01T14:06:34' --arg e '2024-01-01T14:06:34' -r '.results | map(select(.date_joined | . >= $s and . <= $e ))'- example2.json
cat example2.json | jq -r '.menu.popup.menuitem | map(select(.date_joined | . >= "2022-01-01T14:06:34" and . <= "2024-01-01T14:06:34" ))'
#or
cat example2.json | jq --arg s '2022-01-01T14:06:34' --arg e '2024-01-01T14:06:34' -r '.menu.popup.menuitem | map(select(.date_joined | . >= $s and . <= $e ))'What if we create a jq filter for future dates, retrieving users who are timestamped from the future every time the script is executed
Note: In example2.json, there is a user with a future timestamp.
cat example2.json | jq -r '.menu.popup.menuitem | map(select(.date_joined > (now | strftime("%Y-%m-%d"))))'In the case of example, the result will be empty as there are no users with future timestamps.
Sure, before we proceed to the script, let's take a moment to understand some essential syntax for specifying time ranges. In this context, we'll be looking at common formatting characters used with the date command.
These characters are crucial when defining time intervals:
%D – Display date as mm/dd/yy
%Y – Year (e.g., 2020)
%m – Month (01-12)
%B – Long month name (e.g., November)
%b – Short month name (e.g., Nov)
%d – Day of the month (e.g., 01)
%j – Day of the year (001-366)
%u – Day of the week (1-7)
%A – Full weekday name (e.g., Friday)
%a – Short weekday name (e.g., Fri)
%H – Hour (00-23)
%I – Hour (01-12)
%M – Minute (00-59)
%S – Second (00-60)
For more details, you can refer to the date command documentation.
In our script example, we'll use the format date +%Y-%m-%d'T'%T, which represents a date like 2022-01-01T14:06:34.
It's crucial to ensure the selected format aligns with the date and time structure defined in your API. Otherwise, the filtering process may not yield the expected results.
Feel free to experiment with the date command, for instance:
date +%Y-%m-%d'T'%T --date='tomorrow'
date +%Y-%m-%d'T'%T --date='10 sec ago'
date +%Y-%m-%d'T'%T --date='yesterday'
date +%Y-%m-%d'T'%T --date='2 years ago'
Additionally, you can project into the future:
date +%Y-%m-%d'T'%T --date='next monday'
date +%Y-%m-%d'T'%T --date='4 days' (four days into the future)
Please note +%y-%m-%d'T'%T, the result will be something like 22-01-01T14:06:34.
This might not work as expected because %Y represents the full year (e.g., 2023), while %y only represents the last two digits of the year (e.g., 23).
Ensure that you select a format that aligns with the date and time structure defined in your API.
Using an incorrect format may lead to unexpected results or no results at all.
Now, let's consider a practical example of using these concepts in a script.
Let's consider an example for a timestamp representing 15 minutes ago. Feel free to modify the duration as needed. In this script snippet:
#!/bin/bash
DATE_15_MIN_AGO=$(date +%Y-%m-%d'T'%T --date='15 minutes ago')
cat example2.json | jq -r ".results | map(select(.date_joined > \"$DATE_15_MIN_AGO\" ))"Imagine you have an API call, and your goal is to forward all the emails to your internal system. Here's a script example to accomplish this:
#!/bin/bash
DATE_15_MIN_AGO=$(date +%Y-%m-%d'T'%T --date='15 minutes ago')
curl -s -H "Authorization: Token $TOKEN" \
-H "Content-Type: application/json" https://example.com/api/v1/users_api/ \
| jq -r ".results | map(select(.date_joined > \"$DATE_15_MIN_AGO\" )) | .[].email" \
| while read object; do echo "send webh $object" ; doneThis script performs the following:
* Retrieves data from the API using curl.
* Uses jq to filter the results based on a timestamp condition (here, 15 minutes ago).
* Extracts the email addresses.
* Sends the emails to your internal system using a custom command (send webh in this example).
Make sure to replace $TOKEN with your actual authentication token. Feel free to adapt the script based on your specific API structure and requirements."
- In conclusion, we've navigated through the process of parsing JSON data from API calls using the powerful jq tool.
- From understanding the structure of your JSON files to crafting commands for specific filtering needs, we've covered a range of scenarios.
- Remember to tailor the provided scripts to match your API's date and time structure, ensuring accurate results.
- Whether you're extracting records based on timestamps, filtering for future dates, or processing data within specific time intervals, these examples serve as a practical guide.
- Feel free to explore further variations and customize the scripts according to your unique requirements. Happy scripting!