Yukon Tech Collective January 2020 meetup notes

yk-tech.2020-01-09.md

We investigated the tool jq.

Wes first encountered this as a query tool for JSON. Our probing with the tool lead us to believe that it is much more than this.

This author intends to investigate JSONPath in the future, as it feels targetted specifically at querying. jq is capable of arbitrary JSON transformations. There is also JMESpath which is described as query language for JSON (this author was under the impression JMESpath was also capable of JSON transformations, albeit a more limited set than jq allows for the expression of)

First we needed some JSON. @ryanagar noted the considerable breadth of Yukon GIS data. With the help of a query builder, we were able to pull down a blob, and the task for the group was to investigate it's structure with jq, thereby probing jq's expressiveness.

URL='http://mapservices.gov.yk.ca/arcgis/rest/services/GeoYukon/GY_Transportation/MapServer/4/query?where=1%3D1&text=&objectIds=&time=&geometry=&geometryType=esriGeometryEnvelope&inSR=&spatialRel=esriSpatialRelIntersects&relationParam=&outFields=*&returnGeometry=true&returnTrueCurves=false&maxAllowableOffset=&geometryPrecision=&outSR=&having=&returnIdsOnly=false&returnCountOnly=false&orderByFields=&groupByFieldsForStatistics=&outStatistics=&returnZ=false&returnM=false&gdbVersion=&historicMoment=&returnDistinctValues=false&resultOffset=&resultRecordCount=&queryByDistance=&returnExtentOnly=false&datumTransformation=&parameterValues=&rangeValues=&quantizationParameters=&f=pjson'
curl ${URL} > yg_gis.json

What we got was a complex object, the structure of which we wanted to understand.

cat yg_gis.json | jq '. | keys'
[
  "displayFieldName",
  "features",
  "fieldAliases",
  "fields",
  "geometryType",
  "spatialReference"
]

The goal was to use jq to tell us the structure of the object, by reporting the types of each members. Our goal was the following output:

[
  ["displayFieldName", "string"],
  ["features", "array"],
  ["fieldAliases", "object"],
  ["fields", "array"],
  ["geometryType", "string"],
  ["spatialReference", "object"]
]

(i.e. we've identified the type of each object member. Note that cat yg_gis.json | jq '.displayFieldName | type' returns string)

What follows is our collective exploration, mostly mobbed with @anlek reading docs and the broader group hurling suggestions pop-corn style :)

Our eventual solution returned

{
  "displayFieldName": "string"
}
{
  "features": "array"
}
{
  "fieldAliases": "object"
}
{
  "fields": "array"
}
{
  "geometryType": "string"
}
{
  "spatialReference": "object"
}

which isn't quite what we were were after, but close enough ;)

cat yg_gis.json | jq '. as $source | keys[] | . as $key | $source[.] | {($key): type}'

cat yg_gis.json | jq '.[] | .. | {(values): type} '

cat yg_gis.json | jq '. | type'
cat yg_gis.json | jq '. | keys'
cat yg_gis.json | jq '.displayFieldName | type'
cat yg_gis.json | jq '.displayFieldName'
cat yg_gis.json | jq '.features | type'
cat yg_gis.json | jq '.fieldAliases | type'
cat yg_gis.json | jq '.fields | type'
cat yg_gis.json | jq '.geometryType | type'
cat yg_gis.json | jq '.geometryTyp'
cat yg_gis.json | jq '.geometryType'
cat yg_gis.json | jq '.spatialReference | type'
cat yg_gis.json | jq '.' | wc -l
cat yg_gis.json | jq '. | keys'
cat yg_gis.json | jq '. | keys'
cat yg_gis.json | jq '.displayFieldName | array'
cat yg_gis.json | jq '.displayFieldName | type'
cat yg_gis.json | jq '. | keys'
cat yg_gis.json | jq '. | keys | type'
cat yg_gis.json | jq '. | keys | .[] | type'
cat yg_gis.json | jq '{key: .key}'
URL='http://mapservices.gov.yk.ca/arcgis/rest/services/GeoYukon/GY_Transportation/MapServer/4/query?where=1%3D1&text=&objectIds=&time=&geometry=&geometryType=esriGeometryEnvelope&inSR=&spatialRel=esriSpatialRelIntersects&relationParam=&outFields=*&returnGeometry=true&returnTrueCurves=false&maxAllowableOffset=&geometryPrecision=&outSR=&having=&returnIdsOnly=false&returnCountOnly=false&orderByFields=&groupByFieldsForStatistics=&outStatistics=&returnZ=false&returnM=false&gdbVersion=&historicMoment=&returnDistinctValues=false&resultOffset=&resultRecordCount=&queryByDistance=&returnExtentOnly=false&datumTransformation=&parameterValues=&rangeValues=&quantizationParameters=&f=pjson'
curl ${URL} | tee yg_gis.json | jq '.'
cat yg_gis.json | jq '{key: .keys}'
cat yg_gis.json | jq '. | keys'
cat yg_gis.json | jq '[0,1,2] | keys'
cat yg_gis.json | jq '[0,1,2] | type'
cat yg_gis.json | jq '. | keys'
cat yg_gis.json | jq '. | values | '
cat yg_gis.json | jq '. | values'
cat yg_gis.json | jq '. | values'
cat yg_gis.json | jq '. | values | type'
cat yg_gis.json | jq '. | values | keys'
cat yg_gis.json | jq '. | keys'
cat yg_gis.json | jq '. | type'
cat yg_gis.json | jq '.'
cat yg_gis.json | jq '.. | values'
cat yg_gis.json | jq '..'
cat yg_gis.json | jq '...'
cat yg_gis.json | jq '.*'
cat yg_gis.json | jq '..'
cat yg_gis.json | jq '.. | type'
cat yg_gis.json | jq '[..] | length'
cat yg_gis.json | jq '. | .'
cat yg_gis.json | jq '. | . | keys'
cat yg_gis.json | jq '. | keys'
cat yg_gis.json | jq '. | keys'
cat yg_gis.json | jq '. | keys | {(.)}'
cat yg_gis.json | jq '. | keys | {(.): true}'
cat yg_gis.json | jq '. | keys | {.: true}'
cat yg_gis.json | jq '. | keys | {(.): true}'
cat yg_gis.json | jq '. {(.): true}'
cat yg_gis.json | jq '. | {(.): true}'
cat yg_gis.json | jq '. | {(.): true}'
cat yg_gis.json | jq '. | {(keys): true}'
cat yg_gis.json | jq '. | {(keys): type}'
cat yg_gis.json | jq '.. | [keys, type]'
cat yg_gis.json | jq '. | [keys, type]'
cat yg_gis.json | jq '. | [[keys | keys, type]]'
cat yg_gis.json | jq '. | [[keys | values, type]]'
cat yg_gis.json | jq '. | keys | values, type'
cat yg_gis.json | jq '. | keys | .., type'
cat yg_gis.json | jq '. | keys | .., type'
cat yg_gis.json | jq ' recurse(.) | type'
cat yg_gis.json | jq ' recurse(.) |[ keys, type]'
cat yg_gis.json | jq ' recurse(.) |[.keys,.type]'
cat yg_gis.json | jq ' recurse(.)'
cat yg_gis.json | jq ' .'
cat yg_gis.json | jq '. | (keys, type)'
cat yg_gis.json | jq 'type'
cat yg_gis.json | jq 'type .'
cat yg_gis.json | jq '.. type'
cat yg_gis.json | jq '.. | type'
cat yg_gis.json | jq '. | keys, (.. | type)'
cat yg_gis.json | jq '. | [keys, (.. | type)]'
cat yg_gis.json | jq '. | [keys, (. | type)]'
cat yg_gis.json | jq '. | [keys, . | type]'
cat yg_gis.json | jq '.[]'
cat yg_gis.json | jq '.[][]'
cat yg_gis.json | jq '.[].'
cat yg_gis.json | jq '.[] | objects'
cat yg_gis.json | jq '.[] | objects, type'
cat yg_gis.json | jq '.[] | keys, type'
cat yg_gis.json | jq '.[] | keys'
cat yg_gis.json | jq '.[] keys'
cat yg_gis.json | jq '. | keys'
cat yg_gis.json | jq '(. | keys), (. | keys)'
cat yg_gis.json | jq '(. | keys, . | keys)'
cat yg_gis.json | jq '([. | keys], [. | keys]) | keys'
cat yg_gis.json | jq '.[*]'
cat yg_gis.json | jq '.[]'
cat yg_gis.json | jq '[1,2,3]..'
cat yg_gis.json | jq '[1,2,3] | map'
cat yg_gis.json | jq '[1,2,3] | math'
cat yg_gis.json | jq '[1,2,3] | map(type)'
cat yg_gis.json | jq '[1,2,3] | map(keys, type)'
cat yg_gis.json | jq '[1,2,3] | keys'
cat yg_gis.json | jq '[1,2,3] | type'
cat yg_gis.json | jq '[1,2,3]. | type'
cat yg_gis.json | jq '[1,2,3][] | type'
cat yg_gis.json | jq '[1,2,3][.] | type'
cat yg_gis.json | jq '[1,2,3][] | [type]'
cat yg_gis.json | jq '[[1,2,3][] | type]'
cat yg_gis.json | jq '[[1,2,3] | values | type]'
cat yg_gis.json | jq '[[1,2,3] | . | type]'
cat yg_gis.json | jq '[[1,2,3] | .. | type]'
cat yg_gis.json | jq '[[1,2,3] | type]'
cat yg_gis.json | jq '[[1,2,3]. | type]'
cat yg_gis.json | jq '[[1,2,3] | .]'
cat yg_gis.json | jq '[[1,2,3] | values]'
cat yg_gis.json | jq '[4,5,6] | types'
cat yg_gis.json | jq '[4,5,6][] | types'
cat yg_gis.json | jq '[4,5,6][] | type'
cat yg_gis.json | jq '[4,5,6][] | type'
cat yg_gis.json | jq '[4,5,6][] | keys'
cat yg_gis.json | jq '[4,5,6][]'
cat yg_gis.json | jq '[4,5,6][] | {., type}'
cat yg_gis.json | jq '[4,5,6][] | {.: type}'
cat yg_gis.json | jq '[4,5,6][] | {(.): type}'
cat yg_gis.json | jq '[4,5,6][] | [(.), type]'
cat yg_gis.json | jq '[4,5,6][] | {key:values'
cat yg_gis.json | jq '[4,5,6][] | {type: values'
cat yg_gis.json | jq '[4,5,6][] | {type: values}'
cat yg_gis.json | jq '[4,5,6][] | {(type): values}'
cat yg_gis.json | jq '.[] | {(type): values}'
cat yg_gis.json | jq '..[] | {(type): values}'
cat yg_gis.json | jq '.[][] | {(type): values}'
cat yg_gis.json | jq '.[] | {(type): values}'
cat yg_gis.json | jq '. | {(type): values}'
cat yg_gis.json | jq '.. | {(type): values}'
cat yg_gis.json | jq '.. | keys, . | {(type): values}'
cat yg_gis.json | jq '.[] | keys, . | {(type): values}'
cat yg_gis.json | jq '. | keys, . | {(type): values}'
cat yg_gis.json | jq '. | (keys, .) | {(type): values}'
cat yg_gis.json | jq '. | (keys, .)'
cat yg_gis.json | jq '. | (keys, .)'
cat yg_gis.json | jq '. | keys'
cat yg_gis.json | jq '. | keys, (. | type)'
cat yg_gis.json | jq '. | keys, (values | type)'
cat yg_gis.json | jq '. | (keys . | type)'
cat yg_gis.json | jq '. | (keys, . | type)'
cat yg_gis.json | jq '. | {(type), .}'
cat yg_gis.json | jq '. | keys | .[]'
cat yg_gis.json | jq '. | (keys | .[]), type'
cat yg_gis.json | jq '. | (keys | .[]), [] | type'
cat yg_gis.json | jq '. | {(type): values}'
cat yg_gis.json | jq '.. | {(type): values}'
cat yg_gis.json | jq '.. | {(.): type}'
cat yg_gis.json | jq '.. | {('test'): type}'
cat yg_gis.json | jq '.. | {test: type}'
cat yg_gis.json | jq '.. | {(): type}'
cat yg_gis.json | jq '.. | {(keys | []): type}'
cat yg_gis.json | jq '.. | {(keys[]): type}'
cat yg_gis.json | jq '. | {(keys[]): type}'
cat yg_gis.json | jq '. | {([].keys[]): type}'
cat yg_gis.json | jq '. | {([] | keys[]): type}'
cat yg_gis.json | jq '. | {(keys[]): type}'
cat yg_gis.json | jq '. | {(keys[]), type}'
cat yg_gis.json | jq '. | {(keys[]): type}'
cat yg_gis.json | jq '. | {(keys[]): (.. | type)}'
cat yg_gis.json | jq '. | {(keys[]): (.[] | type)}'
cat yg_gis.json | jq '. | {(keys[]): (.[] | keys | type)}'
cat yg_gis.json | jq '. | {(keys[]): (. | keys | type)}'
cat yg_gis.json | jq '. | {(keys): (. | keys | type)}'
cat yg_gis.json | jq '. | {(keys[]): (. | keys)}'
cat yg_gis.json | jq '. | {(keys): (.. | type)}'
cat yg_gis.json | jq '. | {(keys): (.[] | type)}'
cat yg_gis.json | jq '. | {(keys): (key[] | type)}'
cat yg_gis.json | jq '. | {(keys): (keys[] | type)}'
cat yg_gis.json | jq '. | {(keys[]): (keys[] | type)}'
cat yg_gis.json | jq '. | {(keys[]): (keys[])}'
cat yg_gis.json | jq '. | {(keys[]): ([])}'
cat yg_gis.json | jq '. | {(keys[]): (.[])}'
cat yg_gis.json | jq '. | {(keys[]): (keys[])}'
cat yg_gis.json | jq '. | {(keys[]): (.[keys[]])}'
cat yg_gis.json | jq '. | {(keys[]): (.(keys[]))}'
cat yg_gis.json | jq '. | {(keys[]): (.[keys[])]}'
cat yg_gis.json | jq '. | {(keys[]): ([values[])}'
cat yg_gis.json | jq '. | {(keys[]): (values[])}'
cat yg_gis.json | jq '. | {(keys[]): (.)}'
cat yg_gis.json | jq '. | {(keys[]): (..)}'
cat yg_gis.json | jq '. | {(keys[]): (.. | type)}'
cat yg_gis.json | jq '. | {(keys[]): (.(keys[]))}'
cat yg_gis.json | jq '. | {(keys[]): (.. | type)}'
cat yg_gis.json | jq '. | {(..): (.. | type)}'
cat yg_gis.json | jq '. | {(.. | keys): (.. | type)}'
cat yg_gis.json | jq '. | {(. | keys): (.. | type)}'
cat yg_gis.json | jq '. | {([] | keys): (.. | type)}'
cat yg_gis.json | jq '. | {(keys): (.. | type)}'
cat yg_gis.json | jq '. | {(keys[]): (.. | type)}'
cat yg_gis.json | jq '.'
cat yg_gis.json | jq '.' | keys
cat yg_gis.json | jq '.' | {key: keys}
cat yg_gis.json | jq '. | (keys[]): (.. | type)'
cat yg_gis.json | jq '. | (keys[]), (.. | type)'
cat yg_gis.json | jq '.. | (keys[]), (.. | type)'
cat yg_gis.json | jq '. | (keys[])'
cat yg_gis.json | jq '. | (keys[])'
cat yg_gis.json | jq '.. | keys'
cat yg_gis.json | jq '.[] | keys'
cat yg_gis.json | jq '. | keys[]'
cat yg_gis.json | jq '.[]'
cat yg_gis.json | jq '.[][]'
cat yg_gis.json | jq '.[] | keys'
cat yg_gis.json | jq '.[][0]'
cat yg_gis.json | jq '.[0]'
cat yg_gis.json | jq '.[]'
cat yg_gis.json | jq '.[] | values'
cat yg_gis.json | jq '.[].0'
cat yg_gis.json | jq '.0'
cat yg_gis.json | jq '.'
cat yg_gis.json | jq '.[]' 
cat yg_gis.json | jq '. | recurse(keys)' 
cat yg_gis.json | jq '. | keys[] | recurse(.)' 
cat yg_gis.json | jq '. | keys[] ' 
cat yg_gis.json | jq '. | keys ' 
cat yg_gis.json | jq '.[] | keys ' 
cat yg_gis.json | jq '.' 
cat yg_gis.json | jq '.[]' 
cat yg_gis.json | jq '. | (keys[]): (.. | type)'
cat yg_gis.json | jq '. as $source | keys[] | $source(.)'
cat yg_gis.json | jq '. as $source | keys[] | $source(.)'
cat yg_gis.json | jq '. as $source | keys[] | $source'
cat yg_gis.json | jq '. as $source | keys[] | $source.(.)'
cat yg_gis.json | jq '. as $source | keys[] | $source[.]'
cat yg_gis.json | jq '. as $source | keys[] | $source[.] | type'
cat yg_gis.json | jq '. as $source | keys[] as $key | $source[.] | {($key) : type}'
cat yg_gis.json | jq '. as $source | keys[] as $key | $source[.] | {($key): type}'
cat yg_gis.json | jq '. as $source | keys[] as $key | $source[.] | {($key[]): type}'
cat yg_gis.json | jq '. as $source | keys[] | . as $key | $source[.] | {($key): type}'
cat yg_gis.json | jq '. as $source | keys[] | . as $key | $source[.] | ($key), type'
cat yg_gis.json | jq '. as $source | keys[] | . as $key | $source[.] | {($key): type}'
cat yg_gis.json | jq '[4,5,6][] | {(type): values}'
cat yg_gis.json | jq '. as $source | keys[] | . as $key | $source[.] | {($key): type}' 
cat yg_gis.json | jq '[{number: 4},{number:5},{number: 6}][] | {(type): values}'
cat yg_gis.json | jq '[{number: 4},{number:5},{number: 6}] | {(type): values}'
cat yg_gis.json | jq '[{value: 4},{value:5},{value: 6}] | {(type): values}'
cat yg_gis.json | jq '[{value: 4},{value:5},{value: 6}][] | {(type): values}'
cat yg_gis.json | jq '[{value: 4},{value:5},{value: 6}][] | .. | {(type): values}'
cat yg_gis.json | jq '[{value: 4},{value:5},{value: 6}][][] | .. | {(type): values}'
cat yg_gis.json | jq '[{value: 4, test: 6},{value:5},{value: 6}][][] | .. | {(type): values}'
cat yg_gis.json | jq '[{value: 4, test: 6},{value:5},{value: 6}][][] |{test:  .. | {(type): values}} '
cat yg_gis.json | jq '[{value: 4, test: 6},{value:5},{value: 6}][][] |  .. | {(type): values} '
cat yg_gis.json | jq '[{value: 4, test: 6},{value:5,test: 7},{value: 6}][][] |  .. | {(type): values} '
cat yg_gis.json | jq '[{value: 4, test: 6},{value:5,test: 7},{value: 6}][][] | [ .. | {(type): values}]'
cat yg_gis.json | jq '[[{value: 4, test: 6},{value:5,test: 7},{value: 6}][][] |  .. | {(type): values}]'
cat yg_gis.json | jq '.[][] |  .. | {(type): values}] '
cat yg_gis.json | jq '.[] |  .. | {(type): values}] '
jq '[1,2,3] |=.'
jq '[1,2,3] |=.+1'
jq '[1,2,3] |=. + 1'
jq '[1,2,3] |= . + 1'
jq '[1,2,3] |= (. + 1)'
cat yg_gis.json | jq '[[{value: 4, test: 6},{value:5,test: 7},{value: 6}][][] |  .. | {(type): values}]'
cat yg_gis.json | jq '.[][] |  .. | {(type): values}] '
cat yg_gis.json | jq '.[] |  .. | {(type): values}] '
cat yg_gis.json | jq '.[] | .. | {(type): values} '
cat yg_gis.json | jq '.[] | .. | {(values): type} '
cat yg_gis.json | jq '.[] | .. | {(values): type} '
cat yg_gis.json | jq '. | {(.):type}'
cat yg_gis.json | jq '. | {(.|keys):type}'
cat yg_gis.json | jq '. | {(.| keys[]):type}'
cat yg_gis.json | jq '. | {(.| keys[]):(. | keys[] | type)}'
cat yg_gis.json | jq '. | {(.| keys[]):(.[] | type)}'
cat yg_gis.json | jq '. | {(.[]| keys[]):(.[] | type)}'
cat yg_gis.json | jq '. | {(.[]):(.[] | type)}'
cat yg_gis.json | jq '. | {(.[]):(.[] | type)}'

I think we'd get the result we want if we ran cat yg_gis.json | jq '. as $source | keys[] | . as $key | $source[.] | [[$key, type]]'. The change being [[$key, type]] vs {($key): type}