Skip to content

Instantly share code, notes, and snippets.

@jonasfj
Created January 20, 2020 10:47
Show Gist options
  • Save jonasfj/1d4497ce6b444dedb1abf133c17fc82f to your computer and use it in GitHub Desktop.
Save jonasfj/1d4497ce6b444dedb1abf133c17fc82f to your computer and use it in GitHub Desktop.
Notes on how to query all pubspecs on `pub.dev`.

Query pubspec.yaml for all packages

When considering introduction of a new field in pubspec.yaml, or wondering how many packages use a certain feature, it can be useful to query all pubspecs from pub.dev. The following notes have some scripts to make this easy.

File yaml2json.sh

Helper script to convert YAML to JSON.

#!/bin/bash -e

# Copyright 2020 Google LLC.
# SPDX-License-Identifier: Apache-2.0

python2 -c 'import sys, yaml, json; y=yaml.load(sys.stdin.read(),Loader=yaml.SafeLoader); print json.dumps(y)'

File fetch_pubspec.sh

Helper script to fetch the pubspec for a package into the all-pubspecs-json/ folder.

#!/bin/bash -e

# Copyright 2020 Google LLC.
# SPDX-License-Identifier: Apache-2.0

if [ ! -f "all-pubspecs/$1.yaml" ]; then
  echo "$1";
  URL=`curl -s "https://pub.dartlang.org/api/packages/$1/" | jq -r '.latest.archive_url'`
  curl -s -L "$URL" | tar -zxO pubspec.yaml | ./yaml2json.sh | jq -r . | tee > "all-pubspecs-json/$1.yaml"
fi

File fetch_all_pubspecs.sh

Create all-pubspecs-json/ folder and download all pubspecs into this folder as JSON files.

#!/bin/bash -e

# Copyright 2020 Google LLC.
# SPDX-License-Identifier: Apache-2.0

mkdir -p all-pubspecs-json
curl 'https://pub.dartlang.org/api/packages?compact=1' | jq -r '.packages[]' | parallel -j 200 ./fetch_pubspec.sh {}

File query_pubspecs.sh

Query over all pubspecs from the all-pubspecs-json/ folder in parallel using jq queries.

#!/bin/bash -e

# Copyright 2020 Google LLC.
# SPDX-License-Identifier: Apache-2.0

if [ "$1" == "" ]; then
  echo 'Usage: query_pubspecs.sh <jq-query>';
  exit 1;
fi;


export QUERY="$1"
process() {
  VAL=`cat "all-pubspecs-json/$1" | jq -r "$QUERY"`;
  if [ "$VAL" != "null" ]; then
    echo "$1: $VAL";
  fi;
}
export -f process

ls all-pubspecs-json | parallel -j 20 process {};
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment