Skip to content

Instantly share code, notes, and snippets.

@random-robbie

random-robbie/url.sh

Last active July 9, 2018 09:14
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save random-robbie/3e67b0de551265fb397c4035202b795e to your computer and use it in GitHub Desktop.
Save random-robbie/3e67b0de551265fb397c4035202b795e to your computer and use it in GitHub Desktop.
Download ZIP
URL dump
Raw
url.sh
#!/bin/bash
curl -sX GET "http://index.commoncrawl.org/CC-MAIN-2018-26-index?url=*.google.com&output=json" | jq -r .url | sort -
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment