Skip to content

Instantly share code, notes, and snippets.

@dequis dequis/
Created Jun 6, 2015

What would you like to do? index generator/scraper. requires httpie, pup, jq
maxpage=$(http -v HEAD $baseurl/page/9999/ | grep Location | grep -o '[0-9]*')
for page in $(seq 1 $maxpage); do
http --follow $baseurl/page/$page/ | pup -p ' a json{}' | jq '.[] | {"title": .text, "url": .href}' >> output.json
cat output.json | jq -r '" * [\(.title)]('$baseurl'\(.url))"' >
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.