Skip to content

Instantly share code, notes, and snippets.

@dequis

dequis/scrape.sh

Created June 6, 2015 04:53
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dequis/c06f64a855ae5c73972a to your computer and use it in GitHub Desktop.
Save dequis/c06f64a855ae5c73972a to your computer and use it in GitHub Desktop.
Download ZIP
radare.today index generator/scraper. requires httpie, pup, jq https://github.com/radare/radare2/wiki/radare.today-index
Raw
scrape.sh
baseurl=http://radare.today
maxpage=$(http -v HEAD $baseurl/page/9999/ | grep Location | grep -o '[0-9]*')
for page in $(seq 1 $maxpage); do
http --follow $baseurl/page/$page/ | pup -p 'h1.post-title a json{}' | jq '.[] | {"title": .text, "url": .href}' >> output.json
done
cat output.json | jq -r '" * [\(.title)]('$baseurl'\(.url))"' > result.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment