Skip to content

Instantly share code, notes, and snippets.

@dequis dequis/scrape.sh
Created Jun 6, 2015

Embed
What would you like to do?
radare.today index generator/scraper. requires httpie, pup, jq https://github.com/radare/radare2/wiki/radare.today-index
baseurl=http://radare.today
maxpage=$(http -v HEAD $baseurl/page/9999/ | grep Location | grep -o '[0-9]*')
for page in $(seq 1 $maxpage); do
http --follow $baseurl/page/$page/ | pup -p 'h1.post-title a json{}' | jq '.[] | {"title": .text, "url": .href}' >> output.json
done
cat output.json | jq -r '" * [\(.title)]('$baseurl'\(.url))"' > result.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.