Skip to content

Instantly share code, notes, and snippets.

@davad
Last active October 9, 2018 02:28
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save davad/214357ce808c65039be92ce45c8f79c4 to your computer and use it in GitHub Desktop.
Save davad/214357ce808c65039be92ce45c8f79c4 to your computer and use it in GitHub Desktop.
#!/bin/bash
# setup ferret and headless chrome
go get -u github.com/MontFerret/ferret
docker pull alpeware/chrome-headless-trunk
docker run -d -p=0.0.0.0:9222:9222 --name=chrome-headless -v /tmp/chromedata/:/data alpeware/chrome-headless-trunk
# grab fql file and setup results directory
wget https://raw.githubusercontent.com/MontFerret/ferret/master/examples/pagination.fql
mkdir -p pagination-results/$(hostname)
# execute script
for i in {0000..1000}; do
echo $i
ferret --param=criteria:\"ferret\" --param=pages:1 < pagination.fql > pagination-results/$(hostname)/result-${i}.json
sleep 5s
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment