Skip to content

Instantly share code, notes, and snippets.

@bensteinberg

bensteinberg/gist:a4ec60783e813471a6445e7d08f19cd8

Created April 3, 2024 19:15
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bensteinberg/a4ec60783e813471a6445e7d08f19cd8 to your computer and use it in GitHub Desktop.
Save bensteinberg/a4ec60783e813471a6445e7d08f19cd8 to your computer and use it in GitHub Desktop.
Download ZIP
browsertrix-crawler command
Raw
gistfile1.txt
docker run -v $PWD/wikipages.txt:/app/seedFile.txt -v $PWD/crawls:/crawls/ webrecorder/browsertrix-crawler crawl --seedFile /app/seedFile.txt --combineWARC --depth 0 --workers 4 --behaviors autoplay,autofetch --collection wiki
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment