Skip to content

Instantly share code, notes, and snippets.

@pbinkley

pbinkley/crawl-config.yaml

Created March 15, 2022 18:54
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save pbinkley/dce177ee7fb6ee5fb4b1976856722d87 to your computer and use it in GitHub Desktop.
Save pbinkley/dce177ee7fb6ee5fb4b1976856722d87 to your computer and use it in GitHub Desktop.
Download ZIP
SUCHO browsertrix config for a WikiMedia site
Raw
crawl-config.yaml
collection: "wiki-library-kr-ua"
workers: 16
saveState: always
seeds:
- url: https://wiki.library.kr.ua/
include: .*\.wiki\.library\.kr\.ua/
exclude:
- .*action\=.*
- .*page\=.*
- .*limit\=.*
- .*oldid\=.*
- .*title=%D0%A1%D0%BF%D0%B5%D1%86%D1%96%D0%B0%D0%BB%D1%8C%D0%BD%D0%B0\:.*
- .*returnto\=.*
scopeType: "host"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment