Skip to content

Instantly share code, notes, and snippets.

@oltarasenko

oltarasenko/exs

Last active April 16, 2021 21:27
Show Gist options
  • Save oltarasenko/ca66d10aa857cc5d8938607f6a4fa4ed to your computer and use it in GitHub Desktop.
Save oltarasenko/ca66d10aa857cc5d8938607f6a4fa4ed to your computer and use it in GitHub Desktop.
Download ZIP
Raw
exs
# in config.exs
import Config
config :crawly,
closespider_timeout: :disabled,
closespider_itemcount: :disabled,
concurrent_requests_per_domain: 2,
middlewares: [
Crawly.Middlewares.DomainFilter,
Crawly.Middlewares.UniqueRequest
]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment