Skip to content

Instantly share code, notes, and snippets.

@iamumairayub
Created August 26, 2021 09:20
Show Gist options
  • Save iamumairayub/452432a2e78255de890e5e3d925efaa4 to your computer and use it in GitHub Desktop.
Save iamumairayub/452432a2e78255de890e5e3d925efaa4 to your computer and use it in GitHub Desktop.
Use settings of Scrapy Spider as provided in command line arguments or sent by ScrapyD.
import sys
class MySpider(scrapy.Spider):
name = 'my_spider'
CONCURRENT_REQUESTS = 10
for x in sys.argv:
if 'CONCURRENT_REQUESTS=' in x:
CONCURRENT_REQUESTS = x.split('CONCURRENT_REQUESTS=')[1]
break
custom_settings = {
'CONCURRENT_REQUESTS': CONCURRENT_REQUESTS
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment