Skip to content

Instantly share code, notes, and snippets.

@SurendraTamang

SurendraTamang/scrapy_cmdline.py

Created February 9, 2024 04:21
Show Gist options
  • Save SurendraTamang/377da1302dd83cb75266827d39f201f3 to your computer and use it in GitHub Desktop.
Save SurendraTamang/377da1302dd83cb75266827d39f201f3 to your computer and use it in GitHub Desktop.
Download ZIP
Running scrapy spider in python file using scrapy's cmdline module
Raw
scrapy_cmdline.py
from scrapy import cmdline
from datetime import datetime
region = 'depew'
date_now = datetime.now()
# tslco is my scrapy spider and other arguments
cmdline.execute(f"scrapy crawl tslco -a db_connection={region} -o {region}-{date_now.strftime('%Y-%m-%d')}.csv".split())
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment