Skip to content

Instantly share code, notes, and snippets.

@akkuman

akkuman/get_names_from_scrapy.py

Created October 13, 2021 09:25
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save akkuman/7cb5db45261d76f7ba1c46fe90178e44 to your computer and use it in GitHub Desktop.
Save akkuman/7cb5db45261d76f7ba1c46fe90178e44 to your computer and use it in GitHub Desktop.
Download ZIP
[获取scrapy所有spider name] #python #scrapy
Raw
get_names_from_scrapy.py
from scrapy import spiderloader
from scrapy.utils import project
def get_all_spider_names() -> typing.List[str]:
os.environ.setdefault('SCRAPY_SETTINGS_MODULE', 'darkweb.settings')
settings = project.get_project_settings()
spider_loader = spiderloader.SpiderLoader.from_settings(settings)
return spider_loader.list()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment