Skip to content

Instantly share code, notes, and snippets.

@FelipeSBarros
Created February 16, 2022 19:20
Show Gist options
  • Save FelipeSBarros/fd7566a180c47b795a50ad44634b9c0c to your computer and use it in GitHub Desktop.
Save FelipeSBarros/fd7566a180c47b795a50ad44634b9c0c to your computer and use it in GitHub Desktop.
from scrapy.crawler import CrawlerProcess
from scrapy.utils.log import configure_logging
from scrapy.utils.project import get_project_settings
configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})
process = CrawlerProcess(get_project_settings())
process.crawl('instituicoes')
process.crawl('agencias_reguladoras')
process.start() # the script will block here until the crawling is finished
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment