Skip to content

Instantly share code, notes, and snippets.

@YT-er

YT-er/spider.md

Last active January 17, 2019 09:26
Show Gist options
  • Save YT-er/71518efb2201de81e2e40ee0d606ba70 to your computer and use it in GitHub Desktop.
Save YT-er/71518efb2201de81e2e40ee0d606ba70 to your computer and use it in GitHub Desktop.
Download ZIP
scrapy的一些问题记录
Raw
spider.md

Scrapy记录

20190116

start_urls

start_urls是必须的，而且要是list形式，哪怕只有1个，它是Spider类的默认属性。

json输出中文

在settings.py里设置

FEED_EXPORT_ENCODING = 'utf-8'

修改日志级别

在settings.py里设置

LOG_LEVEL = 'WARNING'

20190117

settings.py里ITEM_PIPELINES里的数字

ITEM_PIPELINES = {
   'douban.pipelines.DoubanPipeline': 300,
}

300表示优先级，数字越小，优先级越高

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment