Skip to content

Instantly share code, notes, and snippets.

@kohn

kohn/example.py Secret

Created May 12, 2016 07:51
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save kohn/7d442d1bf4dc6963a4ec405964c5cd83 to your computer and use it in GitHub Desktop.
Save kohn/7d442d1bf4dc6963a4ec405964c5cd83 to your computer and use it in GitHub Desktop.
def parse(self, response):
if banned():
yield Request(url=response.url,
meta={"change_proxy": True},
callback=self.parse)
else:
pass
@ickma
Copy link

ickma commented Jun 6, 2016

你好,banned并不是scrapy的内置方法,怎么识别被ban?无意中发现的这个gist,正是我今天在思考的问题,github已经follow了你!呵呵

@xujipm
Copy link

xujipm commented Jan 5, 2017

band() 是你自己根据趴下来的内容进行判断是否被ban!

@mugbya
Copy link

mugbya commented Nov 8, 2017

哦,我这被 ban 了连网站都打不开,都没内容...

@Developer27149
Copy link

Developer27149 commented May 2, 2018

您好!meta={"change_proxy": True},元数据设置这个值,会引发什么才能更换代理?我找到了免费代理池,再由flask提供API,写入代理中间件,每次请求,都会调用一次中间件是吗?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment