-
-
Save kohn/7d442d1bf4dc6963a4ec405964c5cd83 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def parse(self, response): | |
if banned(): | |
yield Request(url=response.url, | |
meta={"change_proxy": True}, | |
callback=self.parse) | |
else: | |
pass |
band() 是你自己根据趴下来的内容进行判断是否被ban!
哦,我这被 ban 了连网站都打不开,都没内容...
您好!meta={"change_proxy": True},元数据设置这个值,会引发什么才能更换代理?我找到了免费代理池,再由flask提供API,写入代理中间件,每次请求,都会调用一次中间件是吗?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
你好,banned并不是scrapy的内置方法,怎么识别被ban?无意中发现的这个gist,正是我今天在思考的问题,github已经follow了你!呵呵