Skip to content

Instantly share code, notes, and snippets.

@ranafge

ranafge/scrapy shell

Forked from serhii73/scrapy shell
Last active December 30, 2018 04:52
Show Gist options
  • Save ranafge/b5294f14079593b1a0e46839152aa270 to your computer and use it in GitHub Desktop.
Save ranafge/b5294f14079593b1a0e46839152aa270 to your computer and use it in GitHub Desktop.
Download ZIP
scrapy shell and user-agent
Raw
scrapy shell
from scrapy import Request
url = 'http://www.example.com'
req = Request(url, headers={"USER-AGENT" : "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 OPR/45.0.2552.888"})
fetch(req)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment