Skip to content

Instantly share code, notes, and snippets.

@redapple

redapple/scrapyshell

Created December 30, 2014 14:25
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save redapple/8269818915cc2c337dc2 to your computer and use it in GitHub Desktop.
Save redapple/8269818915cc2c337dc2 to your computer and use it in GitHub Desktop.
Download ZIP
YouTube js2xml
Raw
scrapyshell
$ scrapy shell "https://www.youtube.com/watch?v=1EFnX1UkXVU"
/usr/local/lib/python2.7/dist-packages/twisted/internet/_sslverify.py:184: UserWarning: You do not have the service_identity module installed. Please install it from <https://pypi.python.org/pypi/service_identity>. Without the service_identity module and a recent enough pyOpenSSL tosupport it, Twisted can perform only rudimentary TLS client hostnameverification. Many valid certificate/hostname mappings may be rejected.
verifyHostname, VerificationError = _selectVerifyImplementation()
2014-12-30 15:18:08+0100 [scrapy] INFO: Scrapy 0.24.4 started (bot: scrapybot)
2014-12-30 15:18:08+0100 [scrapy] INFO: Optional features available: ssl, http11, boto, django
2014-12-30 15:18:08+0100 [scrapy] INFO: Overridden settings: {'LOGSTATS_INTERVAL': 0}
2014-12-30 15:18:08+0100 [scrapy] INFO: Enabled extensions: TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2014-12-30 15:18:08+0100 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2014-12-30 15:18:08+0100 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2014-12-30 15:18:08+0100 [scrapy] INFO: Enabled item pipelines:
2014-12-30 15:18:08+0100 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2014-12-30 15:18:08+0100 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080
2014-12-30 15:18:08+0100 [default] INFO: Spider opened
2014-12-30 15:18:09+0100 [default] DEBUG: Crawled (200) <GET https://www.youtube.com/watch?v=1EFnX1UkXVU> (referer: None)
[s] Available Scrapy objects:
[s] crawler <scrapy.crawler.Crawler object at 0x7f381d586950>
[s] item {}
[s] request <GET https://www.youtube.com/watch?v=1EFnX1UkXVU>
[s] response <200 https://www.youtube.com/watch?v=1EFnX1UkXVU>
[s] settings <scrapy.settings.Settings object at 0x7f381dc5abd0>
[s] spider <Spider 'default' at 0x7f381cf805d0>
[s] Useful shortcuts:
[s] shelp() Shell help (print this help)
[s] fetch(req_or_url) Fetch request (or URL) and update local objects
[s] view(response) View response in a browser
In [1]: import js2xml
In [2]: import urlparse
In [3]: import pprint
In [4]: for script in response.css('#player script').xpath('string()').extract():
jstree = js2xml.parse(script)
data = js2xml.jsonlike.getall(jstree)
for d in data:
pprint.pprint(d)
...:
{}
{'args': {'account_playback_token': 'QUFFLUhqa0sweExRZno5OHZEaGcwWVVQaXAxVWh0NUNFZ3xBQ3Jtc0tseE9DRUw3cFVRbkFGN1hub2VmQlNERGl3WjFIQV84aTI0b0lxZnhwdDZKRl96N1g5eWN3dkZER1pFbVM4dS1FeWJoc1FJeTBXdS0tbU5LY1NsWngtSHY1R0hoTl9xdy1iWUNoam1nRFM2czEweVdMNA==',
'adaptive_fmts': 'size=1280x720&clen=51269588&fps=15&itag=136&init=0-709&projection_type=1&lmt=1390057935768626&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D716.867%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26keepalive%3Dyes%26clen%3D51269588%26gir%3Dyes%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D136%26mime%3Dvideo%252Fmp4%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26mm%3D31%26lmt%3D1390057935768626%26initcwndbps%3D872500%26sparams%3Dclen%252Cdur%252Cgir%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Ckeepalive%252Clmt%252Cmime%252Cmm%252Cms%252Cmv%252Csource%252Cupn%252Cexpire%26signature%3D5D9C16DB98AE5663FFD7E2B30FA57A5A69BD16FE.1FCDA50C102DFCA2CB68B7CF44FBDBF9D3D2DFA4%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0&bitrate=2209038&index=710-2469&type=video%2Fmp4%3B+codecs%3D%22avc1.4d401f%22,size=1280x720&clen=22039342&fps=1&itag=247&init=0-234&projection_type=1&lmt=1399438405323165&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D716.867%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26keepalive%3Dyes%26clen%3D22039342%26gir%3Dyes%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D247%26mime%3Dvideo%252Fwebm%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26mm%3D31%26lmt%3D1399438405323165%26initcwndbps%3D872500%26sparams%3Dclen%252Cdur%252Cgir%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Ckeepalive%252Clmt%252Cmime%252Cmm%252Cms%252Cmv%252Csource%252Cupn%252Cexpire%26signature%3D7D3B6E4F878C938C73BCA403D48AD9D282E1CEA4.CED0F017994E3005CA844178343D67F9982F928C%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0&bitrate=931883&index=235-2691&type=video%2Fwebm%3B+codecs%3D%22vp9%22,size=854x480&clen=29188874&fps=15&itag=135&init=0-708&projection_type=1&lmt=1390057935768800&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D716.867%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26keepalive%3Dyes%26clen%3D29188874%26gir%3Dyes%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D135%26mime%3Dvideo%252Fmp4%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26mm%3D31%26lmt%3D1390057935768800%26initcwndbps%3D872500%26sparams%3Dclen%252Cdur%252Cgir%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Ckeepalive%252Clmt%252Cmime%252Cmm%252Cms%252Cmv%252Csource%252Cupn%252Cexpire%26signature%3D9BE1D4EC9F3CBD1C4D5B5013FFD960F5DDD3F1B1.046E49BABF2FCA5BBEFFE085C9C6D28F5DCAA4C4%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0&bitrate=1107230&index=709-2468&type=video%2Fmp4%3B+codecs%3D%22avc1.4d401e%22,size=854x480&clen=11746236&fps=1&itag=244&init=0-234&projection_type=1&lmt=1399438459871028&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D716.867%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26keepalive%3Dyes%26clen%3D11746236%26gir%3Dyes%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D244%26mime%3Dvideo%252Fwebm%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26mm%3D31%26lmt%3D1399438459871028%26initcwndbps%3D872500%26sparams%3Dclen%252Cdur%252Cgir%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Ckeepalive%252Clmt%252Cmime%252Cmm%252Cms%252Cmv%252Csource%252Cupn%252Cexpire%26signature%3D0EAAC235D3245D9440F7B8153F35F63524928E21.57A2C1933F956F279EE841DA957D92420D56FFB7%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0&bitrate=600985&index=235-2672&type=video%2Fwebm%3B+codecs%3D%22vp9%22,size=640x360&clen=14680908&fps=15&itag=134&init=0-708&projection_type=1&lmt=1390057931792975&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D716.867%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26keepalive%3Dyes%26clen%3D14680908%26gir%3Dyes%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D134%26mime%3Dvideo%252Fmp4%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26mm%3D31%26lmt%3D1390057931792975%26initcwndbps%3D872500%26sparams%3Dclen%252Cdur%252Cgir%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Ckeepalive%252Clmt%252Cmime%252Cmm%252Cms%252Cmv%252Csource%252Cupn%252Cexpire%26signature%3DB39FB8A6B4CFEEFC2E14C255BB580694D45CF0E0.DD72C2857637440454A12EA4804A687383E20721%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0&bitrate=615912&index=709-2468&type=video%2Fmp4%3B+codecs%3D%22avc1.4d4016%22,size=640x360&clen=7365914&fps=1&itag=243&init=0-234&projection_type=1&lmt=1399438453314211&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D716.867%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26keepalive%3Dyes%26clen%3D7365914%26gir%3Dyes%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D243%26mime%3Dvideo%252Fwebm%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26mm%3D31%26lmt%3D1399438453314211%26initcwndbps%3D872500%26sparams%3Dclen%252Cdur%252Cgir%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Ckeepalive%252Clmt%252Cmime%252Cmm%252Cms%252Cmv%252Csource%252Cupn%252Cexpire%26signature%3D59BBFADA63188608B001EF480CB79A621D2B50F4.47DD8B7C322EA8A4227CADE668D84B8AA710F7A0%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0&bitrate=342280&index=235-2671&type=video%2Fwebm%3B+codecs%3D%22vp9%22,size=426x240&clen=11713153&fps=15&itag=133&init=0-673&projection_type=1&lmt=1390057930787681&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D716.867%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26keepalive%3Dyes%26clen%3D11713153%26gir%3Dyes%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D133%26mime%3Dvideo%252Fmp4%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26mm%3D31%26lmt%3D1390057930787681%26initcwndbps%3D872500%26sparams%3Dclen%252Cdur%252Cgir%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Ckeepalive%252Clmt%252Cmime%252Cmm%252Cms%252Cmv%252Csource%252Cupn%252Cexpire%26signature%3DA7E9B427E61DFC68D72F790A770962A72BAEFC90.994A60EFA6451EEEA869084ED78343511582A136%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0&bitrate=276825&index=674-2433&type=video%2Fmp4%3B+codecs%3D%22avc1.4d4015%22,size=426x240&clen=4124001&fps=1&itag=242&init=0-233&projection_type=1&lmt=1399438372886065&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D716.867%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26keepalive%3Dyes%26clen%3D4124001%26gir%3Dyes%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D242%26mime%3Dvideo%252Fwebm%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26mm%3D31%26lmt%3D1399438372886065%26initcwndbps%3D872500%26sparams%3Dclen%252Cdur%252Cgir%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Ckeepalive%252Clmt%252Cmime%252Cmm%252Cms%252Cmv%252Csource%252Cupn%252Cexpire%26signature%3DF3F2FF91A8E8BA2D5616AA6201E8A072F199C7DD.AF108A121BD6F210D45EBA334590635C3E361322%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0&bitrate=191990&index=234-2669&type=video%2Fwebm%3B+codecs%3D%22vp9%22,size=256x144&clen=5784457&fps=15&itag=160&init=0-670&projection_type=1&lmt=1390057932787940&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D716.867%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26keepalive%3Dyes%26clen%3D5784457%26gir%3Dyes%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D160%26mime%3Dvideo%252Fmp4%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26mm%3D31%26lmt%3D1390057932787940%26initcwndbps%3D872500%26sparams%3Dclen%252Cdur%252Cgir%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Ckeepalive%252Clmt%252Cmime%252Cmm%252Cms%252Cmv%252Csource%252Cupn%252Cexpire%26signature%3D65C73041441D09F2F17C90A49F6F5DB9AA8F2411.E8D2C829504C438CC0B285036BE826FB075B0822%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0&bitrate=124664&index=671-2430&type=video%2Fmp4%3B+codecs%3D%22avc1.4d400c%22,clen=11387826&index=592-1487&projection_type=1&itag=140&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D716.985%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26keepalive%3Dyes%26clen%3D11387826%26gir%3Dyes%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D140%26mime%3Daudio%252Fmp4%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26mm%3D31%26lmt%3D1390057931808185%26initcwndbps%3D872500%26sparams%3Dclen%252Cdur%252Cgir%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Ckeepalive%252Clmt%252Cmime%252Cmm%252Cms%252Cmv%252Csource%252Cupn%252Cexpire%26signature%3DF1B7641F63A24CFDD4390628495E83128C1F55B5.DA38A0617768B1FA9A11A1FB14B96B990EED4167%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0&init=0-591&lmt=1390057931808185&type=audio%2Fmp4%3B+codecs%3D%22mp4a.40.2%22&bitrate=128358,fps=1&clen=6794166&index=4452-5672&projection_type=1&itag=171&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D716.926%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26keepalive%3Dyes%26clen%3D6794166%26gir%3Dyes%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D171%26mime%3Daudio%252Fwebm%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26mm%3D31%26lmt%3D1399438363885243%26initcwndbps%3D872500%26sparams%3Dclen%252Cdur%252Cgir%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Ckeepalive%252Clmt%252Cmime%252Cmm%252Cms%252Cmv%252Csource%252Cupn%252Cexpire%26signature%3D277C9CD03A938BA2CD000A0D8CA2A875ADF3CA91.8D0072E2C5B75FCBF0472784B5EFDF6D71CA78D0%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0&init=0-4451&lmt=1399438363885243&type=audio%2Fwebm%3B+codecs%3D%22vorbis%22&bitrate=80798',
'allow_embed': '1',
'allow_ratings': '1',
'atc': 'a=3&b=nhjwMM7ySu8wj8OhutnokFK8Dvs&c=1419949090&d=1&e=1EFnX1UkXVU&c3a=28&c1a=1&hh=hKbH2J9f2WwblpFs2hvo0H17oZo',
'author': 'Michael Herman',
'avg_rating': '4.948387146',
'c': 'WEB',
'cc3_module': '1',
'cc_asr': '1',
'cc_font': 'Arial Unicode MS, arial, verdana, _sans',
'cc_fonts_url': 'https://s.ytimg.com/yts/swfbin/player-vfly1u_c5/fonts708.swf',
'cc_load_policy': '2',
'cc_module': 'https://s.ytimg.com/yts/swfbin/player-vfly1u_c5/subtitle_module.swf',
'cl': '82697338',
'cr': 'FR',
'csi_page_type': 'watch,watch7',
'dash': '1',
'dashmpd': 'http://manifest.googlevideo.com/api/manifest/dash/id/o-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy/sver/3/mm/31/playback_host/r3---sn-25ge7n7d.googlevideo.com/source/youtube/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm_sd_hd_clear%2Cwebm2_sd_hd_clear/expire/1419970690/sparams/as%2Cid%2Cip%2Cipbits%2Citag%2Cmm%2Cms%2Cmv%2Cplayback_host%2Csource%2Cexpire/key/yt5/itag/0/ip/89.84.122.217/ms/au/fexp/900718%2C927622%2C931342%2C932404%2C938809%2C9405699%2C9406022%2C940927%2C940940%2C941004%2C943917%2C947209%2C947218%2C948124%2C952302%2C952605%2C952901%2C955110%2C955301%2C957103%2C957105%2C957201/mt/1419949043/signature/4604928873F3D6F471BD266E06267080755D4BC3.30D9BE16A530300FE442FA1FB10D74BC6C94EFF3/mv/m/upn/5QFvFRIqKzs/ipbits/0',
'enablecsi': '1',
'enablejsapi': 1,
'eventid': 'IrSiVP-kC4v4cKrwgRg',
'fexp': '900718,927622,931342,932404,938809,9405699,9406022,940927,940940,941004,943917,947209,947218,948124,952302,952605,952901,955110,955301,957103,957105,957201',
'fmt_list': '22/1280x720/9/0/115,43/640x360/99/0/0,18/640x360/9/0/115,5/426x240/7/0/0,36/426x240/99/1/0,17/256x144/99/1/0',
'hl': 'en_US',
'host_language': 'en',
'idpj': '-6',
'iurl': 'https://i.ytimg.com/vi/1EFnX1UkXVU/hqdefault.jpg',
'iurlhq': 'https://i.ytimg.com/vi/1EFnX1UkXVU/hqdefault.jpg',
'iurlmaxres': 'https://i.ytimg.com/vi/1EFnX1UkXVU/maxresdefault.jpg',
'iurlmq': 'https://i.ytimg.com/vi/1EFnX1UkXVU/mqdefault.jpg',
'iurlsd': 'https://i.ytimg.com/vi/1EFnX1UkXVU/sddefault.jpg',
'iv3_module': '1',
'iv_invideo_url': 'https://www.youtube.com/annotations_invideo?cta=2&video_id=1EFnX1UkXVU',
'iv_load_policy': '1',
'iv_module': 'https://s.ytimg.com/yts/swfbin/player-vfly1u_c5/iv_module.swf',
'keywords': 'Scrapy,Python,scraping,python scrapy,web scraping',
'ldpj': '-25',
'length_seconds': '717',
'loaderUrl': 'https://www.youtube.com/watch?v=1EFnX1UkXVU',
'no_get_video_log': '1',
'of': 'lNeUuIm8BRrYa4UFYW3Vbw',
'plid': 'AAULb6kfjbEHoNwt',
'pltype': 'contentugc',
'probe_url': 'http://r5---sn-5hn7ym7z.googlevideo.com/videogoodput?id=o-ACe-sIXL0cLvgJC4v5mIahOxT1PHw4zDPr8ZGMCgqwQI&source=goodput&range=0-99999&expire=1419952690&ip=89.84.122.217&ms=pm&mm=35&nh=EAk&sparams=id,source,range,expire,ip,ms,mm,nh&signature=3B4094AEE2FC1C0142BCEDB115F785607DEC0CF1.04988A5889C0348F50D45D76A7D6831155C91407&key=cms1',
'ptk': 'youtube_none',
'ssl': '1',
'storyboard_spec': 'https://i.ytimg.com/sb/1EFnX1UkXVU/storyboard3_L$L/$N.jpg|48#27#100#10#10#0#default#28F7DFM7_rVji4ZXj1Inr3KDPBE|80#45#145#10#10#5000#M$M#oy8NWkx8UFfdFYJoDyKoK-F6EUo|160#90#145#5#5#5000#M$M#RPAH69FExaDD6f0lYwoCjc64vI8',
't': '1',
'thumbnail_url': 'https://i.ytimg.com/vi/1EFnX1UkXVU/default.jpg',
'timestamp': '1419949090',
'title': 'Scraping Web Pages with Scrapy',
'tmi': '1',
'token': '1',
'ttsurl': 'https://www.youtube.com/api/timedtext?hl=en_US&sparams=asr_langs%2Ccaps%2Cv%2Cexpire&caps=asr&key=yttt1&signature=2FBB2AD234CF9C58E1E461189C0EFBFB77DB2E85.E732FD8A65CAF5F044CBB2C0877093826E88DB7F&v=1EFnX1UkXVU&expire=1419974290&asr_langs=ko%2Cru%2Cde%2Cen%2Cja%2Ces%2Cit%2Cfr%2Cpt%2Cnl',
'ucid': 'UCt7yOnL7bI7yCa1Xe_GTjJQ',
'url_encoded_fmt_stream_map': 'fallback_host=tc.v18.cache4.googlevideo.com&quality=hd720&itag=22&type=video%2Fmp4%3B+codecs%3D%22avc1.64001F%2C+mp4a.40.2%22&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D716.985%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26mm%3D31%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D22%26mime%3Dvideo%252Fmp4%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26initcwndbps%3D872500%26sparams%3Ddur%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Cmime%252Cmm%252Cms%252Cmv%252Cratebypass%252Csource%252Cupn%252Cexpire%26ratebypass%3Dyes%26signature%3D75A8510F49A9C73C72BC4F4A8759320481305D26.EA7ABB7DD01D7B4BA5228ABD8DF8DD47AB73A3A1%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0,fallback_host=tc.v20.cache6.googlevideo.com&quality=medium&itag=43&type=video%2Fwebm%3B+codecs%3D%22vp8.0%2C+vorbis%22&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D0.000%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26mm%3D31%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D43%26mime%3Dvideo%252Fwebm%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26initcwndbps%3D872500%26sparams%3Ddur%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Cmime%252Cmm%252Cms%252Cmv%252Cratebypass%252Csource%252Cupn%252Cexpire%26ratebypass%3Dyes%26signature%3DE17363F74C7068BEB4DB31FC90AEF2EA70A3C233.F634AC2BD1B5A6B27E1DDFB4FB09DE7C04D1DF0E%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0,fallback_host=tc.v13.cache4.googlevideo.com&quality=medium&itag=18&type=video%2Fmp4%3B+codecs%3D%22avc1.42001E%2C+mp4a.40.2%22&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D716.985%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26mm%3D31%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D18%26mime%3Dvideo%252Fmp4%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26initcwndbps%3D872500%26sparams%3Ddur%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Cmime%252Cmm%252Cms%252Cmv%252Cratebypass%252Csource%252Cupn%252Cexpire%26ratebypass%3Dyes%26signature%3D78201511AECE7F328D67AA08EC40E22777C62616.6B0C1787F391F30F1D28D8C2BCD6E67C71F1BB5F%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0,fallback_host=tc.v4.cache4.googlevideo.com&quality=small&itag=5&type=video%2Fx-flv&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D716.983%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26mm%3D31%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D5%26mime%3Dvideo%252Fx-flv%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26initcwndbps%3D872500%26sparams%3Ddur%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Cmime%252Cmm%252Cms%252Cmv%252Csource%252Cupn%252Cexpire%26signature%3DDE27A5283FB425F79CC1ACBB67D0B20FF07D5BD5.DBACE3E830A573BF4092AC442C99278D4CFF549F%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0,fallback_host=tc.v4.cache5.googlevideo.com&quality=small&itag=36&type=video%2F3gpp%3B+codecs%3D%22mp4v.20.3%2C+mp4a.40.2%22&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D717.125%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26mm%3D31%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D36%26mime%3Dvideo%252F3gpp%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26initcwndbps%3D872500%26sparams%3Ddur%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Cmime%252Cmm%252Cms%252Cmv%252Csource%252Cupn%252Cexpire%26signature%3DE9DD3B41DDA5B39F12D7311682DEB24A376F04C9.0C3EEEFED598AF77E877D361B57385CE5941303F%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0,fallback_host=tc.v9.cache5.googlevideo.com&quality=small&itag=17&type=video%2F3gpp%3B+codecs%3D%22mp4v.20.3%2C+mp4a.40.2%22&url=http%3A%2F%2Fr3---sn-25ge7n7d.googlevideo.com%2Fvideoplayback%3Fdur%3D717.217%26id%3Do-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy%26mm%3D31%26ip%3D89.84.122.217%26key%3Dyt5%26itag%3D17%26mime%3Dvideo%252F3gpp%26source%3Dyoutube%26ms%3Dau%26fexp%3D900718%252C927622%252C931342%252C932404%252C938809%252C9405699%252C9406022%252C940927%252C940940%252C941004%252C943917%252C947209%252C947218%252C948124%252C952302%252C952605%252C952901%252C955110%252C955301%252C957103%252C957105%252C957201%26mv%3Dm%26mt%3D1419949043%26sver%3D3%26initcwndbps%3D872500%26sparams%3Ddur%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Cmime%252Cmm%252Cms%252Cmv%252Csource%252Cupn%252Cexpire%26signature%3DE4199F944FC4A5A1DFBAD4562EB628E62B53FD27.FA0A2D69378E3AB8B4E50FD55A2F64CA7A048EA1%26expire%3D1419970690%26upn%3D5QFvFRIqKzs%26ipbits%3D0',
'video_id': '1EFnX1UkXVU',
'view_count': '52035',
'vq': 'auto',
'watermark': ',https://s.ytimg.com/yts/img/watermark/youtube_watermark-vflHX6b6E.png,https://s.ytimg.com/yts/img/watermark/youtube_hd_watermark-vflAzLcD6.png'},
'assets': {'css': '//s.ytimg.com/yts/cssbin/www-player-vflPfi1TF.css',
'html': '/html5_player_template',
'js': '//s.ytimg.com/yts/jsbin/html5player-en_US-vflw4H1P-/html5player.js'},
'attrs': {'id': 'movie_player'},
'html5': False,
'messages': {'player_fallback': ['Adobe Flash Player or an HTML5 supported browser is required for video playback.<br><a href="http://get.adobe.com/flashplayer/">Get the latest Flash Player </a><br><a href="/html5">Learn more about upgrading to an HTML5 browser</a>']},
'min_version': '8.0.0',
'params': {'allowfullscreen': 'true',
'allowscriptaccess': 'always',
'bgcolor': '#000000'},
'sts': 16427,
'url': 'https://s.ytimg.com/yts/swfbin/player-vfly1u_c5/watch_as3.swf',
'url_v8': 'https://s.ytimg.com/yts/swfbin/player-vfly1u_c5/cps.swf',
'url_v9as2': 'https://s.ytimg.com/yts/swfbin/player-vfly1u_c5/cps.swf'}
[]
In [5]: for script in response.css('#player script').xpath('string()').extract():
...: jstree = js2xml.parse(script)
...: data = js2xml.jsonlike.getall(jstree)
...: for d in data:
...: try:
...: if d:
...: pprint.pprint(urlparse.parse_qsl(d.get("args", {}).get("url_encoded_fmt_stream_map", "")))
...: except:
...: pass
...:
[('fallback_host', 'tc.v18.cache4.googlevideo.com'),
('quality', 'hd720'),
('itag', '22'),
('type', 'video/mp4; codecs="avc1.64001F, mp4a.40.2"'),
('url',
'http://r3---sn-25ge7n7d.googlevideo.com/videoplayback?dur=716.985&id=o-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy&mm=31&ip=89.84.122.217&key=yt5&itag=22&mime=video%2Fmp4&source=youtube&ms=au&fexp=900718%2C927622%2C931342%2C932404%2C938809%2C9405699%2C9406022%2C940927%2C940940%2C941004%2C943917%2C947209%2C947218%2C948124%2C952302%2C952605%2C952901%2C955110%2C955301%2C957103%2C957105%2C957201&mv=m&mt=1419949043&sver=3&initcwndbps=872500&sparams=dur%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Cmime%2Cmm%2Cms%2Cmv%2Cratebypass%2Csource%2Cupn%2Cexpire&ratebypass=yes&signature=75A8510F49A9C73C72BC4F4A8759320481305D26.EA7ABB7DD01D7B4BA5228ABD8DF8DD47AB73A3A1&expire=1419970690&upn=5QFvFRIqKzs&ipbits=0,fallback_host=tc.v20.cache6.googlevideo.com'),
('quality', 'medium'),
('itag', '43'),
('type', 'video/webm; codecs="vp8.0, vorbis"'),
('url',
'http://r3---sn-25ge7n7d.googlevideo.com/videoplayback?dur=0.000&id=o-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy&mm=31&ip=89.84.122.217&key=yt5&itag=43&mime=video%2Fwebm&source=youtube&ms=au&fexp=900718%2C927622%2C931342%2C932404%2C938809%2C9405699%2C9406022%2C940927%2C940940%2C941004%2C943917%2C947209%2C947218%2C948124%2C952302%2C952605%2C952901%2C955110%2C955301%2C957103%2C957105%2C957201&mv=m&mt=1419949043&sver=3&initcwndbps=872500&sparams=dur%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Cmime%2Cmm%2Cms%2Cmv%2Cratebypass%2Csource%2Cupn%2Cexpire&ratebypass=yes&signature=E17363F74C7068BEB4DB31FC90AEF2EA70A3C233.F634AC2BD1B5A6B27E1DDFB4FB09DE7C04D1DF0E&expire=1419970690&upn=5QFvFRIqKzs&ipbits=0,fallback_host=tc.v13.cache4.googlevideo.com'),
('quality', 'medium'),
('itag', '18'),
('type', 'video/mp4; codecs="avc1.42001E, mp4a.40.2"'),
('url',
'http://r3---sn-25ge7n7d.googlevideo.com/videoplayback?dur=716.985&id=o-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy&mm=31&ip=89.84.122.217&key=yt5&itag=18&mime=video%2Fmp4&source=youtube&ms=au&fexp=900718%2C927622%2C931342%2C932404%2C938809%2C9405699%2C9406022%2C940927%2C940940%2C941004%2C943917%2C947209%2C947218%2C948124%2C952302%2C952605%2C952901%2C955110%2C955301%2C957103%2C957105%2C957201&mv=m&mt=1419949043&sver=3&initcwndbps=872500&sparams=dur%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Cmime%2Cmm%2Cms%2Cmv%2Cratebypass%2Csource%2Cupn%2Cexpire&ratebypass=yes&signature=78201511AECE7F328D67AA08EC40E22777C62616.6B0C1787F391F30F1D28D8C2BCD6E67C71F1BB5F&expire=1419970690&upn=5QFvFRIqKzs&ipbits=0,fallback_host=tc.v4.cache4.googlevideo.com'),
('quality', 'small'),
('itag', '5'),
('type', 'video/x-flv'),
('url',
'http://r3---sn-25ge7n7d.googlevideo.com/videoplayback?dur=716.983&id=o-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy&mm=31&ip=89.84.122.217&key=yt5&itag=5&mime=video%2Fx-flv&source=youtube&ms=au&fexp=900718%2C927622%2C931342%2C932404%2C938809%2C9405699%2C9406022%2C940927%2C940940%2C941004%2C943917%2C947209%2C947218%2C948124%2C952302%2C952605%2C952901%2C955110%2C955301%2C957103%2C957105%2C957201&mv=m&mt=1419949043&sver=3&initcwndbps=872500&sparams=dur%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Cmime%2Cmm%2Cms%2Cmv%2Csource%2Cupn%2Cexpire&signature=DE27A5283FB425F79CC1ACBB67D0B20FF07D5BD5.DBACE3E830A573BF4092AC442C99278D4CFF549F&expire=1419970690&upn=5QFvFRIqKzs&ipbits=0,fallback_host=tc.v4.cache5.googlevideo.com'),
('quality', 'small'),
('itag', '36'),
('type', 'video/3gpp; codecs="mp4v.20.3, mp4a.40.2"'),
('url',
'http://r3---sn-25ge7n7d.googlevideo.com/videoplayback?dur=717.125&id=o-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy&mm=31&ip=89.84.122.217&key=yt5&itag=36&mime=video%2F3gpp&source=youtube&ms=au&fexp=900718%2C927622%2C931342%2C932404%2C938809%2C9405699%2C9406022%2C940927%2C940940%2C941004%2C943917%2C947209%2C947218%2C948124%2C952302%2C952605%2C952901%2C955110%2C955301%2C957103%2C957105%2C957201&mv=m&mt=1419949043&sver=3&initcwndbps=872500&sparams=dur%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Cmime%2Cmm%2Cms%2Cmv%2Csource%2Cupn%2Cexpire&signature=E9DD3B41DDA5B39F12D7311682DEB24A376F04C9.0C3EEEFED598AF77E877D361B57385CE5941303F&expire=1419970690&upn=5QFvFRIqKzs&ipbits=0,fallback_host=tc.v9.cache5.googlevideo.com'),
('quality', 'small'),
('itag', '17'),
('type', 'video/3gpp; codecs="mp4v.20.3, mp4a.40.2"'),
('url',
'http://r3---sn-25ge7n7d.googlevideo.com/videoplayback?dur=717.217&id=o-AMERlvuyknt71bMvL2Sjki6y2WsGz0TDKn11unO3_SQy&mm=31&ip=89.84.122.217&key=yt5&itag=17&mime=video%2F3gpp&source=youtube&ms=au&fexp=900718%2C927622%2C931342%2C932404%2C938809%2C9405699%2C9406022%2C940927%2C940940%2C941004%2C943917%2C947209%2C947218%2C948124%2C952302%2C952605%2C952901%2C955110%2C955301%2C957103%2C957105%2C957201&mv=m&mt=1419949043&sver=3&initcwndbps=872500&sparams=dur%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Cmime%2Cmm%2Cms%2Cmv%2Csource%2Cupn%2Cexpire&signature=E4199F944FC4A5A1DFBAD4562EB628E62B53FD27.FA0A2D69378E3AB8B4E50FD55A2F64CA7A048EA1&expire=1419970690&upn=5QFvFRIqKzs&ipbits=0')]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment