Skip to content

Instantly share code, notes, and snippets.

View yi-ye-zhi-qiu's full-sized avatar
🐠

liam isaacs yi-ye-zhi-qiu

🐠
  • daash
  • San Francisco, CA
View GitHub Profile
@wvengen
wvengen / scrapy_h2_proxy.py
Last active October 26, 2023 16:17
Scrapy download handler for HTTP/2 over a HTTP/1 proxy (TLS only).
"""
Download handler for HTTP/2 supporting proxy CONNECT over HTTP/1.
You can use this by configuring it in the spider settings:
DOWNLOAD_HANDLERS = {
'https': 'scrapy_h2_proxy.H2DownloadHandler',
}
Tested with Scrapy 2.5.0.
@pawelmhm
pawelmhm / header_order_downloader.py
Last active July 19, 2023 20:02
Scrapy downloader that preserves header order
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, unicode_literals
import logging
from collections import OrderedDict
from time import time
from urllib.parse import urldefrag
from scrapy.core.downloader import Downloader
from scrapy.core.downloader.handlers.http11 import HTTP11DownloadHandler, \