Skip to content

Instantly share code, notes, and snippets.

View luzihang123's full-sized avatar
🎯
疯狂撸码

clark luzihang123

🎯
疯狂撸码
  • Shanghai
View GitHub Profile
import re
import random
import base64
from scrapy import log
class RandomProxy(object):
def __init__(self, settings):
self.proxy_list = settings.get('PROXY_LIST')
fin = open(self.proxy_list)
@miztiik
miztiik / python-scrapy-install-centos.sh
Last active March 3, 2021 04:16
Installing scrapy in CentOS
# Install deltarpm to reduce the size of download
yum install -y deltarpm \
&& yum install -y python-pip \
&& pip install --upgrade pip
# Scrapy Pre-Requisites
yum install -y gcc libffi-devel python-devel openssl-devel
yum install -y libxslt-devel libxml++-devel libxml2-devel
pip install lxml incremental
@gbaman
gbaman / graphql_example.py
Created November 1, 2017 00:18
An example on using the Github GraphQL API with Python 3
# An example to get the remaining rate limit using the Github GraphQL API.
import requests
headers = {"Authorization": "Bearer YOUR API KEY"}
def run_query(query): # A simple function to use requests.post to make the API call. Note the json= section.
request = requests.post('https://api.github.com/graphql', json={'query': query}, headers=headers)
if request.status_code == 200:
@dawand
dawand / download_apk.py
Last active December 9, 2023 01:46
Download APK files from Google Play Store with Python
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
File name: download_apk.py
Author: Dawand Sulaiman
Download APK files from Google Play Store with Python
This script scraps https://apkpure.com to get the apk download link
Make sure you have BeautifulSoup and urllib libraries
"""
@socrateslee
socrateslee / jd_union_api.py
Last active November 1, 2023 03:31
京东联盟开放平台API的一个通用的Client封装
'''
京东联盟开放平台API的一个通用的Client封装。
京东联盟开放平台的文档详见 https://union.jd.com/openplatform
涉及到签名方法的文档见 https://union.jd.com/helpcenter/13246-13247-46301
Example:
client = JdApiClient("<YOUR_APP_KEY>", "<YOUR_SECRET_KEY>")
resp = client.call("jd.union.open.goods.promotiongoodsinfo.query",
{'skuIds':'12072066'})
print(resp.json())
@laggardkernel
laggardkernel / redispipeline.md
Created July 9, 2020 05:05
RedisPipeline for scrapy #python #scrapy

Create a base class RedisPipeline, whichever Pipeline inherits it get a redis connection. Access to the redis conn with self.redis_server.

Dependency: scrapy_redis.

# defaults.py
REDIS_ENCODING = "utf-8"
REDIS_FAILED_URLS_KEY = "%(spidername)s:failed_urls"