Skip to content

Instantly share code, notes, and snippets.

View billy8407's full-sized avatar
🐍
PYTHON

billy8407

🐍
PYTHON
View GitHub Profile
@lorey
lorey / selenium_xhr_requests_via_performance_logging.py
Last active May 13, 2024 09:52
Access Chrome's network tab (e.g. XHR requests) with Selenium
#
# This small example shows you how to access JS-based requests via Selenium
# Like this, one can access raw data for scraping,
# for example on many JS-intensive/React-based websites
#
from time import sleep
from selenium import webdriver
from selenium.webdriver import DesiredCapabilities
@virus-warnning
virus-warnning / ps_exec.py
Created January 16, 2020 09:17
使用 PowerShell 執行程式, 回傳 STDOUT, 並且支援系統管理員身分執行
import os
import random
import subprocess
def ps_exec(cmd, adminPriv=False):
""" 使用 Windows PowerShell Start-Process 執行程式, 回傳 STDOUT, 並且支援以系統管理員身分執行 """
# 產生隨機檔名, 用來儲存 stdout
existed = True
while existed:
@lorey
lorey / avoiding-https-connection-pool-errors.py
Created May 21, 2019 11:54
Dealing with HTTPSConnectionPool errors in requests with adapters and backoff
# this snippet will deal with errors like HTTPSConnectionPool: Max retries exceeded with url...
# by using a backoff factor
# further reading:
# - docs: https://2.python-requests.org/en/master/user/advanced/#transport-adapters
# - stack overflow issue: https://stackoverflow.com/a/47475019
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
@miraculixx
miraculixx / example.MD
Last active February 29, 2024 16:34
Python multiprocess parallel selenium web scraping with improved performance

How to run this

(output as of September 29, 2023)

$ python scraper.py
Does flying slower actually save fuel?
Is non-consented video recording admissable evidence in a civil trial in Maryland?
Iteration counts of AMG solver changes in parallel

Two switch flyback converter MOSFETs voltage stress

@lorey
lorey / firefox-profile-with-automatic-download.py
Created April 1, 2017 10:05
Selenium: Prevent download dialog and download file automatically
# adapted from http://stackoverflow.com/a/25251803
profile = webdriver.FirefoxProfile()
profile.set_preference('browser.download.folderList', 2) # custom location
profile.set_preference('browser.download.manager.showWhenStarting', False)
profile.set_preference('browser.helperApps.neverAsk.saveToDisk', 'text/vcard') # type of file to download
# use the out folder of the script path
profile.set_preference('browser.download.dir', os.path.join(os.path.dirname(os.path.abspath(__file__)), 'out'))
@rtt
rtt / tinder-api-documentation.md
Last active May 5, 2024 15:28
Tinder API Documentation

Tinder API documentation

Note: this was written in April/May 2014 and the API may has definitely changed since. I have nothing to do with Tinder, nor its API, and I do not offer any support for anything you may build on top of this. Proceed with caution

http://rsty.org/

I've sniffed most of the Tinder API to see how it works. You can use this to create bots (etc) very trivially. Some example python bot code is here -> https://gist.github.com/rtt/5a2e0cfa638c938cca59 (horribly quick and dirty, you've been warned!)