Skip to content

Instantly share code, notes, and snippets.

View dimitryzub's full-sized avatar
🇺🇦
Grateful

Dmitiry Zub☀️ dimitryzub

🇺🇦
Grateful
View GitHub Profile
@dimitryzub
dimitryzub / open_source_licenses.md
Created January 21, 2023 09:16 — forked from nicolasdao/open_source_licenses.md
What you need to know to choose an open source license.
@dimitryzub
dimitryzub / researchgate-search-questions.py
Last active September 14, 2022 10:01
Scrape ResearchGate Search - All Questions
# Blog post: https://serpapi.com/blog/web-scraping-all-questions-from-researchgate-search-in-python/
from parsel import Selector
from playwright.sync_api import sync_playwright
import json
def scrape_researchgate_questions(query: str):
with sync_playwright() as p:
@dimitryzub
dimitryzub / scrape-google-finance-markets-page.py
Last active August 2, 2022 09:37
Script for scraping Google Finance Markets with Python
# tutorial blog post: https://serpapi.com/blog/scrape-google-finance-markets-in-python/
import requests
import json
import re
import argparse
from parsel import Selector
parser = argparse.ArgumentParser(prog="Google Finance Markets Options")
parser.add_argument('-i','--indexes', action="store_true")
@dimitryzub
dimitryzub / scrape-google-finance-main-page.py
Created July 28, 2022 10:38
Script for scraping Google Finance main page in Python 🐍
# original blog post: https://serpapi.com/blog/web-scraping-google-finance/
import requests, json, re
from parsel import Selector
def scrape_google_finance_main_page():
# https://docs.python-requests.org/en/master/user/quickstart/#custom-headers
# https://www.whatismybrowser.com/detect/what-is-my-user-agent
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.134 Safari/537.36"
@dimitryzub
dimitryzub / scrape-google-images-python-serpapi.py
Last active October 3, 2022 09:52
Scrape Google Images with Python and SerpApi web scraping library
def serpapi_get_google_images():
image_results = []
for query in ["Coffee", "boat", "skyrim", "minecraft"]:
# search query parameters
params = {
"engine": "google", # search engine. Google, Bing, Yahoo, Naver, Baidu...
"q": query, # search query
"tbm": "isch", # image results
@dimitryzub
dimitryzub / scrape-google-images-python.py
Last active April 28, 2024 11:34
Web scraping all Google Images in Python
# Step-by-step blog post: https://serpapi.com/blog/scrape-google-images-with-python/
# There's an API solution with a video tutorial: https://www.youtube.com/watch?v=QuCPV6_GT6o
import os, requests, lxml, re, json, urllib.request
from bs4 import BeautifulSoup
from serpapi import GoogleSearch
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36"
}
@dimitryzub
dimitryzub / webscraping-all-goolgle-news-articles.py
Created July 13, 2022 10:06
How to Web Scrape all Google News Articles with Python and SerpApi
# video tutorial: https://www.youtube.com/watch?v=fOs_eOsLP54
from serpapi import GoogleSearch
from urllib.parse import (parse_qsl, urlsplit)
params = {
"api_key": "...", # serpapi api key
"engine": "google", # search engine
"q": "minecraft", # search query
"gl": "us", # country of the search
@dimitryzub
dimitryzub / serpapi-google-sports-results-timezone-format.py
Created July 5, 2022 11:10
#AskSerpApi: "When using sport results API, what timezone do you guys use? The time and date responses are changing."
# AskSerpApi video: https://www.youtube.com/watch?v=c26LgucCqDo
from dateutil import parser
from serpapi import GoogleSearch
params = {
"api_key": "your serpapi api key",
"engine": "google",
"q": "liverpool",
"google_domain": "google.co.uk",
@dimitryzub
dimitryzub / scrape-youtube-videos-python-serpapi.py
Last active July 4, 2022 06:06
Web Scraping YouTube Popular Videos Results Data with Python and SerpApi
# video tutorial: https://www.youtube.com/watch?v=PTLpbBiz_sc
from serpapi import GoogleSearch
from urllib.parse import (parse_qsl, urlsplit)
import json
params = {
"api_key": "your api key", # your serpapi api key
"engine": "youtube", # serpapi parsing engine
"search_query": "blender foundation", # search query
@dimitryzub
dimitryzub / scrape-google-maps-place-python.py
Created June 22, 2022 06:48
Web Scraping Google Maps Place in Python | SerpApi
# video: https://www.youtube.com/watch?v=jqoNHXiGeZA
from serpapi import GoogleSearch
import json
params = {
"api_key": "your-serpapi-api-key", # Your SerpApi Api key
"engine": "google_maps", # SerpApi Parsing Engine
"google_domain": "google.com", # Google domain
"ll": "@47.6062572,-122.4086176,12z", # GPS Coordinates