Scraping Resources

1_scraping_python_modules.md

Python Modules for Scraping:

Scraping and Parsing

• selectolax
• AdvancedHTMLParser
• grequests
• parsel
• mechanicalsoup
• beautifulsoup4
• gazpacho
• cloudscraper
• cfscrape
• ipwhois
• saas
• parse-utils
• looter
• xlseries
• sriram-twitter-scraper

Scrapy

• scrapy
• scrapyrt
• scrapy-splash
• scrapy-autoextract
• scrapy-pagestorage
• scrapy-jsonschema
• scrapy-wayback-middleware
• scrapy-rss
• scrapy-rotating-proxies
• django-dynamic-scraper

Specific

• yt-videos-list
• twint
• play-scraper
• instagramscraper
• instalooter
• instabotnet
• linkedin-scraper
• google-search-results-serpwow
• youtubedata
• TikTokApi
• imgur-scraper
• tropescraper
• google-search-results
• pastepwn
• wikitablescrape
• recipe-scrapers
• name-scraper
• lyrics-extractor
• newsman
• ludoj-scraper

JSON

• python-rapidjson
• orjson
• jsonslicer
• nujson
• yapic.json

Text & Data Manipulation

• htmldate
• newspaper3k
• acora
• hext
• boltons (boltons.strutils)
• w3lib
• textnormaliser
• hyperlink
• shorttext
• postal
• readability
• cypunct
• justext
• iso4217parse
• isbnlib

Image, Audio & File Manipulation

• tesserocr
• imagecodecs
• imagecodecs-lite
• miniaudio
• pysndfile
• pdfquery

Performance

• cnamedtuples
• pybase64
• lz4 and zstd
• pikepdf and PyMuPDF
• fortuna, pyewacket, and rng
• cytoolz
• psutil
• libuuid
• hoedowm

PS: Not going to include the obvious ones like requests, pandas and numpy

2_scraping_links.md

Links:

• Making Web Crawlers Using Scrapy for Python
• A Minimalist End-to-End Scrapy Tutorial, Part 1
• A Minimalist End-to-End Scrapy Tutorial, Part 2
• A Minimalist End-to-End Scrapy Tutorial, Part 3
• A Minimalist End-to-End Scrapy Tutorial, Part 4
• Scrapy Tutorial Series: Web Scraping Using Python
• Crawling with Scrapy – Javascript Generated Content
• Scraping with Scrapy and Django Integration
• Managing proxies and Captcha with scrapy and the Scraper API