Skip to content

Instantly share code, notes, and snippets.

View opendidi's full-sized avatar
🐴
An Ordinary JavaScript Developer

oh opendidi

🐴
An Ordinary JavaScript Developer
View GitHub Profile
@opendidi
opendidi / spiders.py
Created November 13, 2024 06:45
crawl website content
bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
def ensure_dir(file_path):
"""确保文件路径中的目录存在"""
directory = os.path.dirname(file_path)
if not os.path.exists(directory):
os.makedirs(directory)
def download_resource(resource_url, base_folder):