Skip to content

Instantly share code, notes, and snippets.

@Chestermozhao

Chestermozhao/ptt_crawler_headers.py

Last active July 30, 2020 08:13
Show Gist options
  • Save Chestermozhao/7ed7e5b12c265e147a372f8c7f4c2361 to your computer and use it in GitHub Desktop.
Save Chestermozhao/7ed7e5b12c265e147a372f8c7f4c2361 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
ptt_crawler_headers.py
def get_hotlist_article_urls():
hotlist_url = "https://moptt.azurewebsites.net/api/v2/hotpost?b=Gossiping&b=Boy-Girl&b=Beauty&b=marvel&b=WomenTalk&b=movie&page=%257B%2522skip%2522%253A0%257D"
headers = {"cookie": "over18=1;", "Authorization": "cMIS1Icr95gnR2U19hxO2K7r6mYQ96vp"}
resp = requests.get(hotlist_url, headers=headers)
res_json = resp.json()["posts"]
article_urls = [r["url"] for r in res_json]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment