Skip to content

Instantly share code, notes, and snippets.

@RooneyMcNibNug
RooneyMcNibNug / SA_public_thread_scrape.py
Last active December 6, 2023 16:57
For when you want to scrape a publicly available thread (including images) on forums.somethingawful.com
### Python script using bs4 to scrape a Something Awful forums thread. ###
#
# ! THIS WILL ONLY WORK ON THREADS THAT ARE PUBLICLY ACCESSIBLE, IE ONE'S YOU DON'T NEED TO LOG IN TO SEE !
# (I didn't want to deal with auth stuff..)
#
# Make sure you replace the thread_url variable with the link you want to scrape, same as the example in the code here.
#
# This will dump to an HTML file with decent formatting (a bit too wall-of-text atm, but I have too many other things to tend to in life).
#
# If you want to have better file naming including something like the ThreadID, do this:
### Keybase proof
I hereby claim:
* I am rooneymcnibnug on github.
* I am atomsk (https://keybase.io/atomsk) on keybase.
* I have a public key ASDG4cphU7n4RO1YDd5nA6G78_Ah1X-hZ7WjGA9o353G6Qo
To claim this, I am signing this object: