Skip to content

Instantly share code, notes, and snippets.

Last active Jan 14, 2020
What would you like to do?
How to generate a Twitter user's unique tweet URL's, and then feed them into ArchiveBot to be saved

Set up Tweep

  1. Download the file from GitHub -

  2. Put it in a brand new folder. Let's call the folder "Tweep". So the full path here would be, as an example, /Users/asparagirl/Desktop/Tweep

  3. Add a folder inside of that one called tmp. So the full path here would be, as an example, /Users/asparagirl/Desktop/Tweep/tmp

  4. Edit slightly to add logging and stop it from getting images from tweets. The top of the file should be edited to look like this:

#!/usr/bin/env python
import logging
from bs4 import BeautifulSoup
from time import gmtime, strftime
# from PIL import Image
from io import BytesIO
import argparse


  1. Install tweep's dependencies:
  • pip install beautifulsoup4
  • pip install requests
  • pip install lxml

(You may need to do sudo pip instead of just pip, depending on your system.)

Run Tweep to start collecting Tweets

  1. Run
python -u TWITTER-USERNAME-HERE > /Users/asparagirl/Desktop/Tweep/tmp/TWITTER-USERNAME-HERE
  1. You should see the screen filling with log data. The tweets will be stored in a text file called TWITTER-USERNAME-HERE inside the tmp sub-directory.

  2. Now we need to parse that text file and just get the full Twitter status URL's, and discard the actual text of the tweets. Run:

awk '{print "" substr($5, 2, length($5) - 2) "/status/" $1}' /Users/asparagirl/Desktop/Tweep/tmp/TWITTER-USERNAME-HERE | pbcopy -selection clipboard
  1. This will copy to your clipboard the list of tweets!

  2. Paste that data to pastebin or whatever you need.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment