How I archive twitter accounts and hashtags and how you can too

how2twitter.avi

Hello!
======

  Table of Contents;
    The Fast Way
    The Thorough Way
    Wait, how do I read this file?

Archiving Twitter accounts and hashtags is easy. Here's how!

       /####\
       |STOP| Twitter is usually very lenient on bans. However, it's still possible to get banned,
       |SIGN|especially as snscrape (mis)uses the public API used by the web-app. I take no responsibility for this.
       \####/
       
       #######
       | NOTE | If you get an error about _make_card missing a positional argument or similar, don't fret.
        |      |The maintainer of snscrape is working on a fix: https://github.com/JustAnotherArchivist/snscrape/issues/407
         |_____|It might be a good idea to temporarily use the "release" version rather than the development one.
                (However, this will retrieve less data. It might be better to wait for a fix.)
       #######
       | NOTE | Most of this guide assumes a sane UNIX environment with Python 3 installed.
        |      |If you don't have that, this guide won't be helpful.
         |_____|On Windows you might use WSL, Cygwin, or Git Bash. WSL is recommended. Or just, y'know, stop using Windows.

              ##### #####           #####           #
              #   # #   #           #  #            #
              ##### #####   =====   # #             #
              ##### #####   =====   # #   ####   ####
              #   # #   #           #  #  #  ##  #  #
              ##### #####           ##### #### # ####

Number One: The Fast Way
----------------------------
The easiest way is of course to fire up [snscrape](https://github.com/JustAnotherArchivist/snscrape) with the --jsonl parameter.

    * The JSONL option is so that instead of only displaying the URL of each post, it'll output the data

Of course you'll want to pipe snscrape's output to a file.
Example for most POSIX-compliant shells (AKA anything but Windows):

    snscrape --jsonl --progress twitter-user Ukraine >> tweets.jsonl
Or:
    snscrape --jsonl --progress twitter-hashtag BeautifulUkraine >> tweets.jsonl

The `>>' tells your shell to append to the file (creating it if it doesn't exist). You could also use `>' to overwrite the file.

        /^\
       / | \ Don't do this to the same file in multiple different shells/terminals! It will likely corrupt the file.
      /__.__\
    
Number Two: The Thorough Way
----------------------------
This is my *preferred* way to do it. This way scrapes a posts replies, its parent replies, its parent replies' replies, its
replies' replies, etc. In other words, it's recursive.

    * This does not traverse hashtags/@mentions. You'll have to do that yourself if you want it, but you'd probably end up scraping
      the whole website by doing that :P

This is a multi-step process.

First. You need to scrape the post IDs for your search term. 
Example for most POSIX-compliant shells (AKA anything but Windows):

    snscrape -f "{id}" --progress twitter-user Ukraine >> twitterids
Or:
    snscrape -f "{id} --progress twitter-hashtag BeautifulUkraine >> twitterids

!!! We interrupt this broadcast to announce that we're currently on Line 69 of the file. Nice.
Aaaaaanyway....

This will take about the same time as the old method, as it's exactly the same as the old method; the only difference is the
only thing being saved is the post ID. The second step is what will take a while. Fortunately, the second step can resume
crawls, so long as the "Archive File" (the `finished.json' file) is intact. This Archive File keeps track of what tweets it's
already downloaded, so it can skip them. Ain't that neat?

Clone the https://github.com/archivist-rs/batch-get-tweets repo and enter it (don't forget what folder you saved twitterids in!)
For simplicity's sake, I'll store the path in a variable. Assuming a sane Unix system, run:

```
export ARCHIVIST_TWEETS_SCRIPT=$(find `pwd`/get_tweets.py)
```
Now go back to the folder where `twitterids' is. Next run:

```
python3 -m pip install alive_progress
python3 $ARCHIVIST_TWEETS_SCRIPT
```

You should now see a progress bar on your screen. This process is going to take a *very long time*. It's not unheard of for it to
take overnight, and it may even take multiple days. With multiple days comes multiple crashes, which is why there's the resume
functionality. It's a great way to find bugs in snscrape :-)
Now, this may also be very quick. It all depends on two factors:

  1. and most importantly:
      The amount of Twitter IDs.
  2. and similarily importantly:
      The amount of *replies* to those Twitter IDs (and their replies, etc).
  3. Your Internet connection.

...And, now you're done!

Wait, so how do I read this file?
---------------------------------
Reading tweets.jsonl isn't straightforward. However, you've got pretty much all the tweet data there, it's just in a very machine-
readable format.
I'm working on a viewer; progress is at https://github.com/archivist-rs/archivist. Still not great though, and I could use some
help, especially by someone that can actually design something. I'm _decent_, but I'm still utterly shit.

Thanks for reading this guide! It's one of my longest... writings ever. I hope it was helpful to you, and that you do great things
with it.
  Ciao!