Skip to content

Instantly share code, notes, and snippets.

Avatar

Ed Summers edsu

View GitHub Profile
View naeb-page-counts.py
#!/usr/bin/env python3
import internetarchive
ia = internetarchive.get_session()
print("ids,pages")
for result in ia.search_items('collection:mediahistory creator:National Association of Educational Broadcasters'):
ia_id = result['identifier']
View jupyterhub_config.py
# Configuration file for jupyterhub.
import os
#------------------------------------------------------------------------------
# Configurable configuration
#------------------------------------------------------------------------------
#------------------------------------------------------------------------------
# LoggingConfigurable configuration
#------------------------------------------------------------------------------
View blacklivesmatter-counts.csv
start end hour_count
2021-06-17T23:16:56.000Z 2021-06-18T00:00:00.000Z 232
2021-06-18T00:00:00.000Z 2021-06-18T01:00:00.000Z 267
2021-06-18T01:00:00.000Z 2021-06-18T02:00:00.000Z 353
2021-06-18T02:00:00.000Z 2021-06-18T03:00:00.000Z 260
2021-06-18T03:00:00.000Z 2021-06-18T04:00:00.000Z 286
2021-06-18T04:00:00.000Z 2021-06-18T05:00:00.000Z 278
2021-06-18T05:00:00.000Z 2021-06-18T06:00:00.000Z 185
2021-06-18T06:00:00.000Z 2021-06-18T07:00:00.000Z 225
2021-06-18T07:00:00.000Z 2021-06-18T08:00:00.000Z 241
@edsu
edsu / replies.py
Last active May 27, 2021
Try to get replies to a particular set of tweets, recursively.
View replies.py
#!/usr/bin/env python
"""
Twitter's API doesn't allow you to get replies to a particular tweet. Strange
but true. But you can use Twitter's Search API to search for tweets that are
directed at a particular user, and then search through the results to see if
any are replies to a given tweet. You probably are also interested in the
replies to any replies as well, so the process is recursive. The big caveat
here is that the search API only returns results for the last 7 days. So
View episode-docs-shared-subjects.py
import json
docs = json.load(open('Projects/airwaves/static/data/documents.json'))
for e in json.load(open('Projects/airwaves/static/data/episodes.json')):
for s in e['subject']:
for d in docs:
related = None
for s2 in d['subject']:
if s['name'] == s2['name']:
View got-retweets.md

Got Retweets?

Even though Twitter's statuses/retweets API endpoint is limited to the last 100 retweets it is possible to use the search/tweets endpoint to search for the retweets using the text of the tweet. Caveat: This is only possible for tweets that have happened in the last 7 days, which is furthest back Twitter allow you to search for tweets in.

For example here is how you can get the retweets for this tweet and analyze the users in a spreadsheet.

View test_api.py
import requests
token = 'CHANGEME'
url = 'https://api.twitter.com/2/tweets/search/all?expansions=author_id%2Cin_reply_to_user_id%2Creferenced_tweets.id%2Creferenced_tweets.id.author_id%2Centities.mentions.username%2Cattachments.poll_ids%2Cattachments.media_keys%2Cgeo.place_id&user.fields=created_at%2Cdescription%2Centities%2Cid%2Clocation%2Cname%2Cpinned_tweet_id%2Cprofile_image_url%2Cprotected%2Cpublic_metrics%2Curl%2Cusername%2Cverified%2Cwithheld&tweet.fields=attachments%2Cauthor_id%2Ccontext_annotations%2Cconversation_id%2Ccreated_at%2Centities%2Cgeo%2Cid%2Cin_reply_to_user_id%2Clang%2Cpublic_metrics%2Ctext%2Cpossibly_sensitive%2Creferenced_tweets%2Creply_settings%2Csource%2Cwithheld&media.fields=duration_ms%2Cheight%2Cmedia_key%2Cpreview_image_url%2Ctype%2Curl%2Cwidth%2Cpublic_metrics&poll.fields=duration_minutes%2Cend_datetime%2Cid%2Coptions%2Cvoting_status&place.fields=contained_within%2Ccountry%2Ccountry_code%2Cfull_name%2Cgeo%2Cid%2Cname%2Cplace_type&max_results=10&query=%23endsars&start_time=2006-0
View videos-directory.txt
videos
├── archive.txt
├── cbsnews_-embed
│   ├── heT2fS5d_26SdE0llYY3eqtTsruVy7Tu
│   │   ├── New_police_shooting_amid_nights_of_unrest_near_St._Louis.description
│   │   ├── New_police_shooting_amid_nights_of_unrest_near_St._Louis.en.ttml
│   │   ├── New_police_shooting_amid_nights_of_unrest_near_St._Louis.info.json
│   │   └── New_police_shooting_amid_nights_of_unrest_near_St._Louis.mp4
│   └── N9L3R_azcDe2VOImjI6dYCHk1mDnsMtq
│   ├── Missouri_residents_doubt_cops_story_of_fatal_teen_shooting.description
@edsu
edsu / errors.py
Last active Apr 29, 2021
This script surfs errors from the sample stream Twitter API. These come from trying to get user and tweet expansions for users and tweets that have been suspended, deleted or protected.
View errors.py
#!/usr/bin/env python3
# This script surfs errors from the sample stream Twitter API
# these come from trying to get user and tweet expansions for users and tweets
# that have been suspended, deleted or protected
import os
import twarc
import dotenv
View results.csv
We can't make this file beautiful and searchable because it's too large.