dannguyen

My Twitter followers who were "purged" between July 11 and July 13. 2018

tl;dr: The file below, purged-users.csv, contains a data table of stats from my Twitter followers who appear to have been "purged" on 2018-07-12 from my followers count. As Twitter said in its announcement, only follower counts have been adjusted. The actual user accounts who constitute the purged followers counts were not deleted, nor does their list of "followings" reflect the change. This makes sense because the "purged" accounts aren't necessarily fake, but they've been "locked" for suspicious behavior and have been inactive since.

Background

Sometime on July 12, 2018, Twitter conducted a mass purge of user accounts suspected to be fake. Via the Twitter blog, Confidence in follower counts (emphasis added):

dannguyen

Pure SQLite solution to creating a tidy/normalized data table from a column of delimited values

The problem: we have a data table in which one of the columns contains a text string that is meant to be multiple values separated by a delimiter (e.g. a comma). For example, the LAPD crime incidents data has a column named MO Codes (short for modus operandi). Every incident may have several MO's -- for example, a particular RESISTING ARREST incident may have a MO Codes value of 1212 0416, which corresponds, respectively, to: LA Police Officer and Hit-Hit w/ weapon:

![image](https://user-ima

dannguyen

"""
Fetches House disbursement CSV files from
    https://projects.propublica.org/represent/expenditures
Saves them to:
    data/raw/{year}Q{q}.csv
"""
import requests
from pathlib import Path

DATADIR = Path('data', 'raw')

dannguyen

Installing and using schemacrawler for MacOS

A recipe for generating cool SQLite database diagrams with schemacrawler on MacOS

This was tested on MacOS 10.14.5 on 2019-07-16

schemacrawler is a free and open-source database schema discovery and comprehension tool. It can be invoked from the command-line to produce, using GraphViz, images/pdfs from a SQLite (or other database type) file. It can be used from the command-line to generate schema diagrams like these:

dannguyen

uid,reported_date,victim_last,victim_first,victim_race,victim_age,victim_sex,city,state,lat,lon,disposition
Alb-000001,20100504,GARCIA,JUAN,Hispanic,78,Male,Albuquerque,NM,35.0957885,-106.5385549,Closed without arrest
Alb-000002,20100216,MONTOYA,CAMERON,Hispanic,17,Male,Albuquerque,NM,35.0568104,-106.715321,Closed by arrest
Alb-000003,20100601,SATTERFIELD,VIVIANA,White,15,Female,Albuquerque,NM,35.086092,-106.695568,Closed without arrest
Alb-000004,20100101,MENDIOLA,CARLOS,Hispanic,32,Male,Albuquerque,NM,35.0784929,-106.5560938,Closed by arrest
Alb-000005,20100102,MULA,VIVIAN,White,72,Female,Albuquerque,NM,35.1303568,-106.5809862,Closed without arrest
Alb-000006,20100126,BOOK,GERALDINE,White,91,Female,Albuquerque,NM,35.15111,-106.537797,Open/No arrest
Alb-000007,20100127,MALDONADO,DAVID,Hispanic,52,Male,Albuquerque,NM,35.1117847,-106.7126144,Closed by arrest
Alb-000008,20100127,MALDONADO,CONNIE,Hispanic,52,Female,Albuquerque,NM,35.1117847,-106.7126144,Closed by arrest
Alb-000009,20100130,MARTIN-LEYVA,GUSTAVO,W

dannguyen

{
  "kind": "youtube#videoListResponse",
  "etag": "\"DuHzAJ-eQIiCIp7p4ldoVcVAOeY/oWEizCC8EIhZ_7DOM2iudlxoTvI\"",
  "pageInfo": {
    "totalResults": 3,
    "resultsPerPage": 3
  },
  "items": [
    {
      "kind": "youtube#video",

dannguyen

Context

Facebook has been under scrutiny because of how its ad platform may have been used by foreign actors during the 2016 election. In May 2018, Facebook released ad data to a House committee, which subsequently published the data online.

As part of that continuing effort to educate the public and seek additional analysis, the Committee Minority is making available all IRA advertisements identified by Facebook. This is an effort to be fully transparent with the public, allow outside experts to analyze the data, and provide the American people a fuller accounting of Russian efforts to sow discord and interfere in our democracy.

You can read more about the events here:

https://democrats-intelligence.house.gov/facebook-ads/

dannguyen

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
from time import sleep


URLS = {
    '2017': 'https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/disengagement_report_2017',
    '2016': 'https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/disengagement_report_2016',
    '2015': 'https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/disengagement_report_2015',

dannguyen

dannguyen

{
    "status": "OK",
    "copyright": "Copyright (c) 2017 Pro Publica Inc. All Rights Reserved.",
    "results": {
        "chamber": "House",
        "offset": 0,
        "num_results": 20,
        "votes": [{
                "congress": 115,
                "chamber": "House",