Skip to content

Instantly share code, notes, and snippets.

@chrislkeller
Last active August 19, 2019 17:19
Show Gist options
  • Save chrislkeller/6abf35b0719dbfe333c071a63d7bd5c3 to your computer and use it in GitHub Desktop.
Save chrislkeller/6abf35b0719dbfe333c071a63d7bd5c3 to your computer and use it in GitHub Desktop.

NOTE: I'll work at fleshing this out a bit better

GOAL: Retrieve, catalog and display a daily record of voter registration figures in Orange County because a spokesperson for the registrar's office said they are unable to provide the figures.

TO PULL THIS OFF:

  • Determine when the figures are updated each day
    • HOW: Phone conversation with registrar, reporter
  • Navigate to the O.C. URL that contains the daily figures
    • Create a scheduler
    • Scrape those figures
      • HOW: Python, requests and BeautifulSoup

      • Save them to a .csv file stored on AWS S3

        • HOW: Tie my AWS Lambada function to S3
      • Send a message to a Slack channel with the figures

        def send_slack_message(webhook_url, data):
            slack_message = 'Updated Orange County Voter Registration\nas of {0}\n'.format(datetime.datetime.now())
            for item in data:
                slack_message += '\t{0}: {1}\n'.format(item['scope'], item['number'])
            post = {'text': '{0}'.format(slack_message)}
            json_data = json.dumps(post)
            try:
                req = urllib.request.Request(
                    webhook_url,
                    data=json_data.encode('ascii'),
                    headers={'Content-Type': 'application/json'}
                )
                resp = urllib.request.urlopen(req)
                logger.info(resp)
            except Exception as em:
                print('exception: ' + str(em))
        
  • Visualize the figures each day in Observable notebook
    • By connecting to the S3 .csv
      • HOW: Create Observable notebook and use the aws-sdk Javascript library
    • Charting the data
      • HOW: Use Vega-Lite
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment