Skip to content

Instantly share code, notes, and snippets.

Dan Nguyen dannguyen

Block or report user

Report or block dannguyen

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@dannguyen
dannguyen / aws-textract-sample-readme.md
Last active May 30, 2019
A gist of AWS Textract sample/demo data for easy reference and preview, in case you're curious how well Amazon does when it comes to pdf-to-csv
View aws-textract-sample-readme.md

AWS Textract -- sample document image and data from the offical demo

AWS Textract is now out of closed beta. You can read the features page here, and you can also read about its limits here (e.g. no handwriting). Basically, if you've ever had to deal with the hell of getting structured data out of a PDF (scanned image or not), Textract is aiming for your business:

image

This short gist contains some of my brief observations about Textract and its demo, as well as direct links to the most relevant and important files, such as the Textract demo sample image and the resulting data files from Textract's API. If you have an AWS account, I h

@dannguyen
dannguyen / aws-textract-demo-readme.md
Last active May 30, 2019
Amazon Textract, i.e. AWS's OCR-as-a-cloud-service, was just released to the public. Here's how well it did with recognizing data tables in a particularly difficult PDF
View aws-textract-demo-readme.md

[Ignore this gist, checkout the github] Testing AWS Textract's ability to correctly extract data tables from a difficult FBI stats report PDF

Update: I've since realized that this writeup would be far easier to do as its own Github repo, given the number of files involved. Please ignore this gist which I'm keeping here as a backup, and instead, visit: https://github.com/dannguyen/aws-textract-pdf-to-csv-demo

tl;dr: pretty good table structure overall, given the issues with the original PDF. However, there were inexplicable and critical data errors, as if Textract converted the PDF to an image, OCRed it, and then attempted to extract the data tables.

Amazon Textract was announced about 6 months ago but was made public today (May 29). If have an AWS account, you can check out Textract's point-and-click demo, which allows you to upload an image or PDF for T

View sample-public-tweets.csv
We can make this file beautiful and searchable if this error is corrected: Unclosed quoted field in line 5.
ID,Posted at,Screen name,Text
1123212586919419905,2019-04-30 13:08:43 +0000,shinya1720777,"妙にテンションの高いまーちゃんにあおられた
キビナゴりせ
#今日のりせ活 https://t.co/qabC3PQi7m"
1123212591109672961,2019-04-30 13:08:44 +0000,inesteiixeira,RT @lunaaaaa20: acho que a coisa mais linda é acompanhar o crescimento da pessoa que gostamos e contribuir para isso
1123212591109689348,2019-04-30 13:08:44 +0000,BTS20520283,#BBMAsTopSocial BTS @BTS_twt kalp
1123212591139045381,2019-04-30 13:08:44 +0000,Dipendr80247123,"RT @YL511: #البتكوين
📌اش الحكاية :
-
@dannguyen
dannguyen / cms-medicare-bulk-downloading.md
Created Apr 14, 2019
Compiling all the Medicare payment data
View cms-medicare-bulk-downloading.md
@dannguyen
dannguyen / census-bulk-downloading-scripts.md
Created Apr 14, 2019
Bulk Census data downloading script
View census-bulk-downloading-scripts.md
@dannguyen
dannguyen / _house-public-disc-scraper-README.md
Last active Apr 8, 2019
Simple scraper of the ASPX search form for U.S. Congress House financial disclosure results
View _house-public-disc-scraper-README.md

Simple scraper of the ASPX search form for U.S. Congress House financial disclosure results

The following script, given someone's last name, prints a CSV of financial disclosure PDFs (the first 20, for simplicity's sake) as found on the House Financial Disclosure Reports. It's meant to be a proof-of-concept of how to scrape ASPX (and other "stateful" websites) with using plain old requests -- without too much inconvenience -- rather than resorting to something heavy like the selenium websdriver

The search page can be found here: http://clerk.house.gov/public_disc/financial-search.aspx

Here's a screenshot of what it does when you search via web browser:

screenshot of disclosure search for "king"

@dannguyen
dannguyen / google-speech-2-text-README.md
Last active Feb 19, 2019
How Google's text-to-speech API performs when reading the New York Times
View google-speech-2-text-README.md

Demo of Google text-to-speech Wavenet API on a NYT article

Was curious if Google's text-to-speech API might be good enough for generating audio versions of stories on-the-fly. Google has offered traditional computer voices for awhile, but last year made available their premium WaveNet voices, which are trained using audio recorded from human speakers, and are purportedly capable of mimicking natural-sounding inflection and rhythm.

tl;dr results

Pretty good...but I honestly can't tell the difference between the standard voice and the WaveNet version, at least when it comes to intonation and inflection. The first 2 grafs of this NYT story, roughly 85 words/560 characters, took less than 2 seconds to process. The result in both cases is a 37-second second audio file.

  • The M
@dannguyen
dannguyen / cardib-politics-talk-transcribe.md
Last active Feb 15, 2019
An example of how to use command-line tools to transcribe a viral video of Cardi B
View cardib-politics-talk-transcribe.md

Transcribing Cardi B's political speech with AWS Transcribe and command-line tools

Inspired by the following exchange on Twitter, in which someone captures and posts a valuable video onto Twitter, but doesn't have the resources to easily transcribe it for the hearing-impaired, I thought it'd be fun to try out Amazon's AWS Transcribe service to help with this problem, and to see if I could do it all from the bash command-line like a Unix dork.

Screencap of @jordanuhl's video tweet, followed by a request for a transcript

The instructions and code below show how to use command-line tools/scripting and Amazon's Transcribe service to transcribe the audio from online video. tl;dr: AWS Transcribe is a pretty amaz

@dannguyen
dannguyen / README.md
Last active Nov 7, 2018
Sample SQLite problem in which, given a table of records, I want to link each record with its prior version. Seems like recursion is the solution -- putting down this correlated query for ow
View README.md

Using recursive CTE in SQLite instead of a correlated query

Given a table of exams, in which each exam belongs to a "student", we want to derive a new table in which for any given student exam, we see how much the score has changed compared to the student's previous exam.

Here's the table of exams, which is generated by the SQL in sql-make-exams-table.sql](#file-sql-make-exams-table-sql):

starting exams table

| exam_id | year | student | score |

@dannguyen
dannguyen / _README-twitter-purge.md
Last active Jul 13, 2018
List of my follower characteristics, sorted by # of followers -- as of 2018-07-11 that were not my followers on 2018-07-13
View _README-twitter-purge.md

My Twitter followers who were "purged" between July 11 and July 13. 2018

tl;dr: The file below, purged-users.csv, contains a data table of stats from my Twitter followers who appear to have been "purged" on 2018-07-12 from my followers count. As Twitter said in its announcement, only follower counts have been adjusted. The actual user accounts who constitute the purged followers counts were not deleted, nor does their list of "followings" reflect the change. This makes sense because the "purged" accounts aren't necessarily fake, but they've been "locked" for suspicious behavior and have been inactive since.

Background

Sometime on July 12, 2018, Twitter conducted a mass purge of user accounts suspected to be fake. Via the Twitter blog, Confidence in follower counts (emphasis added):

You can’t perform that action at this time.