Create a gist now

Instantly share code, notes, and snippets.

Search Google NGrams for total occurrences of pairs of words, in this case CRC32 collisions in dictionary files. Reads in a CSV of pairs of words, one pair per line. Run the shell script in a directory that has the g-zipped 1grams in it, it will slice out just the needed lines out of the ngrams. Then the perl script will total them up.
gunzip -c googlebooks-eng-all-1gram-*-[a-z].gz | pv | grep -P `cat reddit_words.csv | perl -e "while(<>) { s/\'s//g; chomp; push @words, split ','; } printf '^(%s)\\t', join '|', @words;"` > collision_stats.tsv
my %table;
while(<CS>) {
($word, $year, $total, $books) = split;
$word =~ s/\'s$//;
$table{$word}{$year} = [$total, $books];
$table{$word}{'total'}[0] += $total;
$table{$word}{'total'}[1] += $books;
while(<RW>) {
@words = split;
@totaltotals = (0,0);
@texttots = ();
foreach $word (split ',') {
$word =~ s/\'s$//;
@totals = @{$table{$word}{'total'}};
$totaltotals[0] += $totals[0];
$totaltotals[1] += $totals[1];
push(@texttots, sprintf('%s (%d)',$word,$totals[0]));
printf "%d %s\n", $totaltotals[0], join(',',@texttots);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment