Jacob Kahn jacobkahn
jacobkahn
/ librispeech.dev-test.manual.excluded.titles.lst
Last active
January 14, 2020 23:58
List of LibriSpeech dev+test books found in Gutenberg with fuzzy matching procedure (word-level Levenshtein, score = 0.3, distancesim = 0.75)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| abraham lincoln a history volume 01|abraham lincoln a history volume 1|1 | |
| abraham lincoln a history volume 02|abraham lincoln a history volume 1|1 | |
| bunyan's pilgrim's progress in words of one syllable|pilgrim's progress in words of one syllable|1 | |
| bunyan's pilgrim's progress in words of one syllable|pilgrim's progress in words of one syllable|1 | |
| co aytch maury grays first tennessee regiment or a side show of the big show|'co aytch' maury grays first tennessee regiment or a side show of the big show|2 | |
| co aytch maury grays first tennessee regiment or a side show of the big show|'co aytch' maury grays first tennessee regiment or a side show of the big show|2 | |
| co aytch maury grays first tennessee regiment or a side show of the big show|'co aytch' maury grays first tennessee regiment or a side show of the big show|2 | |
| commentary on the epistle to the galatians|commentary on st. paul's epistle to the galatians|2 | |
| commentary on the epistle to the galatians|commentary on st. paul's epistle to the galatians|2 | |
| history of the d |
jacobkahn
/ levenshtein-word.pl
Created
January 12, 2020 21:43
Character Levenshtein with two files in Perl
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/perl | |
| use List::Util qw(min); | |
| sub levenshtein | |
| { | |
| my ($str1, $str2) = @_; | |
| my @ar1 = split ' ', $str1; | |
| my @ar2 = split ' ', $str2; | |
| my @dist; |
jacobkahn
/ gutenberg.librivox.manual.excluded.titles.lst
Last active
January 16, 2020 23:19
List of Libri-Light books found in Gutenberg with fuzzy matching procedure (word-level Levenshtein, score = 0.3, distancesim = 0.75)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 100 the story of a patriot|100 the story of a patriot|0 | |
| 1914 and other poems|1914 and other poems|0 | |
| 32 caliber|32 caliber|0 | |
| 35 sonnets|35 sonnets|0 | |
| 813|813|0 | |
| aaron's rod|aaron's rod|0 | |
| abaft the funnel|abaft the funnel|0 | |
| abandoned|abandoned|0 | |
| abbotsford and newstead abbey|abbotsford and newstead abbey|0 | |
| abe and mawruss being further adventures of potash and perlmutter|abe and mawruss being further adventures of potash and perlmutter|0 |
jacobkahn
/ ArrayFire.3.6.4.log
Last active
December 3, 2019 22:32
wav2letter Conv + GLU model with ArrayFire 3.6.4 and master (pre 3.7)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| epoch: 1 | lr: 0.800000 | lrcriterion: 0.008000 | runtime: 03:03:23 | bch(ms): 1251.95 | smp(ms): 0.64 | fwd(ms): 663.27 | crit-fwd(ms): 30.43 | bwd(ms): 520.30 | optim(ms): 45.35 | loss: 27.04214 | train-LER: 99.35 | train-WER: 99.79 | dev-clean.lst-loss: 19.76846 | dev-clean.lst-LER: 98.35 | dev-clean.lst-WER: 99.76 | dev-other.lst-loss: 18.46809 | dev-other.lst-LER: 98.13 | dev-other.lst-WER: 99.66 | avg-isz: 1228 | avg-tsz: 206 | max-tsz: 430 | hrs: 959.74 | thrpt(sec/sec): 314.00 | |
| epoch: 2 | lr: 0.800000 | lrcriterion: 0.008000 | runtime: 03:04:39 | bch(ms): 1260.59 | smp(ms): 0.62 | fwd(ms): 672.51 | crit-fwd(ms): 31.58 | bwd(ms): 519.22 | optim(ms): 45.44 | loss: 10.13135 | train-LER: 27.68 | train-WER: 58.20 | dev-clean.lst-loss: 2.91011 | dev-clean.lst-LER: 10.81 | dev-clean.lst-WER: 29.63 | dev-other.lst-loss: 5.84437 | dev-other.lst-LER: 23.03 | dev-other.lst-WER: 50.14 | avg-isz: 1228 | avg-tsz: 206 | max-tsz: 430 | hrs: 959.74 | thrpt(sec/sec): 311.85 | |
| epoch: 3 |
jacobkahn
/ gist:153a6b7d4f7b892cb568e3e7bc7a4135
Created
August 15, 2019 05:17
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Remove libiomp to avoid double linkage | |
| mkdir ~/misc && sudo mv /usr/local/lib/libiomp5.so ~/misc | |
| mkdir -p /data/usr | |
| ### Install libsndfile ### | |
| # Dependencies for libsndfile. The entries in Amazon Linux's RHEL | |
| # package repo are too old, so some of these need to be built from source. | |
| # This is likely not an issue on Ubuntu, and packages from apt-get can be used. | |
| # Build and install flac |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /home/ec2-user/anaconda3/bin/mpirun -x FI_PROVIDER="efa" -x FI_OFI_RXR_RX_COPY_UNEXP=1 -x FI_OFI_RXR_RX_COPY_OOO=1 -x FI_EFA_MR_CACHE_ENABLE=1 -x FI_OFI_RXR_INLINE_MR_ENABLE=1 -x LD_LIBRARY_PATH=/home/ec2-user/aws-ofi-nccl/install/lib/:/home/ec2-user/nccl/build/lib:/usr/local/cuda-10.0/lib64:/opt/amazon/efa/lib64:$LD_LIBRARY_PATH -x NCCL_DEBUG=INFO -x NCCL_TREE_THRESHOLD=0 -H 172.31.39.141:16,172.31.43.16:16 -n 16 -N 8 --mca btl tcp,self --mca btl_tcp_if_exclude lo,docker0 --bind-to none /home/ec2-user/nccl-tests/build/all_reduce_perf -b 8 -e 1G -f 2 -g 1 -c 1 -n 100 |
jacobkahn
/ gist:31fc0bedde5d2dd9d084d56b42dc3208
Created
July 24, 2019 06:42
NCCL Tests on 1 node p3dn.24xlarge + ethernet - all_reduce_perf -b 8 -e 1G -f 2 -g 1 -c 1 -n 100
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # | |
| # out-of-place in-place | |
| # size count type redop time algbw busbw error time algbw busbw error | |
| # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) | |
| ip-172-31-39-141:50926:50926 [0] NCCL INFO Launch mode Parallel | |
| 8 2 float sum 13.88 0.00 0.00 1e-07 13.97 0.00 0.00 1e-07 | |
| 16 4 float sum 14.03 0.00 0.00 1e-07 14.04 0.00 0.00 1e-07 | |
| 32 8 float sum 13.93 0.00 0.00 6e-08 13.86 0.00 0.00 6e-08 | |
| 64 16 float sum 14.11 0.00 0.01 6e-08 13.88 0.00 0.01 6e-08 | |
| 128 32 float sum 14.23 0.01 0.02 6e-08 14.11 0.01 0.02 6e-08 |
jacobkahn
/ gist:4e43bfccd79bf864970a98c191cd6c9b
Created
July 24, 2019 06:41
NCCL Tests on 1 node p3dn.24xlarge + efa - all_reduce_perf -b 8 -e 1G -f 2 -g 1 -c 1 -n 100
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # | |
| # out-of-place in-place | |
| # size count type redop time algbw busbw error time algbw busbw error | |
| # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) | |
| ip-172-31-39-141:50814:50814 [0] NCCL INFO Launch mode Parallel | |
| 8 2 float sum 13.84 0.00 0.00 1e-07 13.75 0.00 0.00 1e-07 | |
| 16 4 float sum 13.82 0.00 0.00 1e-07 13.96 0.00 0.00 1e-07 | |
| 32 8 float sum 13.79 0.00 0.00 6e-08 13.79 0.00 0.00 6e-08 | |
| 64 16 float sum 13.88 0.00 0.01 6e-08 13.73 0.00 0.01 6e-08 | |
| 128 32 float sum 14.04 0.01 0.02 6e-08 13.94 0.01 0.02 6e-08 |
jacobkahn
/ gist:71a0c1c7ae2f7260471d5bb5da829a67
Last active
July 24, 2019 07:02
NCCL Tests on 32 node p3dn.24xlarge + ethernet - all_reduce_perf -b 8 -e 1G -f 2 -g 1 -c 1 -n 100
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # | |
| # out-of-place in-place | |
| # size count type redop time algbw busbw error time algbw busbw error | |
| # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) | |
| 8 2 float sum 1790.6 0.00 0.00 1e-06 1787.9 0.00 0.00 1e-06 | |
| 16 4 float sum 1844.8 0.00 0.00 5e-07 1798.8 0.00 0.00 5e-07 | |
| 32 8 float sum 1789.6 0.00 0.00 1e-06 1795.4 0.00 0.00 1e-06 | |
| 64 16 float sum 1799.5 0.00 0.00 1e-06 1801.5 0.00 0.00 1e-06 | |
| 128 32 float sum 1816.3 0.00 0.00 1e-06 1814.9 0.00 0.00 1e-06 | |
| 256 64 float sum 1826.6 0.00 0.00 1e-06 1883.2 0.00 0.00 1e-06 |