Last active
November 23, 2015 16:52
-
-
Save ibo2go/8287f0d0fee118f63ff5 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
0.1832562: compiled and opened IO::Handle<out_sequential/out_o.log>(opened, at octet 0) | |
0.1998176: in a single thread | |
fp: shakespeare/1591-the_second_part_of_king_henry_the_sixth.book | |
3.0877887: 2.8873295 read shakespeare/1591-the_second_part_of_king_henry_the_sixth.book 4560(28002) entries; Σ 4560(28002) | |
3.1011595: read shakespeare/1591-the_second_part_of_king_henry_the_sixth.book 4560(28002) entries; Σ 4560(28002) it took 2.90090776sec | |
fp: shakespeare/1591-the_third_part_of_king_henry_the_sixth.book | |
6.1735347: 3.07037366 read shakespeare/1591-the_third_part_of_king_henry_the_sixth.book 3981(27155) entries; Σ 6320(55157) | |
6.18766135: read shakespeare/1591-the_third_part_of_king_henry_the_sixth.book 3981(27155) entries; Σ 6320(55157) it took 3.08463565sec | |
fp: shakespeare/1592-the_first_part_of_henry_the_sixth.book | |
9.2993940: 3.1100717 read shakespeare/1592-the_first_part_of_henry_the_sixth.book 4375(23914) entries; Σ 7964(79071) | |
9.316804: read shakespeare/1592-the_first_part_of_henry_the_sixth.book 4375(23914) entries; Σ 7964(79071) it took 3.1275845sec | |
fp: shakespeare/1593-king_richard_iii.book | |
13.39975347: 4.08061260 read shakespeare/1593-king_richard_iii.book 4578(32529) entries; Σ 9379(111600) | |
13.4206797: read shakespeare/1593-king_richard_iii.book 4578(32529) entries; Σ 9379(111600) it took 4.1015329sec | |
fp: shakespeare/1593-the_comedy_of_errors.book | |
16.00981524: 2.5877451 read shakespeare/1593-the_comedy_of_errors.book 2833(17145) entries; Σ 10100(128745) | |
16.0275803: read shakespeare/1593-the_comedy_of_errors.book 2833(17145) entries; Σ 10100(128745) it took 2.60544657sec | |
fp: shakespeare/1594-the_taming_of_the_shrew.book | |
19.3077522: 3.27876038 read shakespeare/1594-the_taming_of_the_shrew.book 3620(23244) entries; Σ 11113(151989) | |
19.3348947: read shakespeare/1594-the_taming_of_the_shrew.book 3620(23244) entries; Σ 11113(151989) it took 3.3057301sec | |
fp: shakespeare/1594-the_tragedy_of_titus_andronicus.book | |
22.6907185: 3.3538668 read shakespeare/1594-the_tragedy_of_titus_andronicus.book 3805(22650) entries; Σ 11941(174639) | |
22.7108966: read shakespeare/1594-the_tragedy_of_titus_andronicus.book 3805(22650) entries; Σ 11941(174639) it took 3.3741151sec | |
22.7127638 Stats generation: | |
23.3477712 Stats: | |
23.3940084: done |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
0.1848754: compiled and opened IO::Handle<out_withPromise/out_o.log>(opened, at octet 0) | |
0.2028829: with promises | |
0.20578486 made promise <shakespeare/1591-the_second_part_of_king_henry_the_sixth.book> | |
fp: shakespeare/1591-the_second_part_of_king_henry_the_sixth.book | |
0.20605498 made promise <shakespeare/1591-the_third_part_of_king_henry_the_sixth.book> | |
fp: shakespeare/1591-the_third_part_of_king_henry_the_sixth.book | |
0.2068458 made promise <shakespeare/1593-king_richard_iii.book> | |
fp: shakespeare/1593-king_richard_iii.book | |
0.2079190 made promise <shakespeare/1594-the_tragedy_of_titus_andronicus.book> | |
0.2077397 made promise <shakespeare/1594-the_taming_of_the_shrew.book> | |
fp: shakespeare/1594-the_tragedy_of_titus_andronicus.book | |
fp: shakespeare/1594-the_taming_of_the_shrew.book | |
0.206261880.20737814 made promise <shakespeare/1593-the_comedy_of_errors.book> | |
fp: shakespeare/1593-the_comedy_of_errors.book | |
made promise <shakespeare/1592-the_first_part_of_henry_the_sixth.book> | |
fp: shakespeare/1592-the_first_part_of_henry_the_sixth.book | |
20.71412091: held promise and read shakespeare/1591-the_second_part_of_king_henry_the_sixth.book 4560(28002) entries; Σ 4560(28002) it took 20.5274801sec | |
21.72606213: held promise and read shakespeare/1591-the_third_part_of_king_henry_the_sixth.book 3981(27155) entries; Σ 6320(55157) it took 21.5241056sec | |
23.15874457: held promise and read shakespeare/1592-the_first_part_of_henry_the_sixth.book 4375(23914) entries; Σ 7964(79071) it took 22.9713144sec | |
24.79771647: held promise and read shakespeare/1593-king_richard_iii.book 4578(32529) entries; Σ 9379(111600) it took 24.6171600sec | |
26.3205637: held promise and read shakespeare/1593-the_comedy_of_errors.book 2833(17145) entries; Σ 10100(128745) it took 26.12022201sec | |
28.0417359: held promise and read shakespeare/1594-the_taming_of_the_shrew.book 3620(23244) entries; Σ 11113(151989) it took 27.84465189sec | |
29.92410491: held promise and read shakespeare/1594-the_tragedy_of_titus_andronicus.book 3805(22650) entries; Σ 11941(174639) it took 29.7285703sec | |
29.9535147 Stats generation: | |
30.85693410 Stats: | |
30.91356470: done |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env perl6 | |
# Die Okt 27 12:58:06 CET 2015 | |
### | |
use v6; | |
my $tBegin = BEGIN { now }; | |
my $tInit = INIT { now }; | |
# SUBS | |
### | |
# bag-file | |
# -> file name, log file handle | |
# <- bag of words found | |
sub bag-file( $sFilename, $hFo ){ | |
$hFo.say: " fp: $sFilename"; | |
return ($sFilename, now, $sFilename.IO.slurp.comb(/\w+/).Bag); | |
} | |
### | |
# MAIN | |
# -> $outdir stats written to outdir | |
# $withPromise | |
# @asInfiles set of files to be processed | |
sub MAIN( :$outdir='out', Bool :$withPromise=False, *@asInfiles){ | |
my $fo-log = "$outdir/out_o.log".IO.open(:w); | |
$fo-log.say: $tInit - $tBegin, ": compiled and opened ", $fo-log; | |
my $bagDb = ().BagHash; | |
if $withPromise { | |
$fo-log.say: now - $tBegin, ": with promises"; | |
my @promises = @asInfiles.map( -> $sFile { | |
Promise.start({ | |
$fo-log.say: now - $tBegin, " made promise <$sFile>"; | |
bag-file( $sFile, $fo-log ); | |
}) | |
}); | |
for await @promises -> ($sFile, $starttime, $bag0) { | |
$bagDb ⊎= $bag0; | |
$fo-log.say: now - $tBegin, ": held promise and read $sFile {$bag0.elems}({$bag0.total}) entries; Σ {$bagDb.elems}({$bagDb.total}) it took ", now - $starttime, "sec"; | |
} | |
} else { | |
$fo-log.say: now - $tBegin, ": in a single thread"; | |
@asInfiles.map( -> $sFile { | |
my ($sFilename, $starttime, $bag0) = bag-file($sFile, $fo-log); | |
$bagDb ⊎= $bag0; | |
$fo-log.say: now - $tBegin, ": ", now - $starttime, " read $sFilename {$bag0.elems}({$bag0.total}) entries; Σ {$bagDb.elems}({$bagDb.total})"; | |
$fo-log.say: now - $tBegin, ": read $sFile {$bag0.elems}({$bag0.total}) entries; Σ {$bagDb.elems}({$bagDb.total}) it took ", now - $starttime, "sec"; | |
}); | |
} | |
$fo-log.say: now - $tBegin, " Stats generation:"; | |
my %hStat; | |
for |$bagDb.kv -> $k, $v { | |
push %hStat{$v}, $k; | |
} | |
$fo-log.say: now - $tBegin, " Stats:"; | |
my $fo = "$outdir/out.o.stat".IO.open(:w); | |
$fo.print("$_\n") for %hStat.pairs.sort(+*.key).reverse; | |
$fo-log.say: now -$tBegin, ": done\n"; | |
} | |
# vim: expandtab sw=4 ts=8 sts=4 ai |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We have shakespeare's complete publications as simple txt source gutenberg project in a subfolder.
The script collects the words in bags and prints the statistics to an outfile "$outdir/out.o.stat" and logs to "$outdir/out_o.log".
The parsing is done sequentially or --withPromise uses promises.
./Word_stats.pl6 --outdir=out_sequential shakespeare/159[1234]_book
./Word_stats.pl6 --withPromise --outdir=out_withPromiseshakespeare/159[1234]_boo
But somehow --withPromise is slower. Am I doing some thing wrong?