Skip to content

Instantly share code, notes, and snippets.

@ibo2go
Last active November 23, 2015 16:52
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ibo2go/8287f0d0fee118f63ff5 to your computer and use it in GitHub Desktop.
Save ibo2go/8287f0d0fee118f63ff5 to your computer and use it in GitHub Desktop.
0.1832562: compiled and opened IO::Handle<out_sequential/out_o.log>(opened, at octet 0)
0.1998176: in a single thread
fp: shakespeare/1591-the_second_part_of_king_henry_the_sixth.book
3.0877887: 2.8873295 read shakespeare/1591-the_second_part_of_king_henry_the_sixth.book 4560(28002) entries; Σ 4560(28002)
3.1011595: read shakespeare/1591-the_second_part_of_king_henry_the_sixth.book 4560(28002) entries; Σ 4560(28002) it took 2.90090776sec
fp: shakespeare/1591-the_third_part_of_king_henry_the_sixth.book
6.1735347: 3.07037366 read shakespeare/1591-the_third_part_of_king_henry_the_sixth.book 3981(27155) entries; Σ 6320(55157)
6.18766135: read shakespeare/1591-the_third_part_of_king_henry_the_sixth.book 3981(27155) entries; Σ 6320(55157) it took 3.08463565sec
fp: shakespeare/1592-the_first_part_of_henry_the_sixth.book
9.2993940: 3.1100717 read shakespeare/1592-the_first_part_of_henry_the_sixth.book 4375(23914) entries; Σ 7964(79071)
9.316804: read shakespeare/1592-the_first_part_of_henry_the_sixth.book 4375(23914) entries; Σ 7964(79071) it took 3.1275845sec
fp: shakespeare/1593-king_richard_iii.book
13.39975347: 4.08061260 read shakespeare/1593-king_richard_iii.book 4578(32529) entries; Σ 9379(111600)
13.4206797: read shakespeare/1593-king_richard_iii.book 4578(32529) entries; Σ 9379(111600) it took 4.1015329sec
fp: shakespeare/1593-the_comedy_of_errors.book
16.00981524: 2.5877451 read shakespeare/1593-the_comedy_of_errors.book 2833(17145) entries; Σ 10100(128745)
16.0275803: read shakespeare/1593-the_comedy_of_errors.book 2833(17145) entries; Σ 10100(128745) it took 2.60544657sec
fp: shakespeare/1594-the_taming_of_the_shrew.book
19.3077522: 3.27876038 read shakespeare/1594-the_taming_of_the_shrew.book 3620(23244) entries; Σ 11113(151989)
19.3348947: read shakespeare/1594-the_taming_of_the_shrew.book 3620(23244) entries; Σ 11113(151989) it took 3.3057301sec
fp: shakespeare/1594-the_tragedy_of_titus_andronicus.book
22.6907185: 3.3538668 read shakespeare/1594-the_tragedy_of_titus_andronicus.book 3805(22650) entries; Σ 11941(174639)
22.7108966: read shakespeare/1594-the_tragedy_of_titus_andronicus.book 3805(22650) entries; Σ 11941(174639) it took 3.3741151sec
22.7127638 Stats generation:
23.3477712 Stats:
23.3940084: done
0.1848754: compiled and opened IO::Handle<out_withPromise/out_o.log>(opened, at octet 0)
0.2028829: with promises
0.20578486 made promise <shakespeare/1591-the_second_part_of_king_henry_the_sixth.book>
fp: shakespeare/1591-the_second_part_of_king_henry_the_sixth.book
0.20605498 made promise <shakespeare/1591-the_third_part_of_king_henry_the_sixth.book>
fp: shakespeare/1591-the_third_part_of_king_henry_the_sixth.book
0.2068458 made promise <shakespeare/1593-king_richard_iii.book>
fp: shakespeare/1593-king_richard_iii.book
0.2079190 made promise <shakespeare/1594-the_tragedy_of_titus_andronicus.book>
0.2077397 made promise <shakespeare/1594-the_taming_of_the_shrew.book>
fp: shakespeare/1594-the_tragedy_of_titus_andronicus.book
fp: shakespeare/1594-the_taming_of_the_shrew.book
0.206261880.20737814 made promise <shakespeare/1593-the_comedy_of_errors.book>
fp: shakespeare/1593-the_comedy_of_errors.book
made promise <shakespeare/1592-the_first_part_of_henry_the_sixth.book>
fp: shakespeare/1592-the_first_part_of_henry_the_sixth.book
20.71412091: held promise and read shakespeare/1591-the_second_part_of_king_henry_the_sixth.book 4560(28002) entries; Σ 4560(28002) it took 20.5274801sec
21.72606213: held promise and read shakespeare/1591-the_third_part_of_king_henry_the_sixth.book 3981(27155) entries; Σ 6320(55157) it took 21.5241056sec
23.15874457: held promise and read shakespeare/1592-the_first_part_of_henry_the_sixth.book 4375(23914) entries; Σ 7964(79071) it took 22.9713144sec
24.79771647: held promise and read shakespeare/1593-king_richard_iii.book 4578(32529) entries; Σ 9379(111600) it took 24.6171600sec
26.3205637: held promise and read shakespeare/1593-the_comedy_of_errors.book 2833(17145) entries; Σ 10100(128745) it took 26.12022201sec
28.0417359: held promise and read shakespeare/1594-the_taming_of_the_shrew.book 3620(23244) entries; Σ 11113(151989) it took 27.84465189sec
29.92410491: held promise and read shakespeare/1594-the_tragedy_of_titus_andronicus.book 3805(22650) entries; Σ 11941(174639) it took 29.7285703sec
29.9535147 Stats generation:
30.85693410 Stats:
30.91356470: done
#!/usr/bin/env perl6
# Die Okt 27 12:58:06 CET 2015
###
use v6;
my $tBegin = BEGIN { now };
my $tInit = INIT { now };
# SUBS
###
# bag-file
# -> file name, log file handle
# <- bag of words found
sub bag-file( $sFilename, $hFo ){
$hFo.say: " fp: $sFilename";
return ($sFilename, now, $sFilename.IO.slurp.comb(/\w+/).Bag);
}
###
# MAIN
# -> $outdir stats written to outdir
# $withPromise
# @asInfiles set of files to be processed
sub MAIN( :$outdir='out', Bool :$withPromise=False, *@asInfiles){
my $fo-log = "$outdir/out_o.log".IO.open(:w);
$fo-log.say: $tInit - $tBegin, ": compiled and opened ", $fo-log;
my $bagDb = ().BagHash;
if $withPromise {
$fo-log.say: now - $tBegin, ": with promises";
my @promises = @asInfiles.map( -> $sFile {
Promise.start({
$fo-log.say: now - $tBegin, " made promise <$sFile>";
bag-file( $sFile, $fo-log );
})
});
for await @promises -> ($sFile, $starttime, $bag0) {
$bagDb ⊎= $bag0;
$fo-log.say: now - $tBegin, ": held promise and read $sFile {$bag0.elems}({$bag0.total}) entries; Σ {$bagDb.elems}({$bagDb.total}) it took ", now - $starttime, "sec";
}
} else {
$fo-log.say: now - $tBegin, ": in a single thread";
@asInfiles.map( -> $sFile {
my ($sFilename, $starttime, $bag0) = bag-file($sFile, $fo-log);
$bagDb ⊎= $bag0;
$fo-log.say: now - $tBegin, ": ", now - $starttime, " read $sFilename {$bag0.elems}({$bag0.total}) entries; Σ {$bagDb.elems}({$bagDb.total})";
$fo-log.say: now - $tBegin, ": read $sFile {$bag0.elems}({$bag0.total}) entries; Σ {$bagDb.elems}({$bagDb.total}) it took ", now - $starttime, "sec";
});
}
$fo-log.say: now - $tBegin, " Stats generation:";
my %hStat;
for |$bagDb.kv -> $k, $v {
push %hStat{$v}, $k;
}
$fo-log.say: now - $tBegin, " Stats:";
my $fo = "$outdir/out.o.stat".IO.open(:w);
$fo.print("$_\n") for %hStat.pairs.sort(+*.key).reverse;
$fo-log.say: now -$tBegin, ": done\n";
}
# vim: expandtab sw=4 ts=8 sts=4 ai
@ibo2go
Copy link
Author

ibo2go commented Nov 23, 2015

We have shakespeare's complete publications as simple txt source gutenberg project in a subfolder.
The script collects the words in bags and prints the statistics to an outfile "$outdir/out.o.stat" and logs to "$outdir/out_o.log".
The parsing is done sequentially or --withPromise uses promises.

./Word_stats.pl6 --outdir=out_sequential shakespeare/159[1234]_book
./Word_stats.pl6 --withPromise --outdir=out_withPromiseshakespeare/159[1234]_boo

But somehow --withPromise is slower. Am I doing some thing wrong?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment