-
-
Save Altai-man/bb7a57736ea272553edc2dbbfc61af45 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
use v6.c; | |
use Pod::Load; | |
sub recursive-dir($dir) is export { | |
my @todo = $dir; | |
gather while @todo { | |
my $d = @todo.shift; | |
for dir($d) -> $f { | |
if $f.f { | |
take $f; | |
} | |
else { | |
@todo.append: $f.path; | |
} | |
} | |
} | |
} | |
sub read-dir($dir) { | |
recursive-dir("doc/$dir/") | |
.grep({.path ~~ / '.pod6' $/}) | |
.map({ | |
.path.subst("doc/$dir/", '') | |
.subst(rx{\.pod6$}, '') | |
.subst(:g, '/', '::') | |
=> $_ | |
}); | |
} | |
sub MAIN { | |
my @files = [read-dir($_).Slip for <Type Programs Native Language>][0..100]; | |
# secuencial | |
my $start = now; | |
for @files { | |
load($_.value); | |
} | |
my $end = now; | |
my $elapsed = ($end - $start); | |
say "Secuencial => $elapsed"; | |
# parallel | |
$start = now; | |
my @ts; | |
for ^10 { | |
@ts.push: start { | |
loop { | |
last if @files.elems == 0; | |
my $file = @files.shift; | |
load($file.value); | |
CATCH { | |
default { | |
.note; | |
} | |
} | |
} | |
} | |
} | |
await Promise.allof(@ts); | |
$end = now; | |
$elapsed = ($end - $start ); | |
say "Parallel => $elapsed"; | |
} |
Even if Pod::Load works, which it does, there's still the problem of the low-level filesystem routines, and out-of-Rakudo access to filesystem itself. It's probably never a (very) good idea to parallelize that, because it's going to be sequential no matter what. Even if you eliminate a bit of overhead, it's going to come back at you with a vengeance when you go to the OS to do stuff.
Baseline is: disk drives do not have many cores.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Yes, exactly, you'd need to have a lock around @files. I did not, because I was in a great hurry with this example and to show the approach, this race with @files is not going to hurt a lot while we had an issue with
load
originally. It goes without saying that if you'll go with this code (which I cannot recommend), it would need a proper locking around @files usage.It technicaly works, but still sometimes throw Decoder may not... exception, which is caught, and that is a very bad sign. I just showed an another approach compared to
race.map
that was proposed initially, it still encounters the error, but is likely to "deceive" rakudo that the usage is not concurrent, which it is not, so please, do not use this, this is simply dangerous and will hurt someone's head in the future. My bet in this situation would be to:1)Are you really sure that
load
is the most trouble and it has to be parallelized at any price? Maybe there are other gains to find?2)If yes, then you'd need to talk this over with moarvm dev who is qualified enough to suggest what can and can't be done. Maybe Decoder can be made thread-safe with a couple of locks, maybe there might be some alternative, etc.