-
-
Save muraiki/e53afa88fefa2b5b2c3d to your computer and use it in GitHub Desktop.
Process wikidata using perl6, take 2
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
use JSON::Fast; | |
sub MAIN() { | |
my $lines = open('wikidata-20150831-all.json', :r).lines(20).hyper(batch => 4, degree => 4); | |
my $filter_ends = $lines.grep: -> $line { $line !~~ / ^'[' || $']' / }; | |
my $remove_comma = $filter_ends.map: -> $line { $line.chop if $line.ends-with(',') }; | |
my $json = $remove_comma.map: &from-json; | |
my $ids = $json.map: *<id>; | |
say $ids.list; | |
} |
This is with 4 virtual cores at 2800.000 (1 thread per core), 16GB ram
No hyper:
real 0m21.884s
user 0m21.759s
sys 0m0.107s
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
batch 4 degree 4:
hyper with no args: