bulk-master.log

[Decoder]: Using assembly version of decoder
I1211 16:05:46.512570   24659 init.go:98] 

Dgraph version   : v1.1.1-rc1-5-g6c7d1c14
Dgraph SHA-256   : 41e8dae99b0d1f9418f638a13580706769a63e303960dfd9c6e9b324cf74d9c4
Commit SHA-1     : 6c7d1c14
Commit timestamp : 2019-12-09 19:52:04 +0530
Branch           : master
Go version       : go1.13.1

For Dgraph official documentation, visit https://docs.dgraph.io.
For discussions about Dgraph     , visit https://discuss.dgraph.io.
To say hi to the community       , visit https://dgraph.slack.com.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2018 Dgraph Labs, Inc.


{
	"DataFiles": "ownthink.rdf",
	"DataFormat": "",
	"SchemaFile": "empty.schema",
	"OutDir": "./out",
	"ReplaceOutDir": false,
	"TmpDir": "tmp",
	"BadgerKeyFile": "",
	"NumGoroutines": 3,
	"MapBufSize": 67108864,
	"SkipMapPhase": false,
	"CleanupTmp": true,
	"NumReducers": 1,
	"Version": false,
	"StoreXids": false,
	"ZeroAddr": "localhost:5080",
	"HttpAddr": "localhost:8000",
	"IgnoreErrors": false,
	"CustomTokenizers": "",
	"NewUids": false,
	"MapShards": 8,
	"ReduceShards": 2
}
Connecting to zero at localhost:5080
Processing file (1 out of 1): ownthink.rdf
[16:05:47+0530] MAP 01s nquad_count:405.0k err_count:0.000 nquad_speed:404.8k/sec edge_count:405.0k edge_speed:404.8k/sec
[16:05:48+0530] MAP 02s nquad_count:933.1k err_count:0.000 nquad_speed:466.4k/sec edge_count:933.1k edge_speed:466.4k/sec
[16:05:49+0530] MAP 03s nquad_count:1.513M err_count:0.000 nquad_speed:504.1k/sec edge_count:1.513M edge_speed:504.1k/sec
[16:05:50+0530] MAP 04s nquad_count:1.890M err_count:0.000 nquad_speed:472.4k/sec edge_count:1.890M edge_speed:472.4k/sec
[16:05:51+0530] MAP 05s nquad_count:2.000M err_count:0.000 nquad_speed:399.9k/sec edge_count:2.000M edge_speed:399.9k/sec
[16:05:52+0530] MAP 06s nquad_count:2.000M err_count:0.000 nquad_speed:333.3k/sec edge_count:2.000M edge_speed:333.3k/sec
Shard tmp/map_output/000 -> Reduce tmp/shards/shard_0/000
Shard tmp/map_output/001 -> Reduce tmp/shards/shard_1/001
Shard tmp/map_output/002 -> Reduce tmp/shards/shard_0/002
Shard tmp/map_output/003 -> Reduce tmp/shards/shard_0/003
Shard tmp/map_output/006 -> Reduce tmp/shards/shard_1/006
Shard tmp/map_output/004 -> Reduce tmp/shards/shard_0/004
Shard tmp/map_output/007 -> Reduce tmp/shards/shard_1/007
Shard tmp/map_output/005 -> Reduce tmp/shards/shard_0/005
[16:05:53+0530] REDUCE 07s 2.50% edge_count:50.00k edge_speed:50.00k/sec plist_count:50.00k plist_speed:50.00k/sec
[16:05:54+0530] REDUCE 08s 5.00% edge_count:100.0k edge_speed:99.98k/sec plist_count:100.0k plist_speed:99.98k/sec
[16:05:55+0530] REDUCE 09s 7.50% edge_count:150.0k edge_speed:74.98k/sec plist_count:150.0k plist_speed:74.98k/sec
[16:05:56+0530] REDUCE 10s 7.50% edge_count:150.0k edge_speed:49.99k/sec plist_count:150.0k plist_speed:49.99k/sec
[16:05:57+0530] REDUCE 11s 10.00% edge_count:200.0k edge_speed:49.99k/sec plist_count:200.0k plist_speed:49.99k/sec
[16:05:58+0530] REDUCE 12s 12.50% edge_count:250.0k edge_speed:49.99k/sec plist_count:250.0k plist_speed:49.99k/sec
[16:05:59+0530] REDUCE 13s 15.00% edge_count:300.0k edge_speed:49.99k/sec plist_count:300.0k plist_speed:49.99k/sec
[16:06:00+0530] REDUCE 14s 17.50% edge_count:350.0k edge_speed:49.99k/sec plist_count:350.0k plist_speed:49.99k/sec
[16:06:01+0530] REDUCE 15s 17.50% edge_count:350.0k edge_speed:43.74k/sec plist_count:350.0k plist_speed:43.74k/sec
[16:06:02+0530] REDUCE 16s 20.00% edge_count:400.0k edge_speed:44.44k/sec plist_count:400.0k plist_speed:44.44k/sec
[16:06:03+0530] REDUCE 17s 22.50% edge_count:450.0k edge_speed:44.99k/sec plist_count:450.0k plist_speed:44.99k/sec
[16:06:04+0530] REDUCE 18s 25.00% edge_count:500.0k edge_speed:45.44k/sec plist_count:500.0k plist_speed:45.44k/sec
[16:06:05+0530] REDUCE 19s 25.00% edge_count:500.0k edge_speed:41.66k/sec plist_count:500.0k plist_speed:41.66k/sec
[16:06:06+0530] REDUCE 20s 27.50% edge_count:550.0k edge_speed:42.30k/sec plist_count:550.0k plist_speed:42.30k/sec
[16:06:07+0530] REDUCE 21s 30.00% edge_count:600.0k edge_speed:42.85k/sec plist_count:600.0k plist_speed:42.85k/sec
[16:06:08+0530] REDUCE 22s 31.27% edge_count:625.3k edge_speed:41.68k/sec plist_count:625.3k plist_speed:41.68k/sec
[16:06:09+0530] REDUCE 23s 33.77% edge_count:675.3k edge_speed:42.20k/sec plist_count:675.3k plist_speed:42.20k/sec
[16:06:10+0530] REDUCE 24s 36.27% edge_count:725.3k edge_speed:42.66k/sec plist_count:725.3k plist_speed:42.66k/sec
[16:06:11+0530] REDUCE 25s 36.27% edge_count:725.3k edge_speed:40.29k/sec plist_count:725.3k plist_speed:40.29k/sec
[16:06:12+0530] REDUCE 26s 38.77% edge_count:775.3k edge_speed:40.80k/sec plist_count:775.3k plist_speed:40.80k/sec
[16:06:13+0530] REDUCE 27s 41.27% edge_count:825.3k edge_speed:41.26k/sec plist_count:825.3k plist_speed:41.26k/sec
[16:06:14+0530] REDUCE 28s 43.77% edge_count:875.3k edge_speed:41.67k/sec plist_count:875.3k plist_speed:41.67k/sec
[16:06:15+0530] REDUCE 29s 43.77% edge_count:875.3k edge_speed:39.78k/sec plist_count:875.3k plist_speed:39.78k/sec
[16:06:16+0530] REDUCE 30s 56.66% edge_count:1.133M edge_speed:49.26k/sec plist_count:1.133M plist_speed:49.26k/sec
[16:06:17+0530] REDUCE 31s 74.27% edge_count:1.485M edge_speed:61.88k/sec plist_count:1.485M plist_speed:61.88k/sec
[16:06:18+0530] REDUCE 32s 83.27% edge_count:1.665M edge_speed:66.60k/sec plist_count:1.665M plist_speed:66.60k/sec
[16:06:19+0530] REDUCE 33s 92.27% edge_count:1.845M edge_speed:70.96k/sec plist_count:1.845M plist_speed:70.96k/sec
[16:06:20+0530] REDUCE 34s 96.27% edge_count:1.925M edge_speed:71.30k/sec plist_count:1.925M plist_speed:71.30k/sec
[16:06:21+0530] REDUCE 35s 96.27% edge_count:1.925M edge_speed:68.75k/sec plist_count:1.925M plist_speed:68.75k/sec
[16:06:22+0530] REDUCE 36s 96.27% edge_count:1.925M edge_speed:66.38k/sec plist_count:1.925M plist_speed:66.38k/sec
[16:06:23+0530] REDUCE 37s 98.77% edge_count:1.975M edge_speed:65.83k/sec plist_count:1.975M plist_speed:65.83k/sec
[16:06:24+0530] REDUCE 38s 100.00% edge_count:2.000M edge_speed:64.50k/sec plist_count:2.000M plist_speed:64.50k/sec
[16:06:25+0530] REDUCE 38s 100.00% edge_count:2.000M edge_speed:63.12k/sec plist_count:2.000M plist_speed:63.12k/sec
Total: 38s

bulk-v1.1.0.log

[Decoder]: Using assembly version of decoder
I1211 18:02:24.778903   12948 init.go:98] 

Dgraph version   : v1.1.0
Dgraph SHA-256   : 81e8f579d2f38f493a97fc99c7760da7b8965c8497a5f9b4016d20faee460527
Commit SHA-1     : ef7cdb28
Commit timestamp : 2019-09-04 00:12:51 -0700
Branch           : HEAD
Go version       : go1.13.1

For Dgraph official documentation, visit https://docs.dgraph.io.
For discussions about Dgraph     , visit https://discuss.dgraph.io.
To say hi to the community       , visit https://dgraph.slack.com.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2018 Dgraph Labs, Inc.


{
	"DataFiles": "ownthink.rdf",
	"DataFormat": "",
	"SchemaFile": "empty.schema",
	"OutDir": "./out",
	"ReplaceOutDir": false,
	"TmpDir": "tmp",
	"NumGoroutines": 3,
	"MapBufSize": 67108864,
	"SkipMapPhase": false,
	"CleanupTmp": true,
	"NumReducers": 1,
	"Version": false,
	"StoreXids": false,
	"ZeroAddr": "localhost:5080",
	"HttpAddr": "localhost:8000",
	"IgnoreErrors": false,
	"CustomTokenizers": "",
	"NewUids": false,
	"MapShards": 8,
	"ReduceShards": 2
}
Connecting to zero at localhost:5080
Processing file (1 out of 1): ownthink.rdf
[18:02:25+0530] MAP 01s nquad_count:475.8k err_count:0.000 nquad_speed:471.3k/sec edge_count:475.8k edge_speed:471.3k/sec
[18:02:26+0530] MAP 02s nquad_count:1.070M err_count:0.000 nquad_speed:532.6k/sec edge_count:1.070M edge_speed:532.6k/sec
[18:02:27+0530] MAP 03s nquad_count:1.682M err_count:0.000 nquad_speed:558.7k/sec edge_count:1.682M edge_speed:558.7k/sec
[18:02:28+0530] MAP 04s nquad_count:2.000M err_count:0.000 nquad_speed:498.8k/sec edge_count:2.000M edge_speed:498.8k/sec
[18:02:29+0530] MAP 05s nquad_count:2.000M err_count:0.000 nquad_speed:399.2k/sec edge_count:2.000M edge_speed:399.2k/sec
Shard tmp/shards/001 -> Reduce tmp/shards/shard_1/001
Shard tmp/shards/005 -> Reduce tmp/shards/shard_0/005
Shard tmp/shards/000 -> Reduce tmp/shards/shard_0/000
Shard tmp/shards/006 -> Reduce tmp/shards/shard_0/006
Shard tmp/shards/003 -> Reduce tmp/shards/shard_1/003
Shard tmp/shards/004 -> Reduce tmp/shards/shard_0/004
Shard tmp/shards/002 -> Reduce tmp/shards/shard_1/002
Shard tmp/shards/007 -> Reduce tmp/shards/shard_0/007
[18:02:30+0530] REDUCE 06s 2.50% edge_count:50.00k edge_speed:50.00k/sec plist_count:50.00k plist_speed:50.00k/sec
[18:02:31+0530] REDUCE 07s 31.26% edge_count:625.1k edge_speed:625.0k/sec plist_count:625.1k plist_speed:625.0k/sec
[18:02:32+0530] REDUCE 08s 31.26% edge_count:625.1k edge_speed:312.5k/sec plist_count:625.1k plist_speed:312.5k/sec
[18:02:33+0530] REDUCE 09s 31.26% edge_count:625.1k edge_speed:208.3k/sec plist_count:625.1k plist_speed:208.3k/sec
GC: 24. InUse: 9.4 GB. Idle: 134 MB
[18:02:34+0530] REDUCE 10s 31.26% edge_count:625.1k edge_speed:156.3k/sec plist_count:625.1k plist_speed:156.3k/sec
[18:02:35+0530] REDUCE 11s 31.26% edge_count:625.1k edge_speed:125.0k/sec plist_count:625.1k plist_speed:125.0k/sec
[18:02:36+0530] REDUCE 12s 31.26% edge_count:625.1k edge_speed:104.2k/sec plist_count:625.1k plist_speed:104.2k/sec
[18:02:37+0530] REDUCE 13s 31.26% edge_count:625.1k edge_speed:89.29k/sec plist_count:625.1k plist_speed:89.29k/sec
[18:02:38+0530] REDUCE 14s 31.26% edge_count:625.1k edge_speed:78.13k/sec plist_count:625.1k plist_speed:78.13k/sec
[18:02:39+0530] REDUCE 15s 31.26% edge_count:625.1k edge_speed:69.45k/sec plist_count:625.1k plist_speed:69.45k/sec
[18:02:40+0530] REDUCE 16s 31.26% edge_count:625.1k edge_speed:62.50k/sec plist_count:625.1k plist_speed:62.50k/sec
[18:02:41+0530] REDUCE 17s 31.26% edge_count:625.1k edge_speed:56.82k/sec plist_count:625.1k plist_speed:56.82k/sec
[18:02:42+0530] REDUCE 18s 31.26% edge_count:625.1k edge_speed:52.09k/sec plist_count:625.1k plist_speed:52.09k/sec
[18:02:43+0530] REDUCE 19s 31.26% edge_count:625.1k edge_speed:48.08k/sec plist_count:625.1k plist_speed:48.08k/sec
GC: 25. InUse: 11 GB. Idle: 1.2 GB
[18:02:44+0530] REDUCE 20s 31.26% edge_count:625.1k edge_speed:44.65k/sec plist_count:625.1k plist_speed:44.65k/sec
[18:02:45+0530] REDUCE 21s 31.26% edge_count:625.1k edge_speed:41.67k/sec plist_count:625.1k plist_speed:41.67k/sec
[18:02:46+0530] REDUCE 22s 46.63% edge_count:932.6k edge_speed:58.28k/sec plist_count:932.6k plist_speed:58.28k/sec
[18:02:47+0530] REDUCE 23s 98.76% edge_count:1.975M edge_speed:116.2k/sec plist_count:1.975M plist_speed:116.2k/sec
[18:02:48+0530] REDUCE 24s 100.00% edge_count:2.000M edge_speed:111.1k/sec plist_count:2.000M plist_speed:111.1k/sec
[18:02:49+0530] REDUCE 25s 100.00% edge_count:2.000M edge_speed:105.2k/sec plist_count:2.000M plist_speed:105.2k/sec
[18:02:50+0530] REDUCE 26s 100.00% edge_count:2.000M edge_speed:99.69k/sec plist_count:2.000M plist_speed:99.69k/sec
[18:02:51+0530] REDUCE 27s 100.00% edge_count:2.000M edge_speed:94.95k/sec plist_count:2.000M plist_speed:94.95k/sec
[18:02:52+0530] REDUCE 28s 100.00% edge_count:2.000M edge_speed:90.65k/sec plist_count:2.000M plist_speed:90.65k/sec
[18:02:53+0530] REDUCE 29s 100.00% edge_count:2.000M edge_speed:86.72k/sec plist_count:2.000M plist_speed:86.72k/sec
GC: 27. InUse: 8.1 GB. Idle: 1.3 GB
[18:02:54+0530] REDUCE 30s 100.00% edge_count:2.000M edge_speed:83.11k/sec plist_count:2.000M plist_speed:83.11k/sec
[18:02:55+0530] REDUCE 31s 100.00% edge_count:2.000M edge_speed:79.80k/sec plist_count:2.000M plist_speed:79.80k/sec
[18:02:56+0530] REDUCE 32s 100.00% edge_count:2.000M edge_speed:76.74k/sec plist_count:2.000M plist_speed:76.74k/sec
[18:02:57+0530] REDUCE 32s 100.00% edge_count:2.000M edge_speed:75.74k/sec plist_count:2.000M plist_speed:75.74k/sec
Total: 32s

instructions.md

Generate an RDF :

python knowledgegraph.py &> ownthink.rdf

Run Zero:

dgraph zero

Run Bulk Loader:

dgraph bulk -f ownthink.rdf -s empty.schema --map_shards=8 --reduce_shards=2 --http localhost:8000 --zero=localhost:5080 

Here are the logs:

• master branch: here
• v1.1.0 branch: here

knowledgegraph.py

import csv
import string
import sys
import random

chars = "安吧八爸百北不大岛的弟地东都对多儿二方港哥个关贵国过海好很会家见叫姐京九可老李零六吗妈么没美妹们名明哪那南你您朋七起千去人认日三上谁什生师十识是四他她台天湾万王我五西息系先香想小谢姓休学也一亿英友月再张这中字"
limit = 999999

def id_generator(size=6, chars=string.ascii_uppercase + string.digits):
  return ''.join(random.choice(chars) for _ in range(size))
 
labels = ["生活", "科学", "文化", "科学的", "性质"]

for index, label in enumerate(labels):
  print('_:标签{} <名称> "{}" .'.format(index, label))

for i in range(limit):
  blank_label_node = id_generator(size=5, chars=chars)
  print('_:{}{} <{}> "{}" .'.format(blank_label_node, i, id_generator(size=3, chars=chars), id_generator(size=8, chars=chars)))
  print('_:{}{} <标签> _:标签{} .'.format(blank_label_node, i, i%5))