Skip to content

Instantly share code, notes, and snippets.

@BruJu

BruJu/aExperiment.md

Last active July 8, 2020 10:04
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save BruJu/919c1d3c10dfed21553e4a6e9f910b2a to your computer and use it in GitHub Desktop.
Save BruJu/919c1d3c10dfed21553e4a6e9f910b2a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
aExperiment.md

SPARQL End point benchmark, episode 3

Former experiments

This one

In this gist, we benchmark Wasm Tree Store on SPARQL request using the Comunica engine. We compare it to the default parameters of Comunica, which resorts to a n3 store.

We use the BSBM benchmark to measure the performances on several select sparql requests.

  • BSBM: Download BSBM driver at https://sourceforge.net/projects/bsbmtools/files/latest/download

  • BSBM: Unpack it in ./ so its content is found in ./bsbmtools-0.2

  • Comunica: run the comunica.sh script (provided as the second file of this gist) which clone comunica, builds it for the baseline benchmark and builds another comunica clone which is a slight modification (made by substituing n3 store usage by wasm store)

  • Build the dataset: cd bsbm_tools-0.2

  • Build the dataset: java -cp .:lib/bsbm.jar:lib/jdom.jar:lib/log4j-1.2.12.jar:lib/ssj.jar -Xmx256M benchmark.generator.Generator -fc -pc 1000 -s ttl

  • Build the test suite: echo 9 > queries/explore/ignoreQueries.txt (For some reason, the 9th request of BSBM is very slow)

  • Stop builind : cd ..

Run a sparql end point and benchmark

  • Either cd comunica-untouched or cd comunica-wasm
  • cd comunica/packages/actor-init-sparql-file/
  • NODE_ENV=production node bin/http.js "{ \"sources\": [\"../../../../bsbmtools-0.2/dataset.ttl\"]}"

In another terminal

  • cd bsbmtools-0.2/
  • java -cp .:lib/bsbm.jar:lib/jdom.jar:lib/log4j-1.2.12.jar:lib/ssj.jar -Xmx256M benchmark.testdriver.TestDriver http://localhost:3000/sparql -w 1 -runs 5

Results

  • n3 : ~20 sec per query mix

  • wasm tree : ~5 sec per query mix

  • Oxi point : min/max Querymix runtime: 0.0759s / 0.0936s

(see results files for n3 and wasm tree)

Raw
comunica.sh
#!/bin/bash
set -e
# Get Wasm Tree Store
wget https://bruy.at/wasm_tree.zip
unzip wasm_tree.zip -d wasm_tree/
cd wasm_tree
npm install @types/rdf-js
npm install @graphy/core.data.factory
npm install @graphy/content.ttl.read
cd ..
# Get comunica
git clone https://github.com/comunica/comunica.git
rm comunica-untouched -rf
rm comunica-wasm -rf
mkdir comunica-untouched
cp comunica comunica-untouched/ -r
mkdir comunica-wasm
cp comunica comunica-wasm/ -r
# Modify Comunica-wasm
cd comunica-wasm/comunica/
yarn install
npm run-script build
# Substites every storeStream call to calls to wasm tree storeStream's function
find . -type f -exec sed -i 's/.*const rdf_store_stream_1 = require(\"rdf-store-stream\");.*/const rdf_store_stream_1 = require("..\/..\/..\/..\/..\/wasm_tree\/wrappedtree")/' {} \;
cd ../../
# Build Comunica
cd comunica-untouched/comunica/
yarn install
npm run-script build
cd ../../
Raw
results_n3.txt
-1: 0.00ms, total: 24853ms
0: 23120.49ms, total: 23145ms
1: 21973.43ms, total: 21988ms
2: 19815.62ms, total: 19829ms
3: 20314.33ms, total: 20325ms
4: 23558.83ms, total: 23569ms
Scale factor: 1000
Number of warmup runs: 1
Seed: 808080
Number of query mix runs (without warmups): 5 times
min/max Querymix runtime: 19.8156s / 23.5588s
Total runtime: 108.783 seconds
QMpH: 165.47 query mixes per hour
CQET: 21.75654 seconds average runtime of query mix
CQET (geom.): 21.70560 seconds geometric mean runtime of query mix
Metrics for Query: 1
Count: 5 times executed in whole run
AQET: 0.067887 seconds (arithmetic mean)
AQET(geom.): 0.065879 seconds (geometric mean)
QPS: 14.73 Queries per second
minQET/maxQET: 0.03926194s / 0.07816802s
Average result count: 1.00
min/max result count: 0 / 2
Number of timeouts: 0
Metrics for Query: 2
Count: 30 times executed in whole run
AQET: 0.639208 seconds (arithmetic mean)
AQET(geom.): 0.635785 seconds (geometric mean)
QPS: 1.56 Queries per second
minQET/maxQET: 0.58724101s / 0.81900891s
Average result count: 21.20
min/max result count: 14 / 31
Number of timeouts: 0
Metrics for Query: 3
Count: 5 times executed in whole run
AQET: 0.156196 seconds (arithmetic mean)
AQET(geom.): 0.155096 seconds (geometric mean)
QPS: 6.40 Queries per second
minQET/maxQET: 0.12148070s / 0.16799405s
Average result count: 0.60
min/max result count: 0 / 1
Number of timeouts: 0
Metrics for Query: 4
Count: 5 times executed in whole run
AQET: 0.193751 seconds (arithmetic mean)
AQET(geom.): 0.188189 seconds (geometric mean)
QPS: 5.16 Queries per second
minQET/maxQET: 0.11521924s / 0.22776323s
Average result count: 0.00
min/max result count: 0 / 0
Number of timeouts: 0
Metrics for Query: 5
Count: 10 times executed in whole run
AQET: 2.621023 seconds (arithmetic mean)
AQET(geom.): 2.550287 seconds (geometric mean)
QPS: 0.38 Queries per second
minQET/maxQET: 1.98907923s / 4.58271191s
Average result count: 3.70
min/max result count: 1 / 5
Number of timeouts: 0
Metrics for Query: 7
Count: 20 times executed in whole run
AQET: 2.427925 seconds (arithmetic mean)
AQET(geom.): 2.336032 seconds (geometric mean)
QPS: 0.41 Queries per second
minQET/maxQET: 1.27689330s / 3.12808200s
Average result count: 9.80
min/max result count: 2 / 19
Number of timeouts: 0
Metrics for Query: 8
Count: 10 times executed in whole run
AQET: 0.595510 seconds (arithmetic mean)
AQET(geom.): 0.567969 seconds (geometric mean)
QPS: 1.68 Queries per second
minQET/maxQET: 0.32332906s / 0.83625255s
Average result count: 4.80
min/max result count: 1 / 15
Number of timeouts: 0
Metrics for Query: 9
Count: 0 times executed in whole run
AQET: 0.000000 seconds (arithmetic mean)
AQET(geom.): NaN seconds (geometric mean)
QPS: Infinity Queries per second
minQET/maxQET: 179769313486231570000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.00000000s / 0.00000000s
Average result (Bytes): 0.00
min/max result (Bytes): 2147483647 / -2147483648
Number of timeouts: 0
Metrics for Query: 10
Count: 10 times executed in whole run
AQET: 0.351963 seconds (arithmetic mean)
AQET(geom.): 0.343880 seconds (geometric mean)
QPS: 2.84 Queries per second
minQET/maxQET: 0.28504375s / 0.59144901s
Average result count: 2.60
min/max result count: 0 / 6
Number of timeouts: 0
Metrics for Query: 11
Count: 5 times executed in whole run
AQET: 0.059720 seconds (arithmetic mean)
AQET(geom.): 0.059688 seconds (geometric mean)
QPS: 16.74 Queries per second
minQET/maxQET: 0.05840785s / 0.06368808s
Average result count: 10.00
min/max result count: 10 / 10
Number of timeouts: 0
Metrics for Query: 12
Count: 5 times executed in whole run
AQET: 0.595043 seconds (arithmetic mean)
AQET(geom.): 0.592373 seconds (geometric mean)
QPS: 1.68 Queries per second
minQET/maxQET: 0.55265133s / 0.70729634s
Average result (Bytes): 169.80
min/max result (Bytes): 168 / 171
Number of timeouts: 0
Raw
results_wasmtree.txt
-1: 0.00ms, total: 4394ms
0: 3929.77ms, total: 3952ms
1: 4339.85ms, total: 4354ms
2: 4255.03ms, total: 4287ms
3: 4863.95ms, total: 4881ms
4: 6146.48ms, total: 6158ms
Scale factor: 1000
Number of warmup runs: 1
Seed: 808080
Number of query mix runs (without warmups): 5 times
min/max Querymix runtime: 3.9298s / 6.1465s
Total runtime: 23.535 seconds
QMpH: 764.82 query mixes per hour
CQET: 4.70702 seconds average runtime of query mix
CQET (geom.): 4.64807 seconds geometric mean runtime of query mix
Metrics for Query: 1
Count: 5 times executed in whole run
AQET: 0.015959 seconds (arithmetic mean)
AQET(geom.): 0.014282 seconds (geometric mean)
QPS: 62.66 Queries per second
minQET/maxQET: 0.00694979s / 0.02844427s
Average result count: 1.00
min/max result count: 0 / 2
Number of timeouts: 0
Metrics for Query: 2
Count: 30 times executed in whole run
AQET: 0.127522 seconds (arithmetic mean)
AQET(geom.): 0.123283 seconds (geometric mean)
QPS: 7.84 Queries per second
minQET/maxQET: 0.08767903s / 0.26771191s
Average result count: 21.20
min/max result count: 14 / 31
Number of timeouts: 0
Metrics for Query: 3
Count: 5 times executed in whole run
AQET: 0.057135 seconds (arithmetic mean)
AQET(geom.): 0.056983 seconds (geometric mean)
QPS: 17.50 Queries per second
minQET/maxQET: 0.05018101s / 0.06245950s
Average result count: 0.60
min/max result count: 0 / 1
Number of timeouts: 0
Metrics for Query: 4
Count: 5 times executed in whole run
AQET: 0.059546 seconds (arithmetic mean)
AQET(geom.): 0.059367 seconds (geometric mean)
QPS: 16.79 Queries per second
minQET/maxQET: 0.05078176s / 0.06279754s
Average result count: 0.00
min/max result count: 0 / 0
Number of timeouts: 0
Metrics for Query: 5
Count: 10 times executed in whole run
AQET: 0.606133 seconds (arithmetic mean)
AQET(geom.): 0.541017 seconds (geometric mean)
QPS: 1.65 Queries per second
minQET/maxQET: 0.26759003s / 1.53507565s
Average result count: 3.70
min/max result count: 1 / 5
Number of timeouts: 0
Metrics for Query: 7
Count: 20 times executed in whole run
AQET: 0.479562 seconds (arithmetic mean)
AQET(geom.): 0.449338 seconds (geometric mean)
QPS: 2.09 Queries per second
minQET/maxQET: 0.23224386s / 0.78613831s
Average result count: 9.80
min/max result count: 2 / 19
Number of timeouts: 0
Metrics for Query: 8
Count: 10 times executed in whole run
AQET: 0.099458 seconds (arithmetic mean)
AQET(geom.): 0.096052 seconds (geometric mean)
QPS: 10.05 Queries per second
minQET/maxQET: 0.06113880s / 0.15243254s
Average result count: 4.80
min/max result count: 1 / 15
Number of timeouts: 0
Metrics for Query: 9
Count: 0 times executed in whole run
AQET: 0.000000 seconds (arithmetic mean)
AQET(geom.): NaN seconds (geometric mean)
QPS: Infinity Queries per second
minQET/maxQET: 179769313486231570000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.00000000s / 0.00000000s
Average result (Bytes): 0.00
min/max result (Bytes): 2147483647 / -2147483648
Number of timeouts: 0
Metrics for Query: 10
Count: 10 times executed in whole run
AQET: 0.203234 seconds (arithmetic mean)
AQET(geom.): 0.194290 seconds (geometric mean)
QPS: 4.92 Queries per second
minQET/maxQET: 0.10957040s / 0.31831170s
Average result count: 2.60
min/max result count: 0 / 6
Number of timeouts: 0
Metrics for Query: 11
Count: 5 times executed in whole run
AQET: 0.003526 seconds (arithmetic mean)
AQET(geom.): 0.003502 seconds (geometric mean)
QPS: 283.60 Queries per second
minQET/maxQET: 0.00317889s / 0.00435949s
Average result count: 10.00
min/max result count: 10 / 10
Number of timeouts: 0
Metrics for Query: 12
Count: 5 times executed in whole run
AQET: 0.069821 seconds (arithmetic mean)
AQET(geom.): 0.068406 seconds (geometric mean)
QPS: 14.32 Queries per second
minQET/maxQET: 0.05123923s / 0.09591982s
Average result (Bytes): 161.00
min/max result (Bytes): 159 / 162
Number of timeouts: 0
-1: 0.00ms, total: 4394ms
0: 3929.77ms, total: 3952ms
1: 4339.85ms, total: 4354ms
2: 4255.03ms, total: 4287ms
3: 4863.95ms, total: 4881ms
4: 6146.48ms, total: 6158ms
Scale factor: 1000
Number of warmup runs: 1
Seed: 808080
Number of query mix runs (without warmups): 5 times
min/max Querymix runtime: 3.9298s / 6.1465s
Total runtime: 23.535 seconds
QMpH: 764.82 query mixes per hour
CQET: 4.70702 seconds average runtime of query mix
CQET (geom.): 4.64807 seconds geometric mean runtime of query mix
Metrics for Query: 1
Count: 5 times executed in whole run
AQET: 0.015959 seconds (arithmetic mean)
AQET(geom.): 0.014282 seconds (geometric mean)
QPS: 62.66 Queries per second
minQET/maxQET: 0.00694979s / 0.02844427s
Average result count: 1.00
min/max result count: 0 / 2
Number of timeouts: 0
Metrics for Query: 2
Count: 30 times executed in whole run
AQET: 0.127522 seconds (arithmetic mean)
AQET(geom.): 0.123283 seconds (geometric mean)
QPS: 7.84 Queries per second
minQET/maxQET: 0.08767903s / 0.26771191s
Average result count: 21.20
min/max result count: 14 / 31
Number of timeouts: 0
Metrics for Query: 3
Count: 5 times executed in whole run
AQET: 0.057135 seconds (arithmetic mean)
AQET(geom.): 0.056983 seconds (geometric mean)
QPS: 17.50 Queries per second
minQET/maxQET: 0.05018101s / 0.06245950s
Average result count: 0.60
min/max result count: 0 / 1
Number of timeouts: 0
Metrics for Query: 4
Count: 5 times executed in whole run
AQET: 0.059546 seconds (arithmetic mean)
AQET(geom.): 0.059367 seconds (geometric mean)
QPS: 16.79 Queries per second
minQET/maxQET: 0.05078176s / 0.06279754s
Average result count: 0.00
min/max result count: 0 / 0
Number of timeouts: 0
Metrics for Query: 5
Count: 10 times executed in whole run
AQET: 0.606133 seconds (arithmetic mean)
AQET(geom.): 0.541017 seconds (geometric mean)
QPS: 1.65 Queries per second
minQET/maxQET: 0.26759003s / 1.53507565s
Average result count: 3.70
min/max result count: 1 / 5
Number of timeouts: 0
Metrics for Query: 7
Count: 20 times executed in whole run
AQET: 0.479562 seconds (arithmetic mean)
AQET(geom.): 0.449338 seconds (geometric mean)
QPS: 2.09 Queries per second
minQET/maxQET: 0.23224386s / 0.78613831s
Average result count: 9.80
min/max result count: 2 / 19
Number of timeouts: 0
Metrics for Query: 8
Count: 10 times executed in whole run
AQET: 0.099458 seconds (arithmetic mean)
AQET(geom.): 0.096052 seconds (geometric mean)
QPS: 10.05 Queries per second
minQET/maxQET: 0.06113880s / 0.15243254s
Average result count: 4.80
min/max result count: 1 / 15
Number of timeouts: 0
Metrics for Query: 9
Count: 0 times executed in whole run
AQET: 0.000000 seconds (arithmetic mean)
AQET(geom.): NaN seconds (geometric mean)
QPS: Infinity Queries per second
minQET/maxQET: 179769313486231570000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.00000000s / 0.00000000s
Average result (Bytes): 0.00
min/max result (Bytes): 2147483647 / -2147483648
Number of timeouts: 0
Metrics for Query: 10
Count: 10 times executed in whole run
AQET: 0.203234 seconds (arithmetic mean)
AQET(geom.): 0.194290 seconds (geometric mean)
QPS: 4.92 Queries per second
minQET/maxQET: 0.10957040s / 0.31831170s
Average result count: 2.60
min/max result count: 0 / 6
Number of timeouts: 0
Metrics for Query: 11
Count: 5 times executed in whole run
AQET: 0.003526 seconds (arithmetic mean)
AQET(geom.): 0.003502 seconds (geometric mean)
QPS: 283.60 Queries per second
minQET/maxQET: 0.00317889s / 0.00435949s
Average result count: 10.00
min/max result count: 10 / 10
Number of timeouts: 0
Metrics for Query: 12
Count: 5 times executed in whole run
AQET: 0.069821 seconds (arithmetic mean)
AQET(geom.): 0.068406 seconds (geometric mean)
QPS: 14.32 Queries per second
minQET/maxQET: 0.05123923s / 0.09591982s
Average result (Bytes): 161.00
min/max result (Bytes): 159 / 162
Number of timeouts: 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment