Skip to content

Instantly share code, notes, and snippets.

View hyc's full-sized avatar

hyc

434 followers · 3 following
View GitHub Profile
@hyc
hyc / gist:33f3eec6bae83246209d
Last active March 28, 2016 19:47
Monero LMDB blockchain evolution
Input file
violino:/home/software/bitmonero/build/release/bin.perf> ls -l ~/Public/blockchain.raw
-rw-r--r-- 1 hyc hyc 2463754366 Dec 19 07:04 /home/hyc/Public/blockchain.raw
Monero v0.9.0.0
2016-Mar-28 19:33:28.437077 End of file reached
2016-Mar-28 19:33:31.897485 Number of blocks imported: 874829
2016-Mar-28 19:33:31.897709 Finished at block: 874829 total blocks: 874830
@hyc
hyc / a10-4600m_speed.txt
Created January 25, 2016 15:17
OpenSSL speed test
Doing mdc2 for 3s on 16 size blocks: 1424278 mdc2's in 2.93s
Doing mdc2 for 3s on 64 size blocks: 385955 mdc2's in 3.00s
Doing mdc2 for 3s on 256 size blocks: 98679 mdc2's in 2.99s
Doing mdc2 for 3s on 1024 size blocks: 24859 mdc2's in 2.94s
Doing mdc2 for 3s on 8192 size blocks: 3038 mdc2's in 2.93s
Doing md4 for 3s on 16 size blocks: 8688711 md4's in 2.94s
Doing md4 for 3s on 64 size blocks: 6804170 md4's in 2.93s
Doing md4 for 3s on 256 size blocks: 4225590 md4's in 2.96s
Doing md4 for 3s on 1024 size blocks: 1637754 md4's in 2.92s
Doing md4 for 3s on 8192 size blocks: 247283 md4's in 2.91s
@hyc
hyc / output
Last active November 20, 2016 17:08
LevelDB / LMDB readwhilewriting bench, 1.25TB DB, 256GB RAM, 48 cores, 32 reader threads, 1 writer. NVME SSD (2.7TB capacity)w/XFS 6 hour run
[root@localhost ~]# tail out-nvme-2000-LevelDB.txt
2015/08/19-20:47:11 ... thread 1: (200000,19200000) ops and (797.9,898.5) ops/second in (250.646291,21370.034489) seconds
2015/08/19-20:47:14 ... thread 17: (200000,19200000) ops and (802.3,898.3) ops/second in (249.287673,21372.921579) seconds
2015/08/19-20:47:15 ... thread 6: (200000,19200000) ops and (807.9,898.3) ops/second in (247.562653,21373.652192) seconds
writer : 306.279 micros/op 3264 ops/sec; (desired 0 ops/sec)
readwhilewriting : 34.868 micros/op 28679 ops/sec; (19351999 of 19351999 found)
Usr Sys % Wall RSS Major Minor Volun Invol In Out
57837.84 18798.47 353% 6:00:57 131144636 64664683 197163841 1028101967 4398131 33708875504 3820650328
2015/08/19-20:51:57
1417981924 /mnt/dbbench
@hyc
hyc / totp.c
Created June 9, 2015 12:31
RFC6238 TOTP
/* RFC6238 TOTP */
#include <openssl/sha.h>
#include <openssl/hmac.h>
#define HMAC_setup(ctx, key, len, hash) HMAC_CTX_init(&ctx); HMAC_Init_ex(&ctx, key, len, hash, 0)
#define HMAC_crunch(ctx, buf, len) HMAC_Update(&ctx, buf, len)
#define HMAC_finish(ctx, dig, dlen) HMAC_Final(&ctx, dig, &dlen); HMAC_CTX_cleanup(&ctx)
typedef struct myval {
@hyc
hyc / Integer key
Created February 6, 2015 01:21
LMDB integer key vs string key
violino:/home/software/leveldb> ./db_bench_mdb --intkey=1
LMDB: version LMDB 0.9.14: (September 20, 2014)
Date: Fri Feb 6 01:18:51 2015
CPU: 4 * Intel(R) Core(TM)2 Extreme CPU Q9300 @ 2.53GHz
CPUCache: 6144 KB
Keys: 4 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
RawSize: 99.2 MB (estimated)
FileSize: 51.5 MB (estimated)
@hyc
hyc / gist:c8dc7d17d107bf226d26
Last active August 29, 2015 14:14
Safely managing multiple files in a DB is hard
Look at all the bugs in BerkeleyDB's history that are directly attributable to needing
to manage multiple files in a transactional context. The SleepyCat/BerkeleyDB folks were
smart guys but they still got tripped up by this, over and over again. One of the key
principles to LMDB is "don't do anything that's hard" - this is why it's so reliable,
and why it's so efficient. All of the bugs listed here are a type of bug that is
impossible for LMDB to encounter, because LMDB never performs any such operations.
File-related bugs fixed in BDB 4.0
https://gitorious.org/berkeleydb/berkeleydb/source/347d239a1e44ed4f773ae9274c2a32cf2b8999c0:docs/upgrading/changelog_4_0_14.html
#4307
@hyc
hyc / 01vl32
Last active August 29, 2015 14:06
VL32 mode
violino:/home/software/leveldb> ./db_bench_mdb.vl32
LMDB: version LMDB 0.9.14: (September 15, 2014)
Date: Thu Sep 18 20:52:49 2014
CPU: 4 * Intel(R) Core(TM)2 Extreme CPU Q9300 @ 2.53GHz
CPUCache: 6144 KB
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
RawSize: 110.6 MB (estimated)
FileSize: 62.9 MB (estimated)
@hyc
hyc / 01tdb
Last active August 29, 2015 14:06
Samba TDB/NTDB concurrency test
Using https://github.com/hyc/leveldb/commit/0cbfeaa4caa6f6615c0a0caf611e4cdff909465d
Had to revert the TDB_MUTEX_LOCKING code since that doesn't support transactions, and all of the other tests are transactional.
./db_bench_tdb --stats_interval=100000 --benchmarks=fillseqbatch --new_hash=1
TDB: version 1.3.0
Date: Tue Sep 16 02:56:01 2014
CPU: 4 * Intel(R) Core(TM)2 Extreme CPU Q9300 @ 2.53GHz
CPUCache: 6144 KB
Keys: 16 bytes each
@hyc
hyc / gist:7593d1bf21a804c1c5be
Last active August 29, 2015 14:06
Samba NTDB microbench
https://github.com/hyc/leveldb/commit/d05251bb51138f8f77a08e0e01f22c5048d3ccb5
violino:/home/software/leveldb> ./db_bench_ntdb
NTDB: version 1.0
Date: Mon Sep 15 16:51:55 2014
CPU: 4 * Intel(R) Core(TM)2 Extreme CPU Q9300 @ 2.53GHz
CPUCache: 6144 KB
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
@hyc
hyc / 00Before
Created July 5, 2014 17:57
PGO results for LMDB in-memory
violino:~/OD/mdb/libraries/liblmdb> rm -rf /tmp/leveldbtest-1000/*
violino:~/OD/mdb/libraries/liblmdb> ./db_bench_mdb.no_profile --num=10000000
LMDB: version LMDB 0.9.14: (June 20, 2014)
Date: Sat Jul 5 10:47:17 2014
CPU: 4 * Intel(R) Core(TM)2 Extreme CPU Q9300 @ 2.53GHz
CPUCache: 6144 KB
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 10000000
RawSize: 1106.3 MB (estimated)