Skip to content

Instantly share code, notes, and snippets.

@cswinter

cswinter/locustdb-mem-breakdown

Created August 12, 2018 19:46
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save cswinter/f0a48b80033d1736585dd5f73ae0bab5 to your computer and use it in GitHub Desktop.
Save cswinter/f0a48b80033d1736585dd5f73ae0bab5 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
locustdb-mem-breakdown
➜ ~ cd src/LocustDB
➜ LocustDB git:(master) git fetch origin
^C
➜ LocustDB git:(master) git fetch && rebase
Enter passphrase for key '/home/clemens/.ssh/id_rsa':
➜ LocustDB git:(master) git fetch && git rebase
Enter passphrase for key '/home/clemens/.ssh/id_rsa':
remote: Counting objects: 21, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 21 (delta 18), reused 21 (delta 18), pack-reused 0
Unpacking objects: 100% (21/21), done.
From github.com:cswinter/LocustDB
9f9cffa..3455b20 master -> origin/master
First, rewinding head to replay your work on top of it...
Applying: Fix compilation for older Rust versions
➜ LocustDB git:(master) git lg
* 8a89e0d Fix compilation for older Rust versions (HEAD -> master) (4 seconds ago) <Clemens Winter>
* 3455b20 Hex encoding (origin/master, origin/HEAD) (24 minutes ago) <Clemens Winter>
* 9f9cffa Fix incorrect application of limits during merge (24 hours ago) <Clemens Winter>
* 3596abf Use Cap'n Proto for serialization (25 hours ago) <Clemens Winter>
* a977047 Reduce memory usage for ingestionand bulk load (3 days ago) <Clemens Winter>
* 294bffa Fix travic ci build? (4 days ago) <Clemens Winter>
* f43d8a3 Add common subexpression elimination pass (4 days ago) <Clemens Winter>
* 6b2067b Enforce limit on in-memory size of tables (5 days ago) <Clemens Winter>
* 56e934b Make lz4 work with packed strings (6 days ago) <Clemens Winter>
* 8ccefcb Update usage and fix typo (6 days ago) <Clemens Winter>
* 0c83f98 Make packed string columns work again (6 days ago) <Clemens Winter>
* ca3bb9f Add delta encoding (6 days ago) <Clemens Winter>
* 08ab6d7 Update instructions for lz4 and RocksDB (6 days ago) <Clemens Winter>
* 243eb4a Use clap to improve UX and expose options (#22) (6 days ago) <Clemens Winter>
* d0133d6 Only send email on failing builds (6 days ago) <Clemens Winter>
* 929d5c4 Tweak :memtree output (7 days ago) <Clemens Winter>
* 32c458d Only lz4 encode for > 10% size reduction (7 days ago) <Clemens Winter>
* 2aabddf Add :restore command (7 days ago) <Clemens Winter>
* bc3062c Add :memtree command (7 days ago) <Clemens Winter>
* 8183d16 Brag about storage efficiency (9 days ago) <Clemens Winter>
* 83d4f41 Add lz4 compression (#19) (9 days ago) <Clemens Winter>
* ecb57f5 Add RocksDB storage engine (#18) (11 days ago) <Clemens Winter>
* 731c6ea Add RocksDB dependency (#16) (2 weeks ago) <Clemens Winter>
* 841b925 Git hook (#15) (2 weeks ago) <Clemens Winter>
* 013236c Delete accidentally added file (2 weeks ago) <Clemens Winter>
* a1d9a39 Use normal query plans for decode (2 weeks ago) <Clemens Winter>
* c198832 Add docs for fixing `No such subcommand: +nightly` (3 weeks ago) <Clemens Winter>
* 48b91cc Allow queries to have leading whitespace (3 weeks ago) <Dirk Jonker>
* c14fb20 Add link to "Diving Into Other People's Code" (4 weeks ago) <cswinter>
* 4510a88 Update CONTRIBUTING.md (4 weeks ago) <cswinter>
* a991a68 Fix link to blogpost (4 weeks ago) <Mike Graf>
* fdea721 Typo (4 weeks ago) <cswinter>
* f598995 Update README.md (4 weeks ago) <cswinter>
* 469deb9 Update README.md (4 weeks ago) <cswinter>
* c2da059 Add goals to README.md (4 weeks ago) <cswinter>
* 9ee8362 Steal from the best (4 weeks ago) <cswinter>
* 217af63 Add Gitter badge (4 weeks ago) <The Gitter Badger>
* f329495 Add :show(x) command (4 weeks ago) <Clemens Winter>
* d6fd8bd Make "nerf" feature work (tag: 2018-07-6-nerfed-results, tag: 2018-07-06-bencharks-nerfed) (4 weeks ago) <Clemens Winter>
➜ LocustDB git:(master) git push
Enter passphrase for key '/home/clemens/.ssh/id_rsa':
Counting objects: 4, done.
Delta compression using up to 12 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 405 bytes | 0 bytes/s, done.
Total 4 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To git@github.com:cswinter/LocustDB.git
3455b20..8a89e0d master -> master
➜ LocustDB git:(master)
➜ LocustDB git:(master) rm -f /home/clemens/ldb; RUSTFLAGS="-Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --load /mnt/data/nyc-taxi-data-denormalized/trips_x* --trips --db-path /home/clemens/ldb --table trips
rm: cannot remove '/home/clemens/ldb': Is a directory
Downloading hex v0.3.2
Compiling hex v0.3.2
Compiling locustdb v0.1.0-alpha (file:///home/clemens/src/LocustDB)
^C
➜ LocustDB git:(master) rm -r /home/clemens/ldb; RUSTFLAGS="-Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --load /mnt/data/nyc-taxi-data-denormalized/trips_x* --trips --db-path /home/clemens/ldb --table trips
Compiling locustdb v0.1.0-alpha (file:///home/clemens/src/LocustDB)
^C
➜ LocustDB git:(master) ulimit -a
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 8192
-c: core file size (blocks) 0
-m: resident set size (kbytes) unlimited
-u: processes 256135
-n: file descriptors 1024
-l: locked-in-memory size (kbytes) 64
-v: address space (kbytes) unlimited
-x: file locks unlimited
-i: pending signals 256135
-q: bytes in POSIX msg queues 819200
-e: max nice 0
-r: max rt priority 0
-N 15: unlimited
➜ LocustDB git:(master) ulimit -n 4096
➜ LocustDB git:(master) rm -r /home/clemens/ldb; RUSTFLAGS="-Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --load /mnt/data/nyc-taxi-data-denormalized/trips_x* --trips --db-path /home/clemens/ldb --table trips
rm: cannot remove '/home/clemens/ldb': No such file or directory
Compiling locustdb v0.1.0-alpha (file:///home/clemens/src/LocustDB)
Finished release [optimized] target(s) in 1m 00s
Running `target/release/repl --load /mnt/data/nyc-taxi-data-denormalized/trips_xaa.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xab.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xac.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xad.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xae.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xag.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xah.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xai.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xak.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xal.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xam.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xan.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xao.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xap.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xar.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xas.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xat.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xau.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xav.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaw.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xax.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xay.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaz.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xba.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbb.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbc.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbd.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbe.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbg.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbh.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbi.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbk.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbl.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbm.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbn.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbo.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbp.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbr.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbs.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbt.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbu.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbv.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbw.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbx.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xby.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbz.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xca.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcb.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcc.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcd.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xce.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcg.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xch.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xci.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xck.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcl.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcm.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcn.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xco.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcp.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcr.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcs.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xct.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcu.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcv.csv.gz --trips --db-path /home/clemens/ldb --table trips`
Loading 74 files into table trips.
[1] 2840 killed RUSTFLAGS="-Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin
➜ LocustDB git:(master) rm -r /home/clemens/ldb; RUSTFLAGS="-Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --load /mnt/data/nyc-taxi-data-denormalized/trips_x* --trips --db-path /home/clemens/ldb --table trips --help
Finished release [optimized] target(s) in 0.47s
Running `target/release/repl --load /mnt/data/nyc-taxi-data-denormalized/trips_xaa.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xab.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xac.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xad.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xae.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xag.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xah.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xai.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xak.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xal.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xam.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xan.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xao.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xap.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xar.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xas.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xat.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xau.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xav.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaw.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xax.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xay.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaz.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xba.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbb.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbc.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbd.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbe.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbg.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbh.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbi.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbk.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbl.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbm.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbn.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbo.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbp.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbr.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbs.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbt.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbu.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbv.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbw.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbx.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xby.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbz.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xca.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcb.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcc.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcd.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xce.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcg.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xch.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xci.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xck.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcl.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcm.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcn.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xco.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcp.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcr.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcs.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xct.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcu.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcv.csv.gz --trips --db-path /home/clemens/ldb --table trips --help`
LocustDB 0.1.0-alpha
Clemens Winter <clemenswinter1@gmail.com>
Massively parallel, high performance analytics database that will rapidly devour all of your data.
USAGE:
repl [FLAGS] [OPTIONS]
FLAGS:
-h, --help Prints help information
--reduced-trips Set ingestion schema for select set of columns from nyc taxi ride dataset
--trips Set ingestion schema for nyc taxi ride dataset
-V, --version Prints version information
OPTIONS:
--db-path <PATH> Path to data directory
--load <CSV_FILE>... Load .csv or .csv.gz files into the database
--mem-limit-tables <GB> Limit for in-memory size of tables in GiB [default: 64]
--partition-size <INTEGER> Number of rows per partition when loading new data [default: 1048576]
--table <NAME> Name for the table populated with --load [default: default]
--threads <INTEGER> Number of worker threads. [default: number of cores]
➜ LocustDB git:(master) rm -r /home/clemens/ldb; RUSTFLAGS="-Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --load /mnt/data/nyc-taxi-data-denormalized/trips_x* --trips --db-path /home/clemens/ldb --table trips --mem-limit-tables=32
rm: cannot remove '/home/clemens/ldb': No such file or directory
Finished release [optimized] target(s) in 0.15s
Running `target/release/repl --load /mnt/data/nyc-taxi-data-denormalized/trips_xaa.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xab.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xac.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xad.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xae.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xag.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xah.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xai.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xak.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xal.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xam.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xan.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xao.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xap.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xar.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xas.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xat.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xau.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xav.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaw.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xax.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xay.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaz.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xba.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbb.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbc.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbd.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbe.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbg.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbh.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbi.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbk.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbl.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbm.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbn.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbo.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbp.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbr.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbs.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbt.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbu.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbv.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbw.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbx.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xby.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbz.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xca.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcb.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcc.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcd.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xce.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcg.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xch.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xci.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xck.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcl.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcm.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcn.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xco.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcp.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcr.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcs.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xct.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcu.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcv.csv.gz --trips --db-path /home/clemens/ldb --table trips --mem-li
Loading 74 files into table trips.
Loaded data in 2.54ks.
# Table `_meta_tables` (1 rows, 8.00B) #
timestamp: 0.00B
name: 0.00B
# Table `trips` (1464785771 rows, 32.0GiB) #
trip_id: 26MiB
fare_amount: 1.1GiB
dropoff_boroname: 41MiB
max_temperature: 0.45GiB
trip_distance: 1.6GiB
pickup_nyct2010_gid: 0.89GiB
cab_type: 3.2MiB
improvement_surcharge: 6.9MiB
dropoff_ct2010: 0.17GiB
snow_depth: 94MiB
tip_amount: 0.88GiB
pickup_ct2010: 1.1GiB
mta_tax: 15MiB
dropoff_nyct2010_gid: 0.13GiB
pickup_boroname: 0.11GiB
dropoff_longitude: 40MiB
pickup_longitude: 0.18GiB
payment_type: 0.31GiB
dropoff_boroct2010: 0.18GiB
dropoff_cdeligibil: 51MiB
dropoff_latitude: 11MiB
pickup_ctlabel: 0.89GiB
pickup_puma: 0.45GiB
min_temperature: 0.44GiB
dropoff_puma: 92MiB
dropoff_ctlabel: 0.13GiB
tolls_amount: 0.14GiB
vendor_id: 0.30GiB
dropoff: 5.9GiB
dropoff_ntacode: 84MiB
average_wind_speed: 0.61GiB
pickup_ntacode: 0.45GiB
trip_type: 51KiB
extra: 0.41GiB
dropoff_datetime: 2.3GiB
pickup: 5.9GiB
pickup_boroct2010: 1.1GiB
dropoff_ntaname: 86MiB
passenger_count: 0.30GiB
precipitation: 0.36GiB
dropoff_borocode: 40MiB
pickup_datetime: 2.1GiB
pickup_cdeligibil: 0.17GiB
total_amount: 1.4GiB
store_and_fwd_flag: 0.17GiB
pickup_latitude: 16MiB
rate_code_id: 63MiB
pickup_ntaname: 0.45GiB
snowfall: 71MiB
ehail_fee: 3.1MiB
pickup_borocode: 0.11GiB
locustdb> :memtree
_meta_tables 0.000B 1 rows
├─ name nonresident
└─ timestamp nonresident
trips 32.0GiB 1464785771 rows
├─ average_wind_speed 45% resident 0.61GiB 1.9% 0.99B/row
├─ cab_type 45% resident 3.2MiB 0.01% 5.1mB/row
├─ dropoff 45% resident 5.9GiB 19% 9.8B/row
├─ dropoff_borocode 45% resident 40MiB 0.12% 65mB/row
├─ dropoff_boroct2010 45% resident 0.18GiB 0.55% 0.29B/row
├─ dropoff_boroname 45% resident 41MiB 0.12% 65mB/row
├─ dropoff_cdeligibil 45% resident 51MiB 0.15% 81mB/row
├─ dropoff_ct2010 45% resident 0.17GiB 0.53% 0.28B/row
├─ dropoff_ctlabel 45% resident 0.13GiB 0.39% 0.21B/row
├─ dropoff_datetime 45% resident 2.3GiB 7.2% 3.8B/row
├─ dropoff_latitude 45% resident 11MiB 0.03% 18mB/row
├─ dropoff_longitude 45% resident 40MiB 0.12% 63mB/row
├─ dropoff_ntacode 45% resident 84MiB 0.26% 0.13B/row
├─ dropoff_ntaname 45% resident 86MiB 0.26% 0.14B/row
├─ dropoff_nyct2010_gid 45% resident 0.13GiB 0.40% 0.21B/row
├─ dropoff_puma 45% resident 92MiB 0.28% 0.15B/row
├─ ehail_fee 45% resident 3.1MiB 0.01% 5.0mB/row
├─ extra 45% resident 0.41GiB 1.3% 0.67B/row
├─ fare_amount 45% resident 1.1GiB 3.3% 1.8B/row
├─ improvement_surcharge 45% resident 6.9MiB 0.02% 11mB/row
├─ max_temperature 45% resident 0.45GiB 1.4% 0.73B/row
├─ min_temperature 45% resident 0.44GiB 1.4% 0.73B/row
├─ mta_tax 45% resident 15MiB 0.04% 24mB/row
├─ passenger_count 45% resident 0.30GiB 0.95% 0.50B/row
├─ payment_type 45% resident 0.31GiB 0.96% 0.50B/row
├─ pickup 45% resident 5.9GiB 19% 9.8B/row
├─ pickup_borocode 45% resident 0.11GiB 0.35% 0.18B/row
├─ pickup_boroct2010 45% resident 1.1GiB 3.5% 1.8B/row
├─ pickup_boroname 45% resident 0.11GiB 0.35% 0.18B/row
├─ pickup_cdeligibil 45% resident 0.17GiB 0.52% 0.27B/row
├─ pickup_ct2010 45% resident 1.1GiB 3.5% 1.8B/row
├─ pickup_ctlabel 45% resident 0.89GiB 2.8% 1.5B/row
├─ pickup_datetime 45% resident 2.1GiB 6.6% 3.4B/row
├─ pickup_latitude 45% resident 16MiB 0.05% 25mB/row
├─ pickup_longitude 45% resident 0.17GiB 0.55% 0.29B/row
├─ pickup_ntacode 45% resident 0.45GiB 1.4% 0.73B/row
├─ pickup_ntaname 45% resident 0.45GiB 1.4% 0.74B/row
├─ pickup_nyct2010_gid 45% resident 0.89GiB 2.8% 1.5B/row
├─ pickup_puma 45% resident 0.45GiB 1.4% 0.73B/row
├─ precipitation 45% resident 0.36GiB 1.1% 0.58B/row
├─ rate_code_id 45% resident 63MiB 0.19% 0.10B/row
├─ snow_depth 45% resident 94MiB 0.29% 0.15B/row
├─ snowfall 45% resident 71MiB 0.22% 0.11B/row
├─ store_and_fwd_flag 45% resident 0.16GiB 0.51% 0.27B/row
├─ tip_amount 45% resident 0.88GiB 2.7% 1.4B/row
├─ tolls_amount 45% resident 0.14GiB 0.45% 0.23B/row
├─ total_amount 45% resident 1.4GiB 4.4% 2.3B/row
├─ trip_distance 45% resident 1.6GiB 5.0% 2.7B/row
├─ trip_id 45% resident 26MiB 0.08% 41mB/row
├─ trip_type 45% resident 51KiB 0.00% 81μB/row
└─ vendor_id 45% resident 0.30GiB 0.94% 0.50B/row
locustdb> :memtree(3)
_meta_tables 0.000B 1 rows
├─ name nonresident
└─ timestamp nonresident
trips 32.0GiB 1464785771 rows
├─ average_wind_speed 45% resident 0.61GiB 1.9% 0.99B/row
│ ├─ LZ4(U16) ToI64(U16) 0.61GiB 100% 0.99B/row 100%
│ └─ LZ4(U8) Add(U8) 2.0KiB 0.00% 6.6mB/row 0.05%
├─ cab_type 45% resident 3.2MiB 0.01% 5.1mB/row
│ └─ LZ4(U8) Data(1) Data(2) Dict(U8) 3.2MiB 100% 5.1mB/row 100%
├─ dropoff 45% resident 5.9GiB 19% 9.8B/row
│ ├─ LZ4(U8) StrHexUnpack 5.9GiB 100% 13B/row 73%
│ ├─ LZ4(U16) Data(1) Data(2) Dict(U16) 8.2MiB 0.13% 8.2B/row 0.16%
│ └─ 12KiB 0.00% 72μB/row 27%
├─ dropoff_borocode 45% resident 40MiB 0.12% 65mB/row
│ ├─ LZ4(U8) ToI64(U8) 40MiB 100% 88mB/row 73%
│ └─ 14KiB 0.03% 80μB/row 27%
├─ dropoff_boroct2010 45% resident 0.18GiB 0.55% 0.29B/row
│ ├─ LZ4(U32) ToI64(U32) 0.18GiB 100% 0.39B/row 73%
│ └─ 16KiB 0.01% 96μB/row 27%
├─ dropoff_boroname 45% resident 41MiB 0.12% 65mB/row
│ ├─ LZ4(U8) Data(1) Data(2) Dict(U8) 41MiB 100% 89mB/row 73%
│ └─ 14KiB 0.03% 80μB/row 27%
├─ dropoff_cdeligibil 45% resident 51MiB 0.15% 81mB/row
│ ├─ LZ4(U8) Data(1) Data(2) Dict(U8) 51MiB 100% 0.11B/row 73%
│ └─ 16KiB 0.03% 96μB/row 27%
├─ dropoff_ct2010 45% resident 0.17GiB 0.53% 0.28B/row
│ ├─ LZ4(U32) ToI64(U32) 0.17GiB 100% 0.38B/row 73%
│ └─ 14KiB 0.01% 80μB/row 27%
├─ dropoff_ctlabel 45% resident 0.13GiB 0.39% 0.21B/row
│ ├─ LZ4(U16) ToI64(U16) 0.13GiB 100% 0.28B/row 73%
│ └─ 14KiB 0.01% 80μB/row 27%
├─ dropoff_datetime 45% resident 2.3GiB 7.2% 3.8B/row
│ ├─ ToI64(U32) 1.8GiB 76% 4.0B/row 72%
│ ├─ LZ4(U32) ToI64(U32) 0.53GiB 23% 3.2B/row 27%
│ ├─ LZ4(I64) 20MiB 0.85% 5.0B/row 0.64%
│ └─ Add(U32) 8.0MiB 0.34% 4.0B/row 0.32%
├─ dropoff_latitude 45% resident 11MiB 0.03% 18mB/row
│ ├─ LZ4(U8) ToI64(U8) 5.3MiB 47% 22mB/row 39%
│ ├─ LZ4(U16) Add(U16) 2.5MiB 22% 33mB/row 12%
│ ├─ LZ4(U8) Add(U8) 2.4MiB 21% 23mB/row 16%
│ ├─ LZ4(U16) ToI64(U16) 1.1MiB 9.9% 28mB/row 6.4%
│ └─ 14KiB 0.12% 80μB/row 27%
├─ dropoff_longitude 45% resident 40MiB 0.12% 63mB/row
│ ├─ LZ4(U8) Add(U8) 26MiB 66% 80mB/row 53%
│ ├─ LZ4(U16) Add(U16) 13MiB 33% 0.10B/row 20%
│ └─ 16KiB 0.04% 96μB/row 27%
├─ dropoff_ntacode 45% resident 84MiB 0.26% 0.13B/row
│ ├─ LZ4(U8) Data(1) Data(2) Dict(U8) 84MiB 100% 0.18B/row 73%
│ └─ 14KiB 0.02% 80μB/row 27%
├─ dropoff_ntaname 45% resident 86MiB 0.26% 0.14B/row
│ ├─ LZ4(U8) Data(1) Data(2) Dict(U8) 86MiB 100% 0.19B/row 73%
│ └─ 14KiB 0.02% 80μB/row 27%
├─ dropoff_nyct2010_gid 45% resident 0.13GiB 0.40% 0.21B/row
│ ├─ LZ4(U16) ToI64(U16) 0.13GiB 100% 0.29B/row 73%
│ └─ 16KiB 0.01% 96μB/row 27%
├─ dropoff_puma 45% resident 92MiB 0.28% 0.15B/row
│ ├─ LZ4(U16) ToI64(U16) 91MiB 99% 0.20B/row 73%
│ └─ LZ4(U8) ToI64(U8) 0.83MiB 0.90% 5.0mB/row 27%
├─ ehail_fee 45% resident 3.1MiB 0.01% 5.0mB/row
│ └─ LZ4(U8) ToI64(U8) 3.1MiB 100% 5.0mB/row 100%
├─ extra 45% resident 0.41GiB 1.3% 0.67B/row
│ ├─ LZ4(U16) ToI64(U16) 0.19GiB 46% 0.87B/row 35%
│ ├─ LZ4(U16) Add(U16) 0.18GiB 45% 0.52B/row 58%
│ ├─ LZ4(U32) Add(U32) 20MiB 4.9% 1.1B/row 2.9%
│ ├─ LZ4(U8) ToI64(U8) 16MiB 3.9% 0.62B/row 4.2%
│ └─ LZ4(U8) Add(U8) 0.22MiB 0.05% 0.33B/row 0.11%
├─ fare_amount 45% resident 1.1GiB 3.3% 1.8B/row
│ ├─ LZ4(U32) Add(U32) 0.63GiB 59% 1.8B/row 59%
│ ├─ LZ4(U16) ToI64(U16) 0.41GiB 39% 1.8B/row 39%
│ ├─ LZ4(U16) Add(U16) 20MiB 1.8% 1.8B/row 1.8%
│ ├─ LZ4(U32) ToI64(U32) 3.5MiB 0.32% 1.8B/row 0.32%
│ └─ Add(U16) 0.16MiB 0.01% 2.1B/row 0.01%
├─ improvement_surcharge 45% resident 6.9MiB 0.02% 11mB/row
│ ├─ LZ4(U8) ToI64(U8) 4.0MiB 58% 15mB/row 44%
│ ├─ LZ4(U8) Add(U8) 2.9MiB 41% 8.3mB/row 55%
│ └─ LZ4(U16) Add(U16) 48KiB 0.68% 12mB/row 0.64%
├─ max_temperature 45% resident 0.45GiB 1.4% 0.73B/row
│ ├─ ToI64(U8) 0.45GiB 100% 1.0B/row 73%
│ └─ LZ4(U8) ToI64(U8) 1.1MiB 0.24% 6.7mB/row 27%
├─ min_temperature 45% resident 0.44GiB 1.4% 0.73B/row
│ ├─ ToI64(U8) 0.41GiB 92% 1.0B/row 67%
│ ├─ LZ4(U8) ToI64(U8) 22MiB 4.9% 0.12B/row 31%
│ └─ Add(U8) 12MiB 2.7% 1.0B/row 1.9%
├─ mta_tax 45% resident 15MiB 0.04% 24mB/row
│ ├─ LZ4(U16) Add(U16) 7.6MiB 52% 30mB/row 40%
│ ├─ LZ4(U8) ToI64(U8) 3.7MiB 25% 15mB/row 39%
│ ├─ LZ4(U8) Add(U8) 3.4MiB 23% 26mB/row 21%
│ └─ LZ4(U16) ToI64(U16) 40KiB 0.27% 20mB/row 0.32%
├─ passenger_count 45% resident 0.30GiB 0.95% 0.50B/row
│ └─ LZ4(U8) ToI64(U8) 0.30GiB 100% 0.50B/row 100%
├─ payment_type 45% resident 0.31GiB 0.96% 0.50B/row
│ └─ LZ4(U8) Data(1) Data(2) Dict(U8) 0.31GiB 100% 0.50B/row 100%
├─ pickup 45% resident 5.9GiB 19% 9.8B/row
│ ├─ LZ4(U8) StrHexUnpack 5.9GiB 100% 13B/row 73%
│ ├─ LZ4(U16) Data(1) Data(2) Dict(U16) 8.2MiB 0.13% 8.2B/row 0.16%
│ └─ 12KiB 0.00% 72μB/row 27%
├─ pickup_borocode 45% resident 0.11GiB 0.35% 0.18B/row
│ ├─ LZ4(U8) ToI64(U8) 0.11GiB 100% 0.25B/row 73%
│ └─ 14KiB 0.01% 80μB/row 27%
├─ pickup_boroct2010 45% resident 1.1GiB 3.5% 1.8B/row
│ ├─ LZ4(U32) ToI64(U32) 1.1GiB 100% 2.5B/row 73%
│ └─ 16KiB 0.00% 96μB/row 27%
├─ pickup_boroname 45% resident 0.11GiB 0.35% 0.18B/row
│ ├─ LZ4(U8) Data(1) Data(2) Dict(U8) 0.11GiB 100% 0.25B/row 73%
│ └─ 14KiB 0.01% 80μB/row 27%
├─ pickup_cdeligibil 45% resident 0.17GiB 0.52% 0.27B/row
│ ├─ LZ4(U8) Data(1) Data(2) Dict(U8) 0.17GiB 100% 0.37B/row 73%
│ └─ 16KiB 0.01% 96μB/row 27%
├─ pickup_ct2010 45% resident 1.1GiB 3.5% 1.8B/row
│ ├─ LZ4(U32) ToI64(U32) 1.1GiB 100% 2.5B/row 73%
│ └─ 14KiB 0.00% 80μB/row 27%
├─ pickup_ctlabel 45% resident 0.89GiB 2.8% 1.5B/row
│ ├─ ToI64(U16) 0.89GiB 100% 2.0B/row 73%
│ ├─ LZ4(U16) ToI64(U16) 0.19MiB 0.02% 0.19B/row 0.16%
│ └─ 14KiB 0.00% 80μB/row 27%
├─ pickup_datetime 45% resident 2.1GiB 6.6% 3.4B/row
│ ├─ ToI64(U32) 1.8GiB 85% 4.0B/row 73%
│ ├─ LZ4(U32) ToI64(U32) 0.32GiB 15% 1.9B/row 27%
│ ├─ LZ4(U16) Add(U16) 0.36MiB 0.02% 1.2B/row 0.05%
│ └─ Add(U16) 0.16MiB 0.01% 2.1B/row 0.01%
├─ pickup_latitude 45% resident 16MiB 0.05% 25mB/row
│ ├─ LZ4(U8) ToI64(U8) 9.4MiB 60% 34mB/row 45%
│ ├─ LZ4(U8) Add(U8) 3.5MiB 22% 33mB/row 17%
│ ├─ LZ4(U16) Add(U16) 2.6MiB 16% 40mB/row 10%
│ ├─ LZ4(U16) ToI64(U16) 0.25MiB 1.6% 35mB/row 1.1%
│ └─ 14KiB 0.08% 80μB/row 27%
├─ pickup_longitude 45% resident 0.17GiB 0.55% 0.29B/row
│ ├─ LZ4(U8) Add(U8) 0.13GiB 74% 0.37B/row 57%
│ ├─ LZ4(U16) Add(U16) 47MiB 26% 0.46B/row 16%
│ └─ 14KiB 0.01% 80μB/row 27%
├─ pickup_ntacode 45% resident 0.45GiB 1.4% 0.73B/row
│ ├─ Data(1) Data(2) Dict(U8) 0.45GiB 100% 1.0B/row 73%
│ ├─ LZ4(U8) Data(1) Data(2) Dict(U8) 99KiB 0.02% 97mB/row 0.16%
│ └─ 14KiB 0.00% 80μB/row 27%
├─ pickup_ntaname 45% resident 0.45GiB 1.4% 0.74B/row
│ ├─ Data(1) Data(2) Dict(U8) 0.45GiB 100% 1.0B/row 73%
│ └─ LZ4(U8) Data(1) Data(2) Dict(U8) 0.94MiB 0.20% 5.6mB/row 27%
├─ pickup_nyct2010_gid 45% resident 0.89GiB 2.8% 1.5B/row
│ ├─ ToI64(U16) 0.89GiB 100% 2.0B/row 73%
│ ├─ LZ4(U16) ToI64(U16) 0.19MiB 0.02% 0.19B/row 0.16%
│ └─ 16KiB 0.00% 96μB/row 27%
├─ pickup_puma 45% resident 0.45GiB 1.4% 0.73B/row
│ ├─ LZ4(U16) ToI64(U16) 0.45GiB 100% 1.00B/row 73%
│ └─ LZ4(U8) ToI64(U8) 0.83MiB 0.18% 5.0mB/row 27%
├─ precipitation 45% resident 0.36GiB 1.1% 0.58B/row
│ ├─ LZ4(U16) ToI64(U16) 0.35GiB 98% 0.66B/row 87%
│ └─ LZ4(U8) ToI64(U8) 6.1MiB 1.7% 76mB/row 13%
├─ rate_code_id 45% resident 63MiB 0.19% 0.10B/row
│ └─ LZ4(U8) ToI64(U8) 63MiB 100% 0.10B/row 100%
├─ snow_depth 45% resident 94MiB 0.29% 0.15B/row
│ ├─ LZ4(U16) ToI64(U16) 92MiB 98% 0.56B/row 27%
│ └─ LZ4(U8) ToI64(U8) 2.3MiB 2.4% 5.0mB/row 73%
├─ snowfall 45% resident 71MiB 0.22% 0.11B/row
│ ├─ LZ4(U16) ToI64(U16) 66MiB 92% 0.42B/row 25%
│ └─ LZ4(U8) ToI64(U8) 5.4MiB 7.6% 12mB/row 75%
├─ store_and_fwd_flag 45% resident 0.16GiB 0.51% 0.27B/row
│ └─ LZ4(U8) Data(1) Data(2) Dict(U8) 0.16GiB 100% 0.27B/row 100%
├─ tip_amount 45% resident 0.88GiB 2.7% 1.4B/row
│ ├─ LZ4(U16) Add(U16) 0.46GiB 52% 1.5B/row 49%
│ ├─ LZ4(U16) ToI64(U16) 0.30GiB 34% 1.3B/row 39%
│ └─ LZ4(U32) Add(U32) 0.12GiB 14% 1.7B/row 11%
├─ tolls_amount 45% resident 0.14GiB 0.45% 0.23B/row
│ ├─ LZ4(U32) Add(U32) 59MiB 40% 0.30B/row 31%
│ ├─ LZ4(U16) ToI64(U16) 47MiB 32% 0.19B/row 41%
│ ├─ LZ4(U16) Add(U16) 38MiB 26% 0.22B/row 27%
│ └─ LZ4(U32) ToI64(U32) 2.2MiB 1.5% 0.28B/row 1.3%
├─ total_amount 45% resident 1.4GiB 4.4% 2.3B/row
│ ├─ LZ4(U32) Add(U32) 0.92GiB 65% 2.5B/row 60%
│ ├─ ToI64(U16) 0.47GiB 33% 2.0B/row 38%
│ ├─ LZ4(U32) ToI64(U32) 13MiB 0.88% 2.5B/row 0.81%
│ └─ Add(U16) 6.3MiB 0.44% 2.0B/row 0.50%
├─ trip_distance 45% resident 1.6GiB 5.0% 2.7B/row
│ ├─ LZ4(U32) ToI64(U32) 1.3GiB 80% 2.5B/row 84%
│ ├─ LZ4(I64) 0.32GiB 20% 3.5B/row 15%
│ ├─ ToI64(U16) 2.7MiB 0.16% 2.1B/row 0.20%
│ └─ LZ4(U32) Add(U32) 2.5MiB 0.15% 2.5B/row 0.16%
├─ trip_id 45% resident 26MiB 0.08% 41mB/row
│ ├─ LZ4(U32) Add(U32) Delta(I64) 26MiB 100% 41mB/row 100%
│ └─ LZ4(U32) Delta(U32) 1.6KiB 0.01% 21mB/row 0.01%
├─ trip_type 45% resident 51KiB 0.00% 81μB/row
│ └─ 51KiB 100% 81μB/row 100%
└─ vendor_id 45% resident 0.30GiB 0.94% 0.50B/row
└─ LZ4(U8) Data(1) Data(2) Dict(U8) 0.30GiB 100% 0.50B/row 100%
locustdb> :memtree(4)
_meta_tables 0.000B 1 rows
├─ name nonresident
└─ timestamp nonresident
trips 32.0GiB 1464785771 rows
├─ average_wind_speed 45% resident 0.61GiB 1.9% 0.99B/row
│ ├─ LZ4(U16) ToI64(U16) 0.61GiB 100% 0.99B/row 100%
│ └─ LZ4(U8) Add(U8) 2.0KiB 0.00% 6.6mB/row 0.05%
├─ cab_type 45% resident 3.2MiB 0.01% 5.1mB/row
│ └─ LZ4(U8) Data(1) Data(2) Dict(U8) 3.2MiB 100% 5.1mB/row 100%
│ ├─ .0 3.0MiB
│ ├─ .1 5.1KiB
│ └─ .2 5.1KiB
├─ dropoff 45% resident 5.9GiB 19% 9.8B/row
│ ├─ LZ4(U8) StrHexUnpack 5.9GiB 100% 13B/row 73%
│ ├─ LZ4(U16) Data(1) Data(2) Dict(U16) 8.2MiB 0.13% 8.2B/row 0.16%
│ │ ├─ .0 0.19MiB
│ │ ├─ .1 1.0MiB
│ │ └─ .2 7.0MiB
│ └─ 12KiB 0.00% 72μB/row 27%
├─ dropoff_borocode 45% resident 40MiB 0.12% 65mB/row
│ ├─ LZ4(U8) ToI64(U8) 40MiB 100% 88mB/row 73%
│ └─ 14KiB 0.03% 80μB/row 27%
├─ dropoff_boroct2010 45% resident 0.18GiB 0.55% 0.29B/row
│ ├─ LZ4(U32) ToI64(U32) 0.18GiB 100% 0.39B/row 73%
│ └─ 16KiB 0.01% 96μB/row 27%
├─ dropoff_boroname 45% resident 41MiB 0.12% 65mB/row
│ ├─ LZ4(U8) Data(1) Data(2) Dict(U8) 41MiB 100% 89mB/row 73%
│ │ ├─ .0 41MiB
│ │ ├─ .1 30KiB
│ │ └─ .2 30KiB
│ └─ 14KiB 0.03% 80μB/row 27%
├─ dropoff_cdeligibil 45% resident 51MiB 0.15% 81mB/row
│ ├─ LZ4(U8) Data(1) Data(2) Dict(U8) 51MiB 100% 0.11B/row 73%
│ │ ├─ .0 50MiB
│ │ ├─ .1 15KiB
│ │ └─ .2 3.8KiB
│ └─ 16KiB 0.03% 96μB/row 27%
├─ dropoff_ct2010 45% resident 0.17GiB 0.53% 0.28B/row
│ ├─ LZ4(U32) ToI64(U32) 0.17GiB 100% 0.38B/row 73%
│ └─ 14KiB 0.01% 80μB/row 27%
├─ dropoff_ctlabel 45% resident 0.13GiB 0.39% 0.21B/row
│ ├─ LZ4(U16) ToI64(U16) 0.13GiB 100% 0.28B/row 73%
│ └─ 14KiB 0.01% 80μB/row 27%
├─ dropoff_datetime 45% resident 2.3GiB 7.2% 3.8B/row
│ ├─ ToI64(U32) 1.8GiB 76% 4.0B/row 72%
│ ├─ LZ4(U32) ToI64(U32) 0.53GiB 23% 3.2B/row 27%
│ ├─ LZ4(I64) 20MiB 0.85% 5.0B/row 0.64%
│ └─ Add(U32) 8.0MiB 0.34% 4.0B/row 0.32%
├─ dropoff_latitude 45% resident 11MiB 0.03% 18mB/row
│ ├─ LZ4(U8) ToI64(U8) 5.3MiB 47% 22mB/row 39%
│ ├─ LZ4(U16) Add(U16) 2.5MiB 22% 33mB/row 12%
│ ├─ LZ4(U8) Add(U8) 2.4MiB 21% 23mB/row 16%
│ ├─ LZ4(U16) ToI64(U16) 1.1MiB 9.9% 28mB/row 6.4%
│ └─ 14KiB 0.12% 80μB/row 27%
├─ dropoff_longitude 45% resident 40MiB 0.12% 63mB/row
│ ├─ LZ4(U8) Add(U8) 26MiB 66% 80mB/row 53%
│ ├─ LZ4(U16) Add(U16) 13MiB 33% 0.10B/row 20%
│ └─ 16KiB 0.04% 96μB/row 27%
├─ dropoff_ntacode 45% resident 84MiB 0.26% 0.13B/row
│ ├─ LZ4(U8) Data(1) Data(2) Dict(U8) 84MiB 100% 0.18B/row 73%
│ │ ├─ .0 82MiB
│ │ ├─ .1 0.95MiB
│ │ └─ .2 0.47MiB
│ └─ 14KiB 0.02% 80μB/row 27%
├─ dropoff_ntaname 45% resident 86MiB 0.26% 0.14B/row
│ ├─ LZ4(U8) Data(1) Data(2) Dict(U8) 86MiB 100% 0.19B/row 73%
│ │ ├─ .0 82MiB
│ │ ├─ .1 0.95MiB
│ │ └─ .2 2.3MiB
│ └─ 14KiB 0.02% 80μB/row 27%
├─ dropoff_nyct2010_gid 45% resident 0.13GiB 0.40% 0.21B/row
│ ├─ LZ4(U16) ToI64(U16) 0.13GiB 100% 0.29B/row 73%
│ └─ 16KiB 0.01% 96μB/row 27%
├─ dropoff_puma 45% resident 92MiB 0.28% 0.15B/row
│ ├─ LZ4(U16) ToI64(U16) 91MiB 99% 0.20B/row 73%
│ └─ LZ4(U8) ToI64(U8) 0.83MiB 0.90% 5.0mB/row 27%
├─ ehail_fee 45% resident 3.1MiB 0.01% 5.0mB/row
│ └─ LZ4(U8) ToI64(U8) 3.1MiB 100% 5.0mB/row 100%
├─ extra 45% resident 0.41GiB 1.3% 0.67B/row
│ ├─ LZ4(U16) ToI64(U16) 0.19GiB 46% 0.87B/row 35%
│ ├─ LZ4(U16) Add(U16) 0.18GiB 45% 0.52B/row 58%
│ ├─ LZ4(U32) Add(U32) 20MiB 4.9% 1.1B/row 2.9%
│ ├─ LZ4(U8) ToI64(U8) 16MiB 3.9% 0.62B/row 4.2%
│ └─ LZ4(U8) Add(U8) 0.22MiB 0.05% 0.33B/row 0.11%
├─ fare_amount 45% resident 1.1GiB 3.3% 1.8B/row
│ ├─ LZ4(U32) Add(U32) 0.63GiB 59% 1.8B/row 59%
│ ├─ LZ4(U16) ToI64(U16) 0.41GiB 39% 1.8B/row 39%
│ ├─ LZ4(U16) Add(U16) 20MiB 1.8% 1.8B/row 1.8%
│ ├─ LZ4(U32) ToI64(U32) 3.5MiB 0.32% 1.8B/row 0.32%
│ └─ Add(U16) 0.16MiB 0.01% 2.1B/row 0.01%
├─ improvement_surcharge 45% resident 6.9MiB 0.02% 11mB/row
│ ├─ LZ4(U8) ToI64(U8) 4.0MiB 58% 15mB/row 44%
│ ├─ LZ4(U8) Add(U8) 2.9MiB 41% 8.3mB/row 55%
│ └─ LZ4(U16) Add(U16) 48KiB 0.68% 12mB/row 0.64%
├─ max_temperature 45% resident 0.45GiB 1.4% 0.73B/row
│ ├─ ToI64(U8) 0.45GiB 100% 1.0B/row 73%
│ └─ LZ4(U8) ToI64(U8) 1.1MiB 0.24% 6.7mB/row 27%
├─ min_temperature 45% resident 0.44GiB 1.4% 0.73B/row
│ ├─ ToI64(U8) 0.41GiB 92% 1.0B/row 67%
│ ├─ LZ4(U8) ToI64(U8) 22MiB 4.9% 0.12B/row 31%
│ └─ Add(U8) 12MiB 2.7% 1.0B/row 1.9%
├─ mta_tax 45% resident 15MiB 0.04% 24mB/row
│ ├─ LZ4(U16) Add(U16) 7.6MiB 52% 30mB/row 40%
│ ├─ LZ4(U8) ToI64(U8) 3.7MiB 25% 15mB/row 39%
│ ├─ LZ4(U8) Add(U8) 3.4MiB 23% 26mB/row 21%
│ └─ LZ4(U16) ToI64(U16) 40KiB 0.27% 20mB/row 0.32%
├─ passenger_count 45% resident 0.30GiB 0.95% 0.50B/row
│ └─ LZ4(U8) ToI64(U8) 0.30GiB 100% 0.50B/row 100%
├─ payment_type 45% resident 0.31GiB 0.96% 0.50B/row
│ └─ LZ4(U8) Data(1) Data(2) Dict(U8) 0.31GiB 100% 0.50B/row 100%
│ ├─ .0 0.31GiB
│ ├─ .1 30KiB
│ └─ .2 11KiB
├─ pickup 45% resident 5.9GiB 19% 9.8B/row
│ ├─ LZ4(U8) StrHexUnpack 5.9GiB 100% 13B/row 73%
│ ├─ LZ4(U16) Data(1) Data(2) Dict(U16) 8.2MiB 0.13% 8.2B/row 0.16%
│ │ ├─ .0 0.19MiB
│ │ ├─ .1 1.0MiB
│ │ └─ .2 7.0MiB
│ └─ 12KiB 0.00% 72μB/row 27%
├─ pickup_borocode 45% resident 0.11GiB 0.35% 0.18B/row
│ ├─ LZ4(U8) ToI64(U8) 0.11GiB 100% 0.25B/row 73%
│ └─ 14KiB 0.01% 80μB/row 27%
├─ pickup_boroct2010 45% resident 1.1GiB 3.5% 1.8B/row
│ ├─ LZ4(U32) ToI64(U32) 1.1GiB 100% 2.5B/row 73%
│ └─ 16KiB 0.00% 96μB/row 27%
├─ pickup_boroname 45% resident 0.11GiB 0.35% 0.18B/row
│ ├─ LZ4(U8) Data(1) Data(2) Dict(U8) 0.11GiB 100% 0.25B/row 73%
│ │ ├─ .0 0.11GiB
│ │ ├─ .1 30KiB
│ │ └─ .2 30KiB
│ └─ 14KiB 0.01% 80μB/row 27%
├─ pickup_cdeligibil 45% resident 0.17GiB 0.52% 0.27B/row
│ ├─ LZ4(U8) Data(1) Data(2) Dict(U8) 0.17GiB 100% 0.37B/row 73%
│ │ ├─ .0 0.17GiB
│ │ ├─ .1 15KiB
│ │ └─ .2 3.8KiB
│ └─ 16KiB 0.01% 96μB/row 27%
├─ pickup_ct2010 45% resident 1.1GiB 3.5% 1.8B/row
│ ├─ LZ4(U32) ToI64(U32) 1.1GiB 100% 2.5B/row 73%
│ └─ 14KiB 0.00% 80μB/row 27%
├─ pickup_ctlabel 45% resident 0.89GiB 2.8% 1.5B/row
│ ├─ ToI64(U16) 0.89GiB 100% 2.0B/row 73%
│ ├─ LZ4(U16) ToI64(U16) 0.19MiB 0.02% 0.19B/row 0.16%
│ └─ 14KiB 0.00% 80μB/row 27%
├─ pickup_datetime 45% resident 2.1GiB 6.6% 3.4B/row
│ ├─ ToI64(U32) 1.8GiB 85% 4.0B/row 73%
│ ├─ LZ4(U32) ToI64(U32) 0.32GiB 15% 1.9B/row 27%
│ ├─ LZ4(U16) Add(U16) 0.36MiB 0.02% 1.2B/row 0.05%
│ └─ Add(U16) 0.16MiB 0.01% 2.1B/row 0.01%
├─ pickup_latitude 45% resident 16MiB 0.05% 25mB/row
│ ├─ LZ4(U8) ToI64(U8) 9.4MiB 60% 34mB/row 45%
│ ├─ LZ4(U8) Add(U8) 3.5MiB 22% 33mB/row 17%
│ ├─ LZ4(U16) Add(U16) 2.6MiB 16% 40mB/row 10%
│ ├─ LZ4(U16) ToI64(U16) 0.25MiB 1.6% 35mB/row 1.1%
│ └─ 14KiB 0.08% 80μB/row 27%
├─ pickup_longitude 45% resident 0.17GiB 0.55% 0.29B/row
│ ├─ LZ4(U8) Add(U8) 0.13GiB 74% 0.37B/row 57%
│ ├─ LZ4(U16) Add(U16) 47MiB 26% 0.46B/row 16%
│ └─ 14KiB 0.01% 80μB/row 27%
├─ pickup_ntacode 45% resident 0.45GiB 1.4% 0.73B/row
│ ├─ Data(1) Data(2) Dict(U8) 0.45GiB 100% 1.0B/row 73%
│ │ ├─ .0 0.45GiB
│ │ ├─ .1 0.93MiB
│ │ └─ .2 0.46MiB
│ ├─ LZ4(U8) Data(1) Data(2) Dict(U8) 99KiB 0.02% 97mB/row 0.16%
│ │ ├─ .0 96KiB
│ │ ├─ .1 2.0KiB
│ │ └─ .2 1.0KiB
│ └─ 14KiB 0.00% 80μB/row 27%
├─ pickup_ntaname 45% resident 0.45GiB 1.4% 0.74B/row
│ ├─ Data(1) Data(2) Dict(U8) 0.45GiB 100% 1.0B/row 73%
│ │ ├─ .0 0.45GiB
│ │ ├─ .1 0.93MiB
│ │ └─ .2 2.0MiB
│ └─ LZ4(U8) Data(1) Data(2) Dict(U8) 0.94MiB 0.20% 5.6mB/row 27%
│ ├─ .0 0.91MiB
│ ├─ .1 3.4KiB
│ └─ .2 4.0KiB
├─ pickup_nyct2010_gid 45% resident 0.89GiB 2.8% 1.5B/row
│ ├─ ToI64(U16) 0.89GiB 100% 2.0B/row 73%
│ ├─ LZ4(U16) ToI64(U16) 0.19MiB 0.02% 0.19B/row 0.16%
│ └─ 16KiB 0.00% 96μB/row 27%
├─ pickup_puma 45% resident 0.45GiB 1.4% 0.73B/row
│ ├─ LZ4(U16) ToI64(U16) 0.45GiB 100% 1.00B/row 73%
│ └─ LZ4(U8) ToI64(U8) 0.83MiB 0.18% 5.0mB/row 27%
├─ precipitation 45% resident 0.36GiB 1.1% 0.58B/row
│ ├─ LZ4(U16) ToI64(U16) 0.35GiB 98% 0.66B/row 87%
│ └─ LZ4(U8) ToI64(U8) 6.1MiB 1.7% 76mB/row 13%
├─ rate_code_id 45% resident 63MiB 0.19% 0.10B/row
│ └─ LZ4(U8) ToI64(U8) 63MiB 100% 0.10B/row 100%
├─ snow_depth 45% resident 94MiB 0.29% 0.15B/row
│ ├─ LZ4(U16) ToI64(U16) 92MiB 98% 0.56B/row 27%
│ └─ LZ4(U8) ToI64(U8) 2.3MiB 2.4% 5.0mB/row 73%
├─ snowfall 45% resident 71MiB 0.22% 0.11B/row
│ ├─ LZ4(U16) ToI64(U16) 66MiB 92% 0.42B/row 25%
│ └─ LZ4(U8) ToI64(U8) 5.4MiB 7.6% 12mB/row 75%
├─ store_and_fwd_flag 45% resident 0.16GiB 0.51% 0.27B/row
│ └─ LZ4(U8) Data(1) Data(2) Dict(U8) 0.16GiB 100% 0.27B/row 100%
│ ├─ .0 0.16GiB
│ ├─ .1 14KiB
│ └─ .2 5.1KiB
├─ tip_amount 45% resident 0.88GiB 2.7% 1.4B/row
│ ├─ LZ4(U16) Add(U16) 0.46GiB 52% 1.5B/row 49%
│ ├─ LZ4(U16) ToI64(U16) 0.30GiB 34% 1.3B/row 39%
│ └─ LZ4(U32) Add(U32) 0.12GiB 14% 1.7B/row 11%
├─ tolls_amount 45% resident 0.14GiB 0.45% 0.23B/row
│ ├─ LZ4(U32) Add(U32) 59MiB 40% 0.30B/row 31%
│ ├─ LZ4(U16) ToI64(U16) 47MiB 32% 0.19B/row 41%
│ ├─ LZ4(U16) Add(U16) 38MiB 26% 0.22B/row 27%
│ └─ LZ4(U32) ToI64(U32) 2.2MiB 1.5% 0.28B/row 1.3%
├─ total_amount 45% resident 1.4GiB 4.4% 2.3B/row
│ ├─ LZ4(U32) Add(U32) 0.92GiB 65% 2.5B/row 60%
│ ├─ ToI64(U16) 0.47GiB 33% 2.0B/row 38%
│ ├─ LZ4(U32) ToI64(U32) 13MiB 0.88% 2.5B/row 0.81%
│ └─ Add(U16) 6.3MiB 0.44% 2.0B/row 0.50%
├─ trip_distance 45% resident 1.6GiB 5.0% 2.7B/row
│ ├─ LZ4(U32) ToI64(U32) 1.3GiB 80% 2.5B/row 84%
│ ├─ LZ4(I64) 0.32GiB 20% 3.5B/row 15%
│ ├─ ToI64(U16) 2.7MiB 0.16% 2.1B/row 0.20%
│ └─ LZ4(U32) Add(U32) 2.5MiB 0.15% 2.5B/row 0.16%
├─ trip_id 45% resident 26MiB 0.08% 41mB/row
│ ├─ LZ4(U32) Add(U32) Delta(I64) 26MiB 100% 41mB/row 100%
│ └─ LZ4(U32) Delta(U32) 1.6KiB 0.01% 21mB/row 0.01%
├─ trip_type 45% resident 51KiB 0.00% 81μB/row
│ └─ 51KiB 100% 81μB/row 100%
└─ vendor_id 45% resident 0.30GiB 0.94% 0.50B/row
└─ LZ4(U8) Data(1) Data(2) Dict(U8) 0.30GiB 100% 0.50B/row 100%
├─ .0 0.30GiB
├─ .1 10KiB
└─ .2 5.1KiB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment