Instantly share code, notes, and snippets.

Embed
What would you like to do?
➜ ~ uname -a
Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
➜ ~ sudo lshw -class disk -class storage
[sudo] password for clemens:
*-storage
description: SATA controller
product: ASM1062 Serial ATA Controller
vendor: ASMedia Technology Inc.
physical id: 0
bus info: pci@0000:06:00.0
version: 02
width: 32 bits
clock: 33MHz
capabilities: storage msi pm pciexpress ahci_1.0 bus_master cap_list
configuration: driver=ahci latency=0
resources: irq:43 ioport:d050(size=8) ioport:d040(size=4) ioport:d030(size=8) ioport:d020(size=4) ioport:d000(size=32) memory:fb200000-fb2001ff
*-storage
description: SATA controller
product: C610/X99 series chipset 6-Port SATA Controller [AHCI mode]
vendor: Intel Corporation
physical id: 1f.2
bus info: pci@0000:00:1f.2
version: 05
width: 32 bits
clock: 66MHz
capabilities: storage msi pm ahci_1.0 bus_master cap_list
configuration: driver=ahci latency=0
resources: irq:42 ioport:f070(size=8) ioport:f060(size=4) ioport:f050(size=8) ioport:f040(size=4) ioport:f000(size=32) memory:fb336000-fb3367ff
*-scsi:0
physical id: 1
logical name: scsi0
capabilities: emulated
*-disk
description: ATA Disk
product: Crucial_CT512MX1
physical id: 0.0.0
bus info: scsi@0:0.0.0
logical name: /dev/sda
version: MU01
serial: 14340CFD629E
size: 476GiB (512GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=75bae818-9153-47eb-9cee-c01f70c5c42b logicalsectorsize=512 sectorsize=4096
*-scsi:1
physical id: 2
logical name: scsi1
capabilities: emulated
*-disk
description: ATA Disk
product: Crucial_CT512MX1
physical id: 0.0.0
bus info: scsi@1:0.0.0
logical name: /dev/sdb
version: MU01
serial: 14330CF93238
size: 476GiB (512GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=b3e05cb9-ac11-46b2-a41b-5cf4164bcb34 logicalsectorsize=512 sectorsize=4096
*-scsi:2
physical id: 3
logical name: scsi2
capabilities: emulated
*-disk
description: ATA Disk
product: ST2000DM006-2DM1
vendor: Seagate
physical id: 0.0.0
bus info: scsi@2:0.0.0
logical name: /dev/sdc
version: CC26
serial: Z4Z83385
size: 1863GiB (2TB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=86fe1a86-58ce-448b-bb84-1461f00ab21f logicalsectorsize=512 sectorsize=4096
*-scsi:3
physical id: 4
logical name: scsi3
capabilities: emulated
*-disk
description: ATA Disk
product: ST2000DM006-2DM1
vendor: Seagate
physical id: 0.0.0
bus info: scsi@3:0.0.0
logical name: /dev/sdd
version: CC26
serial: Z4ZB7EV9
size: 1863GiB (2TB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=fa606d24-f1b6-4661-ab08-41e3bae03746 logicalsectorsize=512 sectorsize=4096
➜ ~ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 477G 0 disk
├─sda1 8:1 0 512M 0 part
├─sda2 8:2 0 1M 0 part
├─sda3 8:3 0 15.9G 0 part [SWAP]
└─sda4 8:4 0 460.6G 0 part /
sdb 8:16 0 477G 0 disk
└─sdb1 8:17 0 477G 0 part
sdc 8:32 0 1.8T 0 disk
├─sdc1 8:33 0 128M 0 part
└─sdc2 8:34 0 1.8T 0 part
sdd 8:48 0 1.8T 0 disk
└─sdd1 8:49 0 1.8T 0 part /mnt/data
➜ ~ cd src/LocustDB
➜ LocustDB git:(master) ulimit -n 4096
➜ LocustDB git:(master) dstat
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
23 1 65 12 0 0| 28M 47M| 0 0 | 0 0 |3344 1341
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 42 91
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 23 52
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 28 68
0 0 100 0 0 0| 0 0 |2108B 0 | 0 0 | 23 93 ^C
➜ LocustDB git:(master) git rev-parse HEAD
5fc5d537abffd8172554a4b5a205b7b62c7c502e
➜ LocustDB git:(master) RUSTFLAGS="-Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --load /mnt/data/nyc-taxi-data-denormalized/trips_x* --trips --db-path /home/clemens/ldb-20 --table trips --partition-size=1048576
Compiling cfg-if v0.1.2
Compiling nodrop v0.1.12
Compiling lazy_static v1.0.0
Compiling memoffset v0.2.1
Compiling scopeguard v0.3.3
Compiling rayon-core v1.4.1
Compiling libc v0.2.39
Compiling rayon v1.0.2
Compiling either v1.4.0
Compiling void v1.0.2
Compiling glob v0.2.11
Compiling ucd-util v0.1.1
Compiling regex v0.2.10
Compiling regex v1.0.2
Compiling quick-error v1.2.1
Compiling unicode-xid v0.1.0
Compiling unicode-width v0.1.4
Compiling utf8-ranges v1.0.0
Compiling termcolor v0.3.6
Compiling vec_map v0.8.1
Compiling bitflags v1.0.1
Compiling ansi_term v0.11.0
Compiling strsim v0.7.0
Compiling bindgen v0.37.4
Compiling getopts v0.2.17
Compiling unicode-xid v0.0.4
Compiling bitflags v0.5.0
Compiling remove_dir_all v0.5.1
Compiling num-traits v0.2.1
Compiling peeking_take_while v0.1.2
Compiling make-cmd v0.1.0
Compiling quote v0.3.15
Compiling byteorder v1.2.3
Compiling gcc v0.3.54
Compiling rustc-serialize v0.3.24
Compiling serde v1.0.70
Compiling rustc-demangle v0.1.7
Compiling heapsize v0.4.2
Compiling bitflags v0.4.0
Compiling encode_unicode v0.1.3
Compiling seahash v3.0.5
Compiling hex v0.3.2
Compiling fnv v1.0.6
Compiling lru v0.1.7
Compiling scoped_threadpool v0.1.9
Compiling bit-vec v0.4.4
Compiling crossbeam-utils v0.2.2
Compiling log v0.4.1
Compiling arrayvec v0.4.7
Compiling unreachable v1.0.0
Compiling humantime v1.1.1
Compiling textwrap v0.10.0
Compiling proc-macro2 v0.3.5
Compiling futures-core v0.2.1
Compiling itertools v0.5.10
Compiling regex-syntax v0.5.3
Compiling regex-syntax v0.6.2
Compiling synom v0.11.3
Compiling num_cpus v1.8.0
Compiling memchr v2.0.1
Compiling memchr v1.0.2
Compiling atty v0.2.8
Compiling rand v0.4.2
Compiling which v1.0.5
Compiling iovec v0.1.2
Compiling time v0.1.39
Compiling capnp v0.8.17
Compiling nix v0.5.1
Compiling clang-sys v0.23.0
Compiling thread_local v0.3.5
Compiling num-integer v0.1.36
Compiling pulldown-cmark v0.0.8
Compiling crossbeam-epoch v0.3.1
Compiling futures-channel v0.2.1
Compiling syn v0.11.11
Compiling aho-corasick v0.6.4
Compiling csv-core v0.1.4
Compiling nom v3.2.1
Compiling clap v2.32.0
Compiling futures-io v0.2.1
Compiling num-iter v0.1.35
Compiling futures-sink v0.2.1
Compiling tempdir v0.3.7
Compiling rand v0.3.22
Compiling crossbeam-deque v0.2.0
Compiling lz4-sys v1.8.0
Compiling rust-crypto v0.2.36
Compiling quote v0.5.2
Compiling capnpc v0.8.9
Compiling futures-util v0.2.1
Compiling rustyline v1.0.0
Compiling num-bigint v0.1.43
Compiling num-complex v0.1.43
Compiling skeptic v0.9.0
Compiling futures-executor v0.2.1
Compiling cexpr v0.2.3
Compiling num-rational v0.1.42
Compiling lz4 v1.22.0
Compiling cc v1.0.17
Compiling csv v1.0.0
Compiling synstructure v0.6.1
Compiling synstructure v0.5.2
Compiling num v0.1.42
Compiling chrono v0.4.0
Compiling heapsize_derive v0.1.4
Compiling failure_derive v0.1.1
Compiling libloading v0.5.0
Compiling backtrace-sys v0.1.16
Compiling miniz-sys v0.1.10
Compiling locustdb v0.1.0-alpha (file:///home/clemens/src/LocustDB)
Compiling flate2 v1.0.1
Compiling backtrace v0.3.5
Compiling failure v0.1.1
Compiling env_logger v0.5.6
Compiling librocksdb-sys v5.14.2
warning: redundant linker flag specified for library `stdc++`
Compiling rocksdb v0.10.1
Finished release [optimized] target(s) in 3m 26s
Running `target/release/repl --load /mnt/data/nyc-taxi-data-denormalized/trips_xaa.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xab.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xac.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xad.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xae.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xag.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xah.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xai.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xak.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xal.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xam.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xan.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xao.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xap.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xar.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xas.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xat.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xau.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xav.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaw.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xax.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xay.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaz.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xba.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbb.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbc.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbd.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbe.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbg.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbh.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbi.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbk.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbl.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbm.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbn.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbo.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbp.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbr.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbs.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbt.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbu.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbv.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbw.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbx.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xby.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbz.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xca.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcb.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcc.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcd.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xce.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcg.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xch.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xci.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xck.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcl.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcm.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcn.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xco.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcp.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcr.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcs.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xct.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcu.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcv.csv.gz --trips --db-path /home/clemens/ldb-20 --table trips --par
Loading 74 files into table trips.
Loaded data in 5.34ks.
# Table `trips` (1464785771 rows, 8.02GiB) #
min_temperature: 95MiB
dropoff_datetime: 0.89GiB
passenger_count: 0.13GiB
precipitation: 75MiB
trip_type: 22KiB
total_amount: 0.64GiB
pickup_ntacode: 94MiB
ehail_fee: 1.3MiB
pickup_boroname: 23MiB
trip_id: 7.5MiB
store_and_fwd_flag: 9.5MiB
dropoff_nyct2010_gid: 23MiB
dropoff_latitude: 1.8MiB
dropoff_ct2010: 30MiB
dropoff_ntacode: 15MiB
pickup_ct2010: 0.23GiB
extra: 0.10GiB
cab_type: 1.3MiB
pickup_longitude: 34MiB
dropoff_ntaname: 16MiB
pickup: 0.93GiB
dropoff_puma: 18MiB
rate_code_id: 32MiB
pickup_datetime: 0.68GiB
dropoff: 0.93GiB
pickup_cdeligibil: 34MiB
dropoff_ctlabel: 23MiB
dropoff_boroname: 7.4MiB
pickup_nyct2010_gid: 0.18GiB
pickup_boroct2010: 0.23GiB
tip_amount: 0.40GiB
dropoff_boroct2010: 34MiB
pickup_ntaname: 96MiB
snowfall: 15MiB
max_temperature: 95MiB
snow_depth: 15MiB
pickup_puma: 95MiB
tolls_amount: 73MiB
average_wind_speed: 0.13GiB
dropoff_borocode: 7.4MiB
improvement_surcharge: 2.7MiB
payment_type: 0.13GiB
fare_amount: 0.45GiB
pickup_ctlabel: 0.18GiB
dropoff_cdeligibil: 8.9MiB
trip_distance: 0.67GiB
pickup_borocode: 23MiB
dropoff_longitude: 7.0MiB
vendor_id: 0.12GiB
mta_tax: 8.2MiB
pickup_latitude: 2.9MiB
# Table `_meta_tables` (1 rows, 8.00B) #
name: 0.00B
timestamp: 0.00B
locustdb> SELECT passenger_count, to_year(pickup_datetime), trip_distance / 1000, count(0) FROM trips;
Scanned 1.46 billion rows in 80.7s (18 million rows/s)!
passenger_count | col_0 | col_1 | count_0
----------------+-------+-------+---------
0 | 2009 | 0 | 570
0 | 2009 | 1 | 267
0 | 2009 | 2 | 115
0 | 2009 | 3 | 52
0 | 2009 | 4 | 28
0 | 2009 | 5 | 28
0 | 2009 | 6 | 13
0 | 2009 | 7 | 11
0 | 2009 | 8 | 8
0 | 2009 | 9 | 5
0 | 2009 | 10 | 4
0 | 2009 | 11 | 6
0 | 2009 | 12 | 2
0 | 2009 | 13 | 2
0 | 2009 | 14 | 2
0 | 2009 | 17 | 3
0 | 2009 | 18 | 2
0 | 2009 | 19 | 2
0 | 2009 | 20 | 1
0 | 2009 | 21 | 2
0 | 2009 | 22 | 2
0 | 2009 | 31 | 1
0 | 2010 | 0 | 16834
0 | 2010 | 1 | 10264
0 | 2010 | 2 | 3858
0 | 2010 | 3 | 1797
0 | 2010 | 4 | 831
0 | 2010 | 5 | 643
0 | 2010 | 6 | 418
0 | 2010 | 7 | 207
0 | 2010 | 8 | 108
0 | 2010 | 9 | 100
0 | 2010 | 10 | 109
0 | 2010 | 11 | 118
0 | 2010 | 12 | 58
0 | 2010 | 13 | 30
0 | 2010 | 14 | 20
0 | 2010 | 15 | 17
0 | 2010 | 16 | 13
0 | 2010 | 17 | 10
0 | 2010 | 18 | 3
0 | 2010 | 19 | 8
0 | 2010 | 20 | 8
0 | 2010 | 21 | 5
0 | 2010 | 22 | 2
0 | 2010 | 23 | 3
0 | 2010 | 24 | 2
0 | 2010 | 26 | 3
0 | 2010 | 27 | 1
0 | 2010 | 28 | 1
0 | 2010 | 29 | 1
0 | 2010 | 32 | 1
0 | 2010 | 33 | 1
0 | 2010 | 36 | 2
0 | 2010 | 37 | 2
0 | 2010 | 38 | 1
0 | 2010 | 43 | 1
0 | 2010 | 47 | 1
0 | 2010 | 53 | 1
0 | 2010 | 54 | 1
0 | 2010 | 70 | 1
0 | 2010 | 76 | 1
0 | 2010 | 77 | 1
0 | 2010 | 86 | 1
0 | 2011 | 0 | 679047
0 | 2011 | 1 | 884997
0 | 2011 | 2 | 430866
0 | 2011 | 3 | 211773
0 | 2011 | 4 | 112552
0 | 2011 | 5 | 69700
0 | 2011 | 6 | 43498
0 | 2011 | 7 | 26703
0 | 2011 | 8 | 16962
0 | 2011 | 9 | 13569
0 | 2011 | 10 | 10974
0 | 2011 | 11 | 8656
0 | 2011 | 12 | 6488
0 | 2011 | 13 | 4422
0 | 2011 | 14 | 3534
0 | 2011 | 15 | 3470
0 | 2011 | 16 | 4541
0 | 2011 | 17 | 4617
0 | 2011 | 18 | 3805
0 | 2011 | 19 | 2574
0 | 2011 | 20 | 1710
0 | 2011 | 21 | 965
0 | 2011 | 22 | 581
0 | 2011 | 23 | 340
0 | 2011 | 24 | 214
0 | 2011 | 25 | 189
0 | 2011 | 26 | 153
0 | 2011 | 27 | 138
0 | 2011 | 28 | 114
0 | 2011 | 29 | 63
0 | 2011 | 30 | 73
0 | 2011 | 31 | 58
0 | 2011 | 32 | 72
0 | 2011 | 33 | 39
0 | 2011 | 34 | 46
0 | 2011 | 35 | 39
locustdb> :memtree
trips 8.00GiB 1464785771 rows
├─ average_wind_speed nonresident
├─ cab_type nonresident
├─ dropoff nonresident
├─ dropoff_borocode nonresident
├─ dropoff_boroct2010 nonresident
├─ dropoff_boroname nonresident
├─ dropoff_cdeligibil nonresident
├─ dropoff_ct2010 nonresident
├─ dropoff_ctlabel nonresident
├─ dropoff_datetime nonresident
├─ dropoff_latitude nonresident
├─ dropoff_longitude nonresident
├─ dropoff_ntacode nonresident
├─ dropoff_ntaname nonresident
├─ dropoff_nyct2010_gid nonresident
├─ dropoff_puma nonresident
├─ ehail_fee nonresident
├─ extra nonresident
├─ fare_amount nonresident
├─ improvement_surcharge nonresident
├─ max_temperature nonresident
├─ min_temperature nonresident
├─ mta_tax nonresident
├─ passenger_count 65% resident 0.89GiB 11% 1.0B/row
├─ payment_type nonresident
├─ pickup nonresident
├─ pickup_borocode nonresident
├─ pickup_boroct2010 nonresident
├─ pickup_boroname nonresident
├─ pickup_cdeligibil nonresident
├─ pickup_ct2010 nonresident
├─ pickup_ctlabel nonresident
├─ pickup_datetime 65% resident 3.5GiB 44% 4.0B/row
├─ pickup_latitude nonresident
├─ pickup_longitude nonresident
├─ pickup_ntacode nonresident
├─ pickup_ntaname nonresident
├─ pickup_nyct2010_gid nonresident
├─ pickup_puma nonresident
├─ precipitation nonresident
├─ rate_code_id nonresident
├─ snow_depth nonresident
├─ snowfall nonresident
├─ store_and_fwd_flag nonresident
├─ tip_amount nonresident
├─ tolls_amount nonresident
├─ total_amount nonresident
├─ trip_distance 65% resident 3.6GiB 45% 4.0B/row
├─ trip_id nonresident
├─ trip_type nonresident
└─ vendor_id nonresident
_meta_tables 0.000B 1 rows
├─ name nonresident
└─ timestamp nonresident
locustdb>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment