Instantly share code, notes, and snippets.

Embed
What would you like to do?
➜ ~ uname -a
Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
➜ ~ sudo lshw -class disk -class storage
[sudo] password for clemens:
*-storage
description: SATA controller
product: ASM1062 Serial ATA Controller
vendor: ASMedia Technology Inc.
physical id: 0
bus info: pci@0000:06:00.0
version: 02
width: 32 bits
clock: 33MHz
capabilities: storage msi pm pciexpress ahci_1.0 bus_master cap_list
configuration: driver=ahci latency=0
resources: irq:43 ioport:d050(size=8) ioport:d040(size=4) ioport:d030(size=8) ioport:d020(size=4) ioport:d000(size=32) memory:fb200000-fb2001ff
*-storage
description: SATA controller
product: C610/X99 series chipset 6-Port SATA Controller [AHCI mode]
vendor: Intel Corporation
physical id: 1f.2
bus info: pci@0000:00:1f.2
version: 05
width: 32 bits
clock: 66MHz
capabilities: storage msi pm ahci_1.0 bus_master cap_list
configuration: driver=ahci latency=0
resources: irq:42 ioport:f070(size=8) ioport:f060(size=4) ioport:f050(size=8) ioport:f040(size=4) ioport:f000(size=32) memory:fb336000-fb3367ff
*-scsi:0
physical id: 1
logical name: scsi0
capabilities: emulated
*-disk
description: ATA Disk
product: Crucial_CT512MX1
physical id: 0.0.0
bus info: scsi@0:0.0.0
logical name: /dev/sda
version: MU01
serial: 14340CFD629E
size: 476GiB (512GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=75bae818-9153-47eb-9cee-c01f70c5c42b logicalsectorsize=512 sectorsize=4096
*-scsi:1
physical id: 2
logical name: scsi1
capabilities: emulated
*-disk
description: ATA Disk
product: Crucial_CT512MX1
physical id: 0.0.0
bus info: scsi@1:0.0.0
logical name: /dev/sdb
version: MU01
serial: 14330CF93238
size: 476GiB (512GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=b3e05cb9-ac11-46b2-a41b-5cf4164bcb34 logicalsectorsize=512 sectorsize=4096
*-scsi:2
physical id: 3
logical name: scsi2
capabilities: emulated
*-disk
description: ATA Disk
product: ST2000DM006-2DM1
vendor: Seagate
physical id: 0.0.0
bus info: scsi@2:0.0.0
logical name: /dev/sdc
version: CC26
serial: Z4Z83385
size: 1863GiB (2TB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=86fe1a86-58ce-448b-bb84-1461f00ab21f logicalsectorsize=512 sectorsize=4096
*-scsi:3
physical id: 4
logical name: scsi3
capabilities: emulated
*-disk
description: ATA Disk
product: ST2000DM006-2DM1
vendor: Seagate
physical id: 0.0.0
bus info: scsi@3:0.0.0
logical name: /dev/sdd
version: CC26
serial: Z4ZB7EV9
size: 1863GiB (2TB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=fa606d24-f1b6-4661-ab08-41e3bae03746 logicalsectorsize=512 sectorsize=4096
➜ ~ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 477G 0 disk
├─sda1 8:1 0 512M 0 part
├─sda2 8:2 0 1M 0 part
├─sda3 8:3 0 15.9G 0 part [SWAP]
└─sda4 8:4 0 460.6G 0 part /
sdb 8:16 0 477G 0 disk
└─sdb1 8:17 0 477G 0 part
sdc 8:32 0 1.8T 0 disk
├─sdc1 8:33 0 128M 0 part
└─sdc2 8:34 0 1.8T 0 part
sdd 8:48 0 1.8T 0 disk
└─sdd1 8:49 0 1.8T 0 part /mnt/data
➜ ~ ulimit -n 4096
➜ ~ dstat
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
3 1 96 1 0 0|8313k 541k| 0 0 | 0 0 |2283 14k
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 36 103
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 37 76
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 28 65
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 27 56
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 26 56
0 0 100 0 0 0| 0 36k| 309B 0 | 0 0 | 40 85
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 23 45
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 24 59
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 24 50
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 103 368
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 61 283 ^C
➜ ~ cd src/LocustDB
➜ LocustDB git:(master) git rev-parse HEAD
573557575140c94e0aa3a0576ece04beb718712c
➜ LocustDB git:(master) du -h /home/clemens/ldb
80G /home/clemens/ldb
➜ LocustDB git:(master) du /home/clemens/ldb
83738212 /home/clemens/ldb
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb --mem-limit-tables=45 --mem-lz4
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.44s
Running `target/release/repl --db-path /home/clemens/ldb --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 192KiB) #
dropoff_puma: 0.00B
pickup_ntaname: 0.00B
mta_tax: 0.00B
pickup_ntacode: 0.00B
snow_depth: 0.00B
pickup_nyct2010_gid: 0.00B
payment_type: 0.00B
tip_amount: 0.00B
pickup_borocode: 0.00B
pickup_boroname: 0.00B
vendor_id: 0.00B
dropoff_latitude: 0.00B
dropoff_nyct2010_gid: 0.00B
pickup_longitude: 0.00B
passenger_count: 0.00B
pickup: 0.00B
dropoff_boroct2010: 0.00B
dropoff_ntacode: 0.00B
pickup_boroct2010: 0.00B
dropoff_cdeligibil: 0.00B
store_and_fwd_flag: 0.00B
pickup_puma: 0.00B
precipitation: 0.00B
fare_amount: 0.00B
dropoff_longitude: 0.00B
dropoff: 0.00B
total_amount: 0.00B
pickup_latitude: 0.00B
pickup_ct2010: 0.00B
dropoff_boroname: 0.00B
pickup_ctlabel: 0.00B
improvement_surcharge: 0.00B
trip_id: 0.00B
average_wind_speed: 0.00B
trip_type: 0.00B
rate_code_id: 0.00B
pickup_datetime: 0.00B
ehail_fee: 0.00B
snowfall: 0.00B
tolls_amount: 0.00B
cab_type: 0.00B
dropoff_ntaname: 0.00B
dropoff_datetime: 0.00B
extra: 0.00B
dropoff_ctlabel: 0.00B
dropoff_ct2010: 0.00B
max_temperature: 0.00B
dropoff_borocode: 0.00B
trip_distance: 0.00B
min_temperature: 0.00B
pickup_cdeligibil: 0.00B
locustdb> :restore
Restored DB from disk in 164s
trips 45.2GiB 1464785771 rows
├─ average_wind_speed 58% resident 0.86GiB 1.9% 1.1B/row
├─ cab_type fully resident 12MiB 0.03% 8.6mB/row
├─ dropoff 54% resident 8.0GiB 18% 11B/row
├─ dropoff_borocode fully resident 0.13GiB 0.28% 93mB/row
├─ dropoff_boroct2010 74% resident 0.35GiB 0.78% 0.35B/row
├─ dropoff_boroname fully resident 0.13GiB 0.29% 96mB/row
├─ dropoff_cdeligibil fully resident 0.16GiB 0.34% 0.11B/row
├─ dropoff_ct2010 73% resident 0.33GiB 0.73% 0.33B/row
├─ dropoff_ctlabel 57% resident 0.18GiB 0.40% 0.23B/row
├─ dropoff_datetime 55% resident 2.9GiB 6.4% 3.8B/row
├─ dropoff_latitude fully resident 36MiB 0.08% 26mB/row
├─ dropoff_longitude fully resident 0.11GiB 0.25% 83mB/row
├─ dropoff_ntacode fully resident 0.31GiB 0.68% 0.23B/row
├─ dropoff_ntaname fully resident 0.36GiB 0.80% 0.26B/row
├─ dropoff_nyct2010_gid 57% resident 0.19GiB 0.41% 0.24B/row
├─ dropoff_puma fully resident 0.30GiB 0.66% 0.22B/row
├─ ehail_fee fully resident 9.9MiB 0.02% 7.1mB/row
├─ extra 69% resident 0.54GiB 1.2% 0.58B/row
├─ fare_amount 59% resident 1.4GiB 3.1% 1.8B/row
├─ improvement_surcharge fully resident 19MiB 0.04% 14mB/row
├─ max_temperature 62% resident 0.69GiB 1.5% 0.81B/row
├─ min_temperature 62% resident 0.68GiB 1.5% 0.81B/row
├─ mta_tax fully resident 42MiB 0.09% 30mB/row
├─ passenger_count 69% resident 0.53GiB 1.2% 0.56B/row
├─ payment_type 68% resident 0.50GiB 1.1% 0.54B/row
├─ pickup 54% resident 7.9GiB 18% 11B/row
├─ pickup_borocode fully resident 0.28GiB 0.63% 0.21B/row
├─ pickup_boroct2010 57% resident 1.5GiB 3.4% 2.0B/row
├─ pickup_boroname fully resident 0.29GiB 0.64% 0.21B/row
├─ pickup_cdeligibil fully resident 0.43GiB 0.95% 0.32B/row
├─ pickup_ct2010 58% resident 1.6GiB 3.5% 2.0B/row
├─ pickup_ctlabel 60% resident 1.3GiB 2.9% 1.6B/row
├─ pickup_datetime 55% resident 2.6GiB 5.8% 3.5B/row
├─ pickup_latitude fully resident 44MiB 0.09% 31mB/row
├─ pickup_longitude 57% resident 0.24GiB 0.53% 0.31B/row
├─ pickup_ntacode 61% resident 0.70GiB 1.5% 0.83B/row
├─ pickup_ntaname 63% resident 0.74GiB 1.6% 0.86B/row
├─ pickup_nyct2010_gid 60% resident 1.3GiB 2.9% 1.6B/row
├─ pickup_puma 60% resident 0.65GiB 1.4% 0.80B/row
├─ precipitation 60% resident 0.54GiB 1.2% 0.65B/row
├─ rate_code_id fully resident 0.12GiB 0.26% 86mB/row
├─ snow_depth fully resident 0.21GiB 0.46% 0.15B/row
├─ snowfall fully resident 0.15GiB 0.34% 0.11B/row
├─ store_and_fwd_flag 65% resident 0.34GiB 0.75% 0.38B/row
├─ tip_amount 64% resident 1.2GiB 2.6% 1.4B/row
├─ tolls_amount fully resident 0.25GiB 0.56% 0.19B/row
├─ total_amount 55% resident 1.5GiB 3.4% 2.1B/row
├─ trip_distance 58% resident 1.9GiB 4.1% 2.3B/row
├─ trip_id fully resident 65MiB 0.14% 47mB/row
├─ trip_type fully resident 8.8MiB 0.02% 6.3mB/row
└─ vendor_id 68% resident 0.50GiB 1.1% 0.54B/row
locustdb>
➜ LocustDB git:(master) du -h /mnt/data/ldb
80G /mnt/data/ldb
➜ LocustDB git:(master) du /mnt/data/ldb
83736088 /mnt/data/ldb
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.28s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
^C
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.35s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
cab_type: 0.00B
dropoff_longitude: 0.00B
pickup_ntacode: 0.00B
pickup_nyct2010_gid: 0.00B
tip_amount: 0.00B
snow_depth: 0.00B
improvement_surcharge: 0.00B
dropoff_datetime: 0.00B
pickup_ct2010: 0.00B
pickup_ntaname: 0.00B
pickup_boroct2010: 0.00B
min_temperature: 0.00B
pickup_boroname: 0.00B
dropoff_latitude: 0.00B
dropoff_borocode: 0.00B
store_and_fwd_flag: 0.00B
pickup_cdeligibil: 0.00B
pickup: 0.00B
pickup_ctlabel: 0.00B
pickup_datetime: 0.00B
average_wind_speed: 0.00B
dropoff_ntacode: 0.00B
dropoff_boroct2010: 0.00B
pickup_puma: 0.00B
ehail_fee: 0.00B
dropoff_boroname: 0.00B
pickup_longitude: 0.00B
payment_type: 0.00B
mta_tax: 0.00B
passenger_count: 0.00B
pickup_latitude: 0.00B
dropoff: 0.00B
dropoff_ctlabel: 0.00B
fare_amount: 0.00B
dropoff_ct2010: 0.00B
precipitation: 0.00B
trip_type: 0.00B
trip_id: 0.00B
trip_distance: 0.00B
dropoff_nyct2010_gid: 0.00B
dropoff_ntaname: 0.00B
pickup_borocode: 0.00B
snowfall: 0.00B
dropoff_cdeligibil: 0.00B
extra: 0.00B
max_temperature: 0.00B
vendor_id: 0.00B
tolls_amount: 0.00B
rate_code_id: 0.00B
dropoff_puma: 0.00B
total_amount: 0.00B
locustdb> :restore
Restored DB from disk in 625s
trips 45.1GiB 1464785771 rows
├─ average_wind_speed 58% resident 0.86GiB 1.9% 1.1B/row
├─ cab_type fully resident 12MiB 0.03% 8.6mB/row
├─ dropoff 53% resident 7.7GiB 17% 11B/row
├─ dropoff_borocode fully resident 0.13GiB 0.28% 93mB/row
├─ dropoff_boroct2010 74% resident 0.35GiB 0.78% 0.35B/row
├─ dropoff_boroname fully resident 0.13GiB 0.29% 96mB/row
├─ dropoff_cdeligibil fully resident 0.16GiB 0.35% 0.11B/row
├─ dropoff_ct2010 73% resident 0.33GiB 0.73% 0.33B/row
├─ dropoff_ctlabel 57% resident 0.18GiB 0.40% 0.23B/row
├─ dropoff_datetime 54% resident 2.8GiB 6.3% 3.8B/row
├─ dropoff_latitude fully resident 36MiB 0.08% 26mB/row
├─ dropoff_longitude fully resident 0.11GiB 0.25% 83mB/row
├─ dropoff_ntacode fully resident 0.31GiB 0.68% 0.23B/row
├─ dropoff_ntaname fully resident 0.36GiB 0.80% 0.26B/row
├─ dropoff_nyct2010_gid 57% resident 0.19GiB 0.41% 0.24B/row
├─ dropoff_puma fully resident 0.30GiB 0.67% 0.22B/row
├─ ehail_fee fully resident 9.9MiB 0.02% 7.1mB/row
├─ extra 69% resident 0.54GiB 1.2% 0.58B/row
├─ fare_amount 59% resident 1.4GiB 3.2% 1.8B/row
├─ improvement_surcharge fully resident 19MiB 0.04% 14mB/row
├─ max_temperature 62% resident 0.69GiB 1.5% 0.81B/row
├─ min_temperature 62% resident 0.68GiB 1.5% 0.81B/row
├─ mta_tax fully resident 42MiB 0.09% 30mB/row
├─ passenger_count 69% resident 0.53GiB 1.2% 0.56B/row
├─ payment_type 68% resident 0.50GiB 1.1% 0.54B/row
├─ pickup 54% resident 7.9GiB 18% 11B/row
├─ pickup_borocode fully resident 0.28GiB 0.63% 0.21B/row
├─ pickup_boroct2010 60% resident 1.6GiB 3.6% 2.0B/row
├─ pickup_boroname fully resident 0.29GiB 0.64% 0.21B/row
├─ pickup_cdeligibil fully resident 0.43GiB 0.96% 0.32B/row
├─ pickup_ct2010 60% resident 1.6GiB 3.6% 2.0B/row
├─ pickup_ctlabel 60% resident 1.3GiB 2.9% 1.6B/row
├─ pickup_datetime 54% resident 2.6GiB 5.7% 3.5B/row
├─ pickup_latitude fully resident 44MiB 0.09% 31mB/row
├─ pickup_longitude fully resident 0.46GiB 1.0% 0.34B/row
├─ pickup_ntacode 61% resident 0.70GiB 1.5% 0.83B/row
├─ pickup_ntaname 63% resident 0.74GiB 1.6% 0.86B/row
├─ pickup_nyct2010_gid 60% resident 1.3GiB 2.9% 1.6B/row
├─ pickup_puma 60% resident 0.65GiB 1.5% 0.80B/row
├─ precipitation 55% resident 0.47GiB 1.0% 0.63B/row
├─ rate_code_id fully resident 0.12GiB 0.26% 86mB/row
├─ snow_depth fully resident 0.21GiB 0.46% 0.15B/row
├─ snowfall fully resident 0.15GiB 0.34% 0.11B/row
├─ store_and_fwd_flag 65% resident 0.34GiB 0.75% 0.38B/row
├─ tip_amount 64% resident 1.2GiB 2.6% 1.4B/row
├─ tolls_amount fully resident 0.25GiB 0.56% 0.19B/row
├─ total_amount 55% resident 1.5GiB 3.4% 2.1B/row
├─ trip_distance 58% resident 1.9GiB 4.1% 2.3B/row
├─ trip_id fully resident 65MiB 0.14% 47mB/row
├─ trip_type fully resident 8.8MiB 0.02% 6.3mB/row
└─ vendor_id 68% resident 0.50GiB 1.1% 0.54B/row
locustdb>
➜ LocustDB git:(master)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment