➜ ~ uname -a | |
Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | |
➜ ~ sudo lshw -class disk -class storage | |
[sudo] password for clemens: | |
*-storage | |
description: SATA controller | |
product: ASM1062 Serial ATA Controller | |
vendor: ASMedia Technology Inc. | |
physical id: 0 | |
bus info: pci@0000:06:00.0 | |
version: 02 | |
width: 32 bits | |
clock: 33MHz | |
capabilities: storage msi pm pciexpress ahci_1.0 bus_master cap_list | |
configuration: driver=ahci latency=0 | |
resources: irq:43 ioport:d050(size=8) ioport:d040(size=4) ioport:d030(size=8) ioport:d020(size=4) ioport:d000(size=32) memory:fb200000-fb2001ff | |
*-storage | |
description: SATA controller | |
product: C610/X99 series chipset 6-Port SATA Controller [AHCI mode] | |
vendor: Intel Corporation | |
physical id: 1f.2 | |
bus info: pci@0000:00:1f.2 | |
version: 05 | |
width: 32 bits | |
clock: 66MHz | |
capabilities: storage msi pm ahci_1.0 bus_master cap_list | |
configuration: driver=ahci latency=0 | |
resources: irq:42 ioport:f070(size=8) ioport:f060(size=4) ioport:f050(size=8) ioport:f040(size=4) ioport:f000(size=32) memory:fb336000-fb3367ff | |
*-scsi:0 | |
physical id: 1 | |
logical name: scsi0 | |
capabilities: emulated | |
*-disk | |
description: ATA Disk | |
product: Crucial_CT512MX1 | |
physical id: 0.0.0 | |
bus info: scsi@0:0.0.0 | |
logical name: /dev/sda | |
version: MU01 | |
serial: 14340CFD629E | |
size: 476GiB (512GB) | |
capabilities: gpt-1.00 partitioned partitioned:gpt | |
configuration: ansiversion=5 guid=75bae818-9153-47eb-9cee-c01f70c5c42b logicalsectorsize=512 sectorsize=4096 | |
*-scsi:1 | |
physical id: 2 | |
logical name: scsi1 | |
capabilities: emulated | |
*-disk | |
description: ATA Disk | |
product: Crucial_CT512MX1 | |
physical id: 0.0.0 | |
bus info: scsi@1:0.0.0 | |
logical name: /dev/sdb | |
version: MU01 | |
serial: 14330CF93238 | |
size: 476GiB (512GB) | |
capabilities: gpt-1.00 partitioned partitioned:gpt | |
configuration: ansiversion=5 guid=b3e05cb9-ac11-46b2-a41b-5cf4164bcb34 logicalsectorsize=512 sectorsize=4096 | |
*-scsi:2 | |
physical id: 3 | |
logical name: scsi2 | |
capabilities: emulated | |
*-disk | |
description: ATA Disk | |
product: ST2000DM006-2DM1 | |
vendor: Seagate | |
physical id: 0.0.0 | |
bus info: scsi@2:0.0.0 | |
logical name: /dev/sdc | |
version: CC26 | |
serial: Z4Z83385 | |
size: 1863GiB (2TB) | |
capabilities: gpt-1.00 partitioned partitioned:gpt | |
configuration: ansiversion=5 guid=86fe1a86-58ce-448b-bb84-1461f00ab21f logicalsectorsize=512 sectorsize=4096 | |
*-scsi:3 | |
physical id: 4 | |
logical name: scsi3 | |
capabilities: emulated | |
*-disk | |
description: ATA Disk | |
product: ST2000DM006-2DM1 | |
vendor: Seagate | |
physical id: 0.0.0 | |
bus info: scsi@3:0.0.0 | |
logical name: /dev/sdd | |
version: CC26 | |
serial: Z4ZB7EV9 | |
size: 1863GiB (2TB) | |
capabilities: gpt-1.00 partitioned partitioned:gpt | |
configuration: ansiversion=5 guid=fa606d24-f1b6-4661-ab08-41e3bae03746 logicalsectorsize=512 sectorsize=4096 | |
➜ ~ lsblk | |
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT | |
sda 8:0 0 477G 0 disk | |
├─sda1 8:1 0 512M 0 part | |
├─sda2 8:2 0 1M 0 part | |
├─sda3 8:3 0 15.9G 0 part [SWAP] | |
└─sda4 8:4 0 460.6G 0 part / | |
sdb 8:16 0 477G 0 disk | |
└─sdb1 8:17 0 477G 0 part | |
sdc 8:32 0 1.8T 0 disk | |
├─sdc1 8:33 0 128M 0 part | |
└─sdc2 8:34 0 1.8T 0 part | |
sdd 8:48 0 1.8T 0 disk | |
└─sdd1 8:49 0 1.8T 0 part /mnt/data | |
➜ ~ ulimit -n 4096 | |
➜ ~ dstat | |
You did not select any stats, using -cdngy by default. | |
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- | |
usr sys idl wai hiq siq| read writ| recv send| in out | int csw | |
3 1 96 1 0 0|8313k 541k| 0 0 | 0 0 |2283 14k | |
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 36 103 | |
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 37 76 | |
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 28 65 | |
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 27 56 | |
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 26 56 | |
0 0 100 0 0 0| 0 36k| 309B 0 | 0 0 | 40 85 | |
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 23 45 | |
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 24 59 | |
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 24 50 | |
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 103 368 | |
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 61 283 ^C | |
➜ ~ cd src/LocustDB | |
➜ LocustDB git:(master) git rev-parse HEAD | |
573557575140c94e0aa3a0576ece04beb718712c | |
➜ LocustDB git:(master) du -h /home/clemens/ldb | |
80G /home/clemens/ldb | |
➜ LocustDB git:(master) du /home/clemens/ldb | |
83738212 /home/clemens/ldb | |
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb --mem-limit-tables=45 --mem-lz4 | |
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches" | |
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb --mem-limit-tables=45 --mem-lz4 | |
Finished release [optimized] target(s) in 0.44s | |
Running `target/release/repl --db-path /home/clemens/ldb --mem-limit-tables=45 --mem-lz4` | |
# Table `trips` (1464785771 rows, 192KiB) # | |
dropoff_puma: 0.00B | |
pickup_ntaname: 0.00B | |
mta_tax: 0.00B | |
pickup_ntacode: 0.00B | |
snow_depth: 0.00B | |
pickup_nyct2010_gid: 0.00B | |
payment_type: 0.00B | |
tip_amount: 0.00B | |
pickup_borocode: 0.00B | |
pickup_boroname: 0.00B | |
vendor_id: 0.00B | |
dropoff_latitude: 0.00B | |
dropoff_nyct2010_gid: 0.00B | |
pickup_longitude: 0.00B | |
passenger_count: 0.00B | |
pickup: 0.00B | |
dropoff_boroct2010: 0.00B | |
dropoff_ntacode: 0.00B | |
pickup_boroct2010: 0.00B | |
dropoff_cdeligibil: 0.00B | |
store_and_fwd_flag: 0.00B | |
pickup_puma: 0.00B | |
precipitation: 0.00B | |
fare_amount: 0.00B | |
dropoff_longitude: 0.00B | |
dropoff: 0.00B | |
total_amount: 0.00B | |
pickup_latitude: 0.00B | |
pickup_ct2010: 0.00B | |
dropoff_boroname: 0.00B | |
pickup_ctlabel: 0.00B | |
improvement_surcharge: 0.00B | |
trip_id: 0.00B | |
average_wind_speed: 0.00B | |
trip_type: 0.00B | |
rate_code_id: 0.00B | |
pickup_datetime: 0.00B | |
ehail_fee: 0.00B | |
snowfall: 0.00B | |
tolls_amount: 0.00B | |
cab_type: 0.00B | |
dropoff_ntaname: 0.00B | |
dropoff_datetime: 0.00B | |
extra: 0.00B | |
dropoff_ctlabel: 0.00B | |
dropoff_ct2010: 0.00B | |
max_temperature: 0.00B | |
dropoff_borocode: 0.00B | |
trip_distance: 0.00B | |
min_temperature: 0.00B | |
pickup_cdeligibil: 0.00B | |
locustdb> :restore | |
Restored DB from disk in 164s | |
trips 45.2GiB 1464785771 rows | |
├─ average_wind_speed 58% resident 0.86GiB 1.9% 1.1B/row | |
├─ cab_type fully resident 12MiB 0.03% 8.6mB/row | |
├─ dropoff 54% resident 8.0GiB 18% 11B/row | |
├─ dropoff_borocode fully resident 0.13GiB 0.28% 93mB/row | |
├─ dropoff_boroct2010 74% resident 0.35GiB 0.78% 0.35B/row | |
├─ dropoff_boroname fully resident 0.13GiB 0.29% 96mB/row | |
├─ dropoff_cdeligibil fully resident 0.16GiB 0.34% 0.11B/row | |
├─ dropoff_ct2010 73% resident 0.33GiB 0.73% 0.33B/row | |
├─ dropoff_ctlabel 57% resident 0.18GiB 0.40% 0.23B/row | |
├─ dropoff_datetime 55% resident 2.9GiB 6.4% 3.8B/row | |
├─ dropoff_latitude fully resident 36MiB 0.08% 26mB/row | |
├─ dropoff_longitude fully resident 0.11GiB 0.25% 83mB/row | |
├─ dropoff_ntacode fully resident 0.31GiB 0.68% 0.23B/row | |
├─ dropoff_ntaname fully resident 0.36GiB 0.80% 0.26B/row | |
├─ dropoff_nyct2010_gid 57% resident 0.19GiB 0.41% 0.24B/row | |
├─ dropoff_puma fully resident 0.30GiB 0.66% 0.22B/row | |
├─ ehail_fee fully resident 9.9MiB 0.02% 7.1mB/row | |
├─ extra 69% resident 0.54GiB 1.2% 0.58B/row | |
├─ fare_amount 59% resident 1.4GiB 3.1% 1.8B/row | |
├─ improvement_surcharge fully resident 19MiB 0.04% 14mB/row | |
├─ max_temperature 62% resident 0.69GiB 1.5% 0.81B/row | |
├─ min_temperature 62% resident 0.68GiB 1.5% 0.81B/row | |
├─ mta_tax fully resident 42MiB 0.09% 30mB/row | |
├─ passenger_count 69% resident 0.53GiB 1.2% 0.56B/row | |
├─ payment_type 68% resident 0.50GiB 1.1% 0.54B/row | |
├─ pickup 54% resident 7.9GiB 18% 11B/row | |
├─ pickup_borocode fully resident 0.28GiB 0.63% 0.21B/row | |
├─ pickup_boroct2010 57% resident 1.5GiB 3.4% 2.0B/row | |
├─ pickup_boroname fully resident 0.29GiB 0.64% 0.21B/row | |
├─ pickup_cdeligibil fully resident 0.43GiB 0.95% 0.32B/row | |
├─ pickup_ct2010 58% resident 1.6GiB 3.5% 2.0B/row | |
├─ pickup_ctlabel 60% resident 1.3GiB 2.9% 1.6B/row | |
├─ pickup_datetime 55% resident 2.6GiB 5.8% 3.5B/row | |
├─ pickup_latitude fully resident 44MiB 0.09% 31mB/row | |
├─ pickup_longitude 57% resident 0.24GiB 0.53% 0.31B/row | |
├─ pickup_ntacode 61% resident 0.70GiB 1.5% 0.83B/row | |
├─ pickup_ntaname 63% resident 0.74GiB 1.6% 0.86B/row | |
├─ pickup_nyct2010_gid 60% resident 1.3GiB 2.9% 1.6B/row | |
├─ pickup_puma 60% resident 0.65GiB 1.4% 0.80B/row | |
├─ precipitation 60% resident 0.54GiB 1.2% 0.65B/row | |
├─ rate_code_id fully resident 0.12GiB 0.26% 86mB/row | |
├─ snow_depth fully resident 0.21GiB 0.46% 0.15B/row | |
├─ snowfall fully resident 0.15GiB 0.34% 0.11B/row | |
├─ store_and_fwd_flag 65% resident 0.34GiB 0.75% 0.38B/row | |
├─ tip_amount 64% resident 1.2GiB 2.6% 1.4B/row | |
├─ tolls_amount fully resident 0.25GiB 0.56% 0.19B/row | |
├─ total_amount 55% resident 1.5GiB 3.4% 2.1B/row | |
├─ trip_distance 58% resident 1.9GiB 4.1% 2.3B/row | |
├─ trip_id fully resident 65MiB 0.14% 47mB/row | |
├─ trip_type fully resident 8.8MiB 0.02% 6.3mB/row | |
└─ vendor_id 68% resident 0.50GiB 1.1% 0.54B/row | |
locustdb> | |
➜ LocustDB git:(master) du -h /mnt/data/ldb | |
80G /mnt/data/ldb | |
➜ LocustDB git:(master) du /mnt/data/ldb | |
83736088 /mnt/data/ldb | |
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read | |
Finished release [optimized] target(s) in 0.28s | |
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read` | |
^C | |
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches" | |
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read | |
Finished release [optimized] target(s) in 0.35s | |
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read` | |
# Table `trips` (1464785771 rows, 192KiB) # | |
cab_type: 0.00B | |
dropoff_longitude: 0.00B | |
pickup_ntacode: 0.00B | |
pickup_nyct2010_gid: 0.00B | |
tip_amount: 0.00B | |
snow_depth: 0.00B | |
improvement_surcharge: 0.00B | |
dropoff_datetime: 0.00B | |
pickup_ct2010: 0.00B | |
pickup_ntaname: 0.00B | |
pickup_boroct2010: 0.00B | |
min_temperature: 0.00B | |
pickup_boroname: 0.00B | |
dropoff_latitude: 0.00B | |
dropoff_borocode: 0.00B | |
store_and_fwd_flag: 0.00B | |
pickup_cdeligibil: 0.00B | |
pickup: 0.00B | |
pickup_ctlabel: 0.00B | |
pickup_datetime: 0.00B | |
average_wind_speed: 0.00B | |
dropoff_ntacode: 0.00B | |
dropoff_boroct2010: 0.00B | |
pickup_puma: 0.00B | |
ehail_fee: 0.00B | |
dropoff_boroname: 0.00B | |
pickup_longitude: 0.00B | |
payment_type: 0.00B | |
mta_tax: 0.00B | |
passenger_count: 0.00B | |
pickup_latitude: 0.00B | |
dropoff: 0.00B | |
dropoff_ctlabel: 0.00B | |
fare_amount: 0.00B | |
dropoff_ct2010: 0.00B | |
precipitation: 0.00B | |
trip_type: 0.00B | |
trip_id: 0.00B | |
trip_distance: 0.00B | |
dropoff_nyct2010_gid: 0.00B | |
dropoff_ntaname: 0.00B | |
pickup_borocode: 0.00B | |
snowfall: 0.00B | |
dropoff_cdeligibil: 0.00B | |
extra: 0.00B | |
max_temperature: 0.00B | |
vendor_id: 0.00B | |
tolls_amount: 0.00B | |
rate_code_id: 0.00B | |
dropoff_puma: 0.00B | |
total_amount: 0.00B | |
locustdb> :restore | |
Restored DB from disk in 625s | |
trips 45.1GiB 1464785771 rows | |
├─ average_wind_speed 58% resident 0.86GiB 1.9% 1.1B/row | |
├─ cab_type fully resident 12MiB 0.03% 8.6mB/row | |
├─ dropoff 53% resident 7.7GiB 17% 11B/row | |
├─ dropoff_borocode fully resident 0.13GiB 0.28% 93mB/row | |
├─ dropoff_boroct2010 74% resident 0.35GiB 0.78% 0.35B/row | |
├─ dropoff_boroname fully resident 0.13GiB 0.29% 96mB/row | |
├─ dropoff_cdeligibil fully resident 0.16GiB 0.35% 0.11B/row | |
├─ dropoff_ct2010 73% resident 0.33GiB 0.73% 0.33B/row | |
├─ dropoff_ctlabel 57% resident 0.18GiB 0.40% 0.23B/row | |
├─ dropoff_datetime 54% resident 2.8GiB 6.3% 3.8B/row | |
├─ dropoff_latitude fully resident 36MiB 0.08% 26mB/row | |
├─ dropoff_longitude fully resident 0.11GiB 0.25% 83mB/row | |
├─ dropoff_ntacode fully resident 0.31GiB 0.68% 0.23B/row | |
├─ dropoff_ntaname fully resident 0.36GiB 0.80% 0.26B/row | |
├─ dropoff_nyct2010_gid 57% resident 0.19GiB 0.41% 0.24B/row | |
├─ dropoff_puma fully resident 0.30GiB 0.67% 0.22B/row | |
├─ ehail_fee fully resident 9.9MiB 0.02% 7.1mB/row | |
├─ extra 69% resident 0.54GiB 1.2% 0.58B/row | |
├─ fare_amount 59% resident 1.4GiB 3.2% 1.8B/row | |
├─ improvement_surcharge fully resident 19MiB 0.04% 14mB/row | |
├─ max_temperature 62% resident 0.69GiB 1.5% 0.81B/row | |
├─ min_temperature 62% resident 0.68GiB 1.5% 0.81B/row | |
├─ mta_tax fully resident 42MiB 0.09% 30mB/row | |
├─ passenger_count 69% resident 0.53GiB 1.2% 0.56B/row | |
├─ payment_type 68% resident 0.50GiB 1.1% 0.54B/row | |
├─ pickup 54% resident 7.9GiB 18% 11B/row | |
├─ pickup_borocode fully resident 0.28GiB 0.63% 0.21B/row | |
├─ pickup_boroct2010 60% resident 1.6GiB 3.6% 2.0B/row | |
├─ pickup_boroname fully resident 0.29GiB 0.64% 0.21B/row | |
├─ pickup_cdeligibil fully resident 0.43GiB 0.96% 0.32B/row | |
├─ pickup_ct2010 60% resident 1.6GiB 3.6% 2.0B/row | |
├─ pickup_ctlabel 60% resident 1.3GiB 2.9% 1.6B/row | |
├─ pickup_datetime 54% resident 2.6GiB 5.7% 3.5B/row | |
├─ pickup_latitude fully resident 44MiB 0.09% 31mB/row | |
├─ pickup_longitude fully resident 0.46GiB 1.0% 0.34B/row | |
├─ pickup_ntacode 61% resident 0.70GiB 1.5% 0.83B/row | |
├─ pickup_ntaname 63% resident 0.74GiB 1.6% 0.86B/row | |
├─ pickup_nyct2010_gid 60% resident 1.3GiB 2.9% 1.6B/row | |
├─ pickup_puma 60% resident 0.65GiB 1.5% 0.80B/row | |
├─ precipitation 55% resident 0.47GiB 1.0% 0.63B/row | |
├─ rate_code_id fully resident 0.12GiB 0.26% 86mB/row | |
├─ snow_depth fully resident 0.21GiB 0.46% 0.15B/row | |
├─ snowfall fully resident 0.15GiB 0.34% 0.11B/row | |
├─ store_and_fwd_flag 65% resident 0.34GiB 0.75% 0.38B/row | |
├─ tip_amount 64% resident 1.2GiB 2.6% 1.4B/row | |
├─ tolls_amount fully resident 0.25GiB 0.56% 0.19B/row | |
├─ total_amount 55% resident 1.5GiB 3.4% 2.1B/row | |
├─ trip_distance 58% resident 1.9GiB 4.1% 2.3B/row | |
├─ trip_id fully resident 65MiB 0.14% 47mB/row | |
├─ trip_type fully resident 8.8MiB 0.02% 6.3mB/row | |
└─ vendor_id 68% resident 0.50GiB 1.1% 0.54B/row | |
locustdb> | |
➜ LocustDB git:(master) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment