This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ➜ ~ uname -a | |
| Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | |
| ➜ ~ sudo lshw -class disk -class storage | |
| [sudo] password for clemens: | |
| *-storage | |
| description: SATA controller | |
| product: ASM1062 Serial ATA Controller | |
| vendor: ASMedia Technology Inc. | |
| physical id: 0 | |
| bus info: pci@0000:06:00.0 | |
| version: 02 | |
| width: 32 bits | |
| clock: 33MHz | |
| capabilities: storage msi pm pciexpress ahci_1.0 bus_master cap_list | |
| configuration: driver=ahci latency=0 | |
| resources: irq:43 ioport:d050(size=8) ioport:d040(size=4) ioport:d030(size=8) ioport:d020(size=4) ioport:d000(size=32) memory:fb200000-fb2001ff | |
| *-storage | |
| description: SATA controller | |
| product: C610/X99 series chipset 6-Port SATA Controller [AHCI mode] | |
| vendor: Intel Corporation | |
| physical id: 1f.2 | |
| bus info: pci@0000:00:1f.2 | |
| version: 05 | |
| width: 32 bits | |
| clock: 66MHz | |
| capabilities: storage msi pm ahci_1.0 bus_master cap_list | |
| configuration: driver=ahci latency=0 | |
| resources: irq:42 ioport:f070(size=8) ioport:f060(size=4) ioport:f050(size=8) ioport:f040(size=4) ioport:f000(size=32) memory:fb336000-fb3367ff | |
| *-scsi:0 | |
| physical id: 1 | |
| logical name: scsi0 | |
| capabilities: emulated | |
| *-disk | |
| description: ATA Disk | |
| product: Crucial_CT512MX1 | |
| physical id: 0.0.0 | |
| bus info: scsi@0:0.0.0 | |
| logical name: /dev/sda | |
| version: MU01 | |
| serial: 14340CFD629E | |
| size: 476GiB (512GB) | |
| capabilities: gpt-1.00 partitioned partitioned:gpt | |
| configuration: ansiversion=5 guid=75bae818-9153-47eb-9cee-c01f70c5c42b logicalsectorsize=512 sectorsize=4096 | |
| *-scsi:1 | |
| physical id: 2 | |
| logical name: scsi1 | |
| capabilities: emulated | |
| *-disk | |
| description: ATA Disk | |
| product: Crucial_CT512MX1 | |
| physical id: 0.0.0 | |
| bus info: scsi@1:0.0.0 | |
| logical name: /dev/sdb | |
| version: MU01 | |
| serial: 14330CF93238 | |
| size: 476GiB (512GB) | |
| capabilities: gpt-1.00 partitioned partitioned:gpt | |
| configuration: ansiversion=5 guid=b3e05cb9-ac11-46b2-a41b-5cf4164bcb34 logicalsectorsize=512 sectorsize=4096 | |
| *-scsi:2 | |
| physical id: 3 | |
| logical name: scsi2 | |
| capabilities: emulated | |
| *-disk | |
| description: ATA Disk | |
| product: ST2000DM006-2DM1 | |
| vendor: Seagate | |
| physical id: 0.0.0 | |
| bus info: scsi@2:0.0.0 | |
| logical name: /dev/sdc | |
| version: CC26 | |
| serial: Z4Z83385 | |
| size: 1863GiB (2TB) | |
| capabilities: gpt-1.00 partitioned partitioned:gpt | |
| configuration: ansiversion=5 guid=86fe1a86-58ce-448b-bb84-1461f00ab21f logicalsectorsize=512 sectorsize=4096 | |
| *-scsi:3 | |
| physical id: 4 | |
| logical name: scsi3 | |
| capabilities: emulated | |
| *-disk | |
| description: ATA Disk | |
| product: ST2000DM006-2DM1 | |
| vendor: Seagate | |
| physical id: 0.0.0 | |
| bus info: scsi@3:0.0.0 | |
| logical name: /dev/sdd | |
| version: CC26 | |
| serial: Z4ZB7EV9 | |
| size: 1863GiB (2TB) | |
| capabilities: gpt-1.00 partitioned partitioned:gpt | |
| configuration: ansiversion=5 guid=fa606d24-f1b6-4661-ab08-41e3bae03746 logicalsectorsize=512 sectorsize=4096 | |
| ➜ ~ lsblk | |
| NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT | |
| sda 8:0 0 477G 0 disk | |
| ├─sda1 8:1 0 512M 0 part | |
| ├─sda2 8:2 0 1M 0 part | |
| ├─sda3 8:3 0 15.9G 0 part [SWAP] | |
| └─sda4 8:4 0 460.6G 0 part / | |
| sdb 8:16 0 477G 0 disk | |
| └─sdb1 8:17 0 477G 0 part | |
| sdc 8:32 0 1.8T 0 disk | |
| ├─sdc1 8:33 0 128M 0 part | |
| └─sdc2 8:34 0 1.8T 0 part | |
| sdd 8:48 0 1.8T 0 disk | |
| └─sdd1 8:49 0 1.8T 0 part /mnt/data | |
| ➜ ~ ulimit -n 4096 | |
| ➜ ~ dstat | |
| You did not select any stats, using -cdngy by default. | |
| ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- | |
| usr sys idl wai hiq siq| read writ| recv send| in out | int csw | |
| 3 1 96 1 0 0|8313k 541k| 0 0 | 0 0 |2283 14k | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 36 103 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 37 76 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 28 65 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 27 56 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 26 56 | |
| 0 0 100 0 0 0| 0 36k| 309B 0 | 0 0 | 40 85 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 23 45 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 24 59 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 24 50 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 103 368 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 61 283 ^C | |
| ➜ ~ cd src/LocustDB | |
| ➜ LocustDB git:(master) git rev-parse HEAD | |
| 573557575140c94e0aa3a0576ece04beb718712c | |
| ➜ LocustDB git:(master) du -h /home/clemens/ldb | |
| 80G /home/clemens/ldb | |
| ➜ LocustDB git:(master) du /home/clemens/ldb | |
| 83738212 /home/clemens/ldb | |
| ➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb --mem-limit-tables=45 --mem-lz4 | |
| ➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches" | |
| ➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb --mem-limit-tables=45 --mem-lz4 | |
| Finished release [optimized] target(s) in 0.44s | |
| Running `target/release/repl --db-path /home/clemens/ldb --mem-limit-tables=45 --mem-lz4` | |
| # Table `trips` (1464785771 rows, 192KiB) # | |
| dropoff_puma: 0.00B | |
| pickup_ntaname: 0.00B | |
| mta_tax: 0.00B | |
| pickup_ntacode: 0.00B | |
| snow_depth: 0.00B | |
| pickup_nyct2010_gid: 0.00B | |
| payment_type: 0.00B | |
| tip_amount: 0.00B | |
| pickup_borocode: 0.00B | |
| pickup_boroname: 0.00B | |
| vendor_id: 0.00B | |
| dropoff_latitude: 0.00B | |
| dropoff_nyct2010_gid: 0.00B | |
| pickup_longitude: 0.00B | |
| passenger_count: 0.00B | |
| pickup: 0.00B | |
| dropoff_boroct2010: 0.00B | |
| dropoff_ntacode: 0.00B | |
| pickup_boroct2010: 0.00B | |
| dropoff_cdeligibil: 0.00B | |
| store_and_fwd_flag: 0.00B | |
| pickup_puma: 0.00B | |
| precipitation: 0.00B | |
| fare_amount: 0.00B | |
| dropoff_longitude: 0.00B | |
| dropoff: 0.00B | |
| total_amount: 0.00B | |
| pickup_latitude: 0.00B | |
| pickup_ct2010: 0.00B | |
| dropoff_boroname: 0.00B | |
| pickup_ctlabel: 0.00B | |
| improvement_surcharge: 0.00B | |
| trip_id: 0.00B | |
| average_wind_speed: 0.00B | |
| trip_type: 0.00B | |
| rate_code_id: 0.00B | |
| pickup_datetime: 0.00B | |
| ehail_fee: 0.00B | |
| snowfall: 0.00B | |
| tolls_amount: 0.00B | |
| cab_type: 0.00B | |
| dropoff_ntaname: 0.00B | |
| dropoff_datetime: 0.00B | |
| extra: 0.00B | |
| dropoff_ctlabel: 0.00B | |
| dropoff_ct2010: 0.00B | |
| max_temperature: 0.00B | |
| dropoff_borocode: 0.00B | |
| trip_distance: 0.00B | |
| min_temperature: 0.00B | |
| pickup_cdeligibil: 0.00B | |
| locustdb> :restore | |
| Restored DB from disk in 164s | |
| trips 45.2GiB 1464785771 rows | |
| ├─ average_wind_speed 58% resident 0.86GiB 1.9% 1.1B/row | |
| ├─ cab_type fully resident 12MiB 0.03% 8.6mB/row | |
| ├─ dropoff 54% resident 8.0GiB 18% 11B/row | |
| ├─ dropoff_borocode fully resident 0.13GiB 0.28% 93mB/row | |
| ├─ dropoff_boroct2010 74% resident 0.35GiB 0.78% 0.35B/row | |
| ├─ dropoff_boroname fully resident 0.13GiB 0.29% 96mB/row | |
| ├─ dropoff_cdeligibil fully resident 0.16GiB 0.34% 0.11B/row | |
| ├─ dropoff_ct2010 73% resident 0.33GiB 0.73% 0.33B/row | |
| ├─ dropoff_ctlabel 57% resident 0.18GiB 0.40% 0.23B/row | |
| ├─ dropoff_datetime 55% resident 2.9GiB 6.4% 3.8B/row | |
| ├─ dropoff_latitude fully resident 36MiB 0.08% 26mB/row | |
| ├─ dropoff_longitude fully resident 0.11GiB 0.25% 83mB/row | |
| ├─ dropoff_ntacode fully resident 0.31GiB 0.68% 0.23B/row | |
| ├─ dropoff_ntaname fully resident 0.36GiB 0.80% 0.26B/row | |
| ├─ dropoff_nyct2010_gid 57% resident 0.19GiB 0.41% 0.24B/row | |
| ├─ dropoff_puma fully resident 0.30GiB 0.66% 0.22B/row | |
| ├─ ehail_fee fully resident 9.9MiB 0.02% 7.1mB/row | |
| ├─ extra 69% resident 0.54GiB 1.2% 0.58B/row | |
| ├─ fare_amount 59% resident 1.4GiB 3.1% 1.8B/row | |
| ├─ improvement_surcharge fully resident 19MiB 0.04% 14mB/row | |
| ├─ max_temperature 62% resident 0.69GiB 1.5% 0.81B/row | |
| ├─ min_temperature 62% resident 0.68GiB 1.5% 0.81B/row | |
| ├─ mta_tax fully resident 42MiB 0.09% 30mB/row | |
| ├─ passenger_count 69% resident 0.53GiB 1.2% 0.56B/row | |
| ├─ payment_type 68% resident 0.50GiB 1.1% 0.54B/row | |
| ├─ pickup 54% resident 7.9GiB 18% 11B/row | |
| ├─ pickup_borocode fully resident 0.28GiB 0.63% 0.21B/row | |
| ├─ pickup_boroct2010 57% resident 1.5GiB 3.4% 2.0B/row | |
| ├─ pickup_boroname fully resident 0.29GiB 0.64% 0.21B/row | |
| ├─ pickup_cdeligibil fully resident 0.43GiB 0.95% 0.32B/row | |
| ├─ pickup_ct2010 58% resident 1.6GiB 3.5% 2.0B/row | |
| ├─ pickup_ctlabel 60% resident 1.3GiB 2.9% 1.6B/row | |
| ├─ pickup_datetime 55% resident 2.6GiB 5.8% 3.5B/row | |
| ├─ pickup_latitude fully resident 44MiB 0.09% 31mB/row | |
| ├─ pickup_longitude 57% resident 0.24GiB 0.53% 0.31B/row | |
| ├─ pickup_ntacode 61% resident 0.70GiB 1.5% 0.83B/row | |
| ├─ pickup_ntaname 63% resident 0.74GiB 1.6% 0.86B/row | |
| ├─ pickup_nyct2010_gid 60% resident 1.3GiB 2.9% 1.6B/row | |
| ├─ pickup_puma 60% resident 0.65GiB 1.4% 0.80B/row | |
| ├─ precipitation 60% resident 0.54GiB 1.2% 0.65B/row | |
| ├─ rate_code_id fully resident 0.12GiB 0.26% 86mB/row | |
| ├─ snow_depth fully resident 0.21GiB 0.46% 0.15B/row | |
| ├─ snowfall fully resident 0.15GiB 0.34% 0.11B/row | |
| ├─ store_and_fwd_flag 65% resident 0.34GiB 0.75% 0.38B/row | |
| ├─ tip_amount 64% resident 1.2GiB 2.6% 1.4B/row | |
| ├─ tolls_amount fully resident 0.25GiB 0.56% 0.19B/row | |
| ├─ total_amount 55% resident 1.5GiB 3.4% 2.1B/row | |
| ├─ trip_distance 58% resident 1.9GiB 4.1% 2.3B/row | |
| ├─ trip_id fully resident 65MiB 0.14% 47mB/row | |
| ├─ trip_type fully resident 8.8MiB 0.02% 6.3mB/row | |
| └─ vendor_id 68% resident 0.50GiB 1.1% 0.54B/row | |
| locustdb> | |
| ➜ LocustDB git:(master) du -h /mnt/data/ldb | |
| 80G /mnt/data/ldb | |
| ➜ LocustDB git:(master) du /mnt/data/ldb | |
| 83736088 /mnt/data/ldb | |
| ➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read | |
| Finished release [optimized] target(s) in 0.28s | |
| Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read` | |
| ^C | |
| ➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches" | |
| ➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read | |
| Finished release [optimized] target(s) in 0.35s | |
| Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read` | |
| # Table `trips` (1464785771 rows, 192KiB) # | |
| cab_type: 0.00B | |
| dropoff_longitude: 0.00B | |
| pickup_ntacode: 0.00B | |
| pickup_nyct2010_gid: 0.00B | |
| tip_amount: 0.00B | |
| snow_depth: 0.00B | |
| improvement_surcharge: 0.00B | |
| dropoff_datetime: 0.00B | |
| pickup_ct2010: 0.00B | |
| pickup_ntaname: 0.00B | |
| pickup_boroct2010: 0.00B | |
| min_temperature: 0.00B | |
| pickup_boroname: 0.00B | |
| dropoff_latitude: 0.00B | |
| dropoff_borocode: 0.00B | |
| store_and_fwd_flag: 0.00B | |
| pickup_cdeligibil: 0.00B | |
| pickup: 0.00B | |
| pickup_ctlabel: 0.00B | |
| pickup_datetime: 0.00B | |
| average_wind_speed: 0.00B | |
| dropoff_ntacode: 0.00B | |
| dropoff_boroct2010: 0.00B | |
| pickup_puma: 0.00B | |
| ehail_fee: 0.00B | |
| dropoff_boroname: 0.00B | |
| pickup_longitude: 0.00B | |
| payment_type: 0.00B | |
| mta_tax: 0.00B | |
| passenger_count: 0.00B | |
| pickup_latitude: 0.00B | |
| dropoff: 0.00B | |
| dropoff_ctlabel: 0.00B | |
| fare_amount: 0.00B | |
| dropoff_ct2010: 0.00B | |
| precipitation: 0.00B | |
| trip_type: 0.00B | |
| trip_id: 0.00B | |
| trip_distance: 0.00B | |
| dropoff_nyct2010_gid: 0.00B | |
| dropoff_ntaname: 0.00B | |
| pickup_borocode: 0.00B | |
| snowfall: 0.00B | |
| dropoff_cdeligibil: 0.00B | |
| extra: 0.00B | |
| max_temperature: 0.00B | |
| vendor_id: 0.00B | |
| tolls_amount: 0.00B | |
| rate_code_id: 0.00B | |
| dropoff_puma: 0.00B | |
| total_amount: 0.00B | |
| locustdb> :restore | |
| Restored DB from disk in 625s | |
| trips 45.1GiB 1464785771 rows | |
| ├─ average_wind_speed 58% resident 0.86GiB 1.9% 1.1B/row | |
| ├─ cab_type fully resident 12MiB 0.03% 8.6mB/row | |
| ├─ dropoff 53% resident 7.7GiB 17% 11B/row | |
| ├─ dropoff_borocode fully resident 0.13GiB 0.28% 93mB/row | |
| ├─ dropoff_boroct2010 74% resident 0.35GiB 0.78% 0.35B/row | |
| ├─ dropoff_boroname fully resident 0.13GiB 0.29% 96mB/row | |
| ├─ dropoff_cdeligibil fully resident 0.16GiB 0.35% 0.11B/row | |
| ├─ dropoff_ct2010 73% resident 0.33GiB 0.73% 0.33B/row | |
| ├─ dropoff_ctlabel 57% resident 0.18GiB 0.40% 0.23B/row | |
| ├─ dropoff_datetime 54% resident 2.8GiB 6.3% 3.8B/row | |
| ├─ dropoff_latitude fully resident 36MiB 0.08% 26mB/row | |
| ├─ dropoff_longitude fully resident 0.11GiB 0.25% 83mB/row | |
| ├─ dropoff_ntacode fully resident 0.31GiB 0.68% 0.23B/row | |
| ├─ dropoff_ntaname fully resident 0.36GiB 0.80% 0.26B/row | |
| ├─ dropoff_nyct2010_gid 57% resident 0.19GiB 0.41% 0.24B/row | |
| ├─ dropoff_puma fully resident 0.30GiB 0.67% 0.22B/row | |
| ├─ ehail_fee fully resident 9.9MiB 0.02% 7.1mB/row | |
| ├─ extra 69% resident 0.54GiB 1.2% 0.58B/row | |
| ├─ fare_amount 59% resident 1.4GiB 3.2% 1.8B/row | |
| ├─ improvement_surcharge fully resident 19MiB 0.04% 14mB/row | |
| ├─ max_temperature 62% resident 0.69GiB 1.5% 0.81B/row | |
| ├─ min_temperature 62% resident 0.68GiB 1.5% 0.81B/row | |
| ├─ mta_tax fully resident 42MiB 0.09% 30mB/row | |
| ├─ passenger_count 69% resident 0.53GiB 1.2% 0.56B/row | |
| ├─ payment_type 68% resident 0.50GiB 1.1% 0.54B/row | |
| ├─ pickup 54% resident 7.9GiB 18% 11B/row | |
| ├─ pickup_borocode fully resident 0.28GiB 0.63% 0.21B/row | |
| ├─ pickup_boroct2010 60% resident 1.6GiB 3.6% 2.0B/row | |
| ├─ pickup_boroname fully resident 0.29GiB 0.64% 0.21B/row | |
| ├─ pickup_cdeligibil fully resident 0.43GiB 0.96% 0.32B/row | |
| ├─ pickup_ct2010 60% resident 1.6GiB 3.6% 2.0B/row | |
| ├─ pickup_ctlabel 60% resident 1.3GiB 2.9% 1.6B/row | |
| ├─ pickup_datetime 54% resident 2.6GiB 5.7% 3.5B/row | |
| ├─ pickup_latitude fully resident 44MiB 0.09% 31mB/row | |
| ├─ pickup_longitude fully resident 0.46GiB 1.0% 0.34B/row | |
| ├─ pickup_ntacode 61% resident 0.70GiB 1.5% 0.83B/row | |
| ├─ pickup_ntaname 63% resident 0.74GiB 1.6% 0.86B/row | |
| ├─ pickup_nyct2010_gid 60% resident 1.3GiB 2.9% 1.6B/row | |
| ├─ pickup_puma 60% resident 0.65GiB 1.5% 0.80B/row | |
| ├─ precipitation 55% resident 0.47GiB 1.0% 0.63B/row | |
| ├─ rate_code_id fully resident 0.12GiB 0.26% 86mB/row | |
| ├─ snow_depth fully resident 0.21GiB 0.46% 0.15B/row | |
| ├─ snowfall fully resident 0.15GiB 0.34% 0.11B/row | |
| ├─ store_and_fwd_flag 65% resident 0.34GiB 0.75% 0.38B/row | |
| ├─ tip_amount 64% resident 1.2GiB 2.6% 1.4B/row | |
| ├─ tolls_amount fully resident 0.25GiB 0.56% 0.19B/row | |
| ├─ total_amount 55% resident 1.5GiB 3.4% 2.1B/row | |
| ├─ trip_distance 58% resident 1.9GiB 4.1% 2.3B/row | |
| ├─ trip_id fully resident 65MiB 0.14% 47mB/row | |
| ├─ trip_type fully resident 8.8MiB 0.02% 6.3mB/row | |
| └─ vendor_id 68% resident 0.50GiB 1.1% 0.54B/row | |
| locustdb> | |
| ➜ LocustDB git:(master) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment