/locustdb-bulk-load-ssd-hdd
Created Aug 12, 2018
| ➜ ~ uname -a | |
| Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | |
| ➜ ~ sudo lshw -class disk -class storage | |
| [sudo] password for clemens: | |
| *-storage | |
| description: SATA controller | |
| product: ASM1062 Serial ATA Controller | |
| vendor: ASMedia Technology Inc. | |
| physical id: 0 | |
| bus info: pci@0000:06:00.0 | |
| version: 02 | |
| width: 32 bits | |
| clock: 33MHz | |
| capabilities: storage msi pm pciexpress ahci_1.0 bus_master cap_list | |
| configuration: driver=ahci latency=0 | |
| resources: irq:43 ioport:d050(size=8) ioport:d040(size=4) ioport:d030(size=8) ioport:d020(size=4) ioport:d000(size=32) memory:fb200000-fb2001ff | |
| *-storage | |
| description: SATA controller | |
| product: C610/X99 series chipset 6-Port SATA Controller [AHCI mode] | |
| vendor: Intel Corporation | |
| physical id: 1f.2 | |
| bus info: pci@0000:00:1f.2 | |
| version: 05 | |
| width: 32 bits | |
| clock: 66MHz | |
| capabilities: storage msi pm ahci_1.0 bus_master cap_list | |
| configuration: driver=ahci latency=0 | |
| resources: irq:42 ioport:f070(size=8) ioport:f060(size=4) ioport:f050(size=8) ioport:f040(size=4) ioport:f000(size=32) memory:fb336000-fb3367ff | |
| *-scsi:0 | |
| physical id: 1 | |
| logical name: scsi0 | |
| capabilities: emulated | |
| *-disk | |
| description: ATA Disk | |
| product: Crucial_CT512MX1 | |
| physical id: 0.0.0 | |
| bus info: scsi@0:0.0.0 | |
| logical name: /dev/sda | |
| version: MU01 | |
| serial: 14340CFD629E | |
| size: 476GiB (512GB) | |
| capabilities: gpt-1.00 partitioned partitioned:gpt | |
| configuration: ansiversion=5 guid=75bae818-9153-47eb-9cee-c01f70c5c42b logicalsectorsize=512 sectorsize=4096 | |
| *-scsi:1 | |
| physical id: 2 | |
| logical name: scsi1 | |
| capabilities: emulated | |
| *-disk | |
| description: ATA Disk | |
| product: Crucial_CT512MX1 | |
| physical id: 0.0.0 | |
| bus info: scsi@1:0.0.0 | |
| logical name: /dev/sdb | |
| version: MU01 | |
| serial: 14330CF93238 | |
| size: 476GiB (512GB) | |
| capabilities: gpt-1.00 partitioned partitioned:gpt | |
| configuration: ansiversion=5 guid=b3e05cb9-ac11-46b2-a41b-5cf4164bcb34 logicalsectorsize=512 sectorsize=4096 | |
| *-scsi:2 | |
| physical id: 3 | |
| logical name: scsi2 | |
| capabilities: emulated | |
| *-disk | |
| description: ATA Disk | |
| product: ST2000DM006-2DM1 | |
| vendor: Seagate | |
| physical id: 0.0.0 | |
| bus info: scsi@2:0.0.0 | |
| logical name: /dev/sdc | |
| version: CC26 | |
| serial: Z4Z83385 | |
| size: 1863GiB (2TB) | |
| capabilities: gpt-1.00 partitioned partitioned:gpt | |
| configuration: ansiversion=5 guid=86fe1a86-58ce-448b-bb84-1461f00ab21f logicalsectorsize=512 sectorsize=4096 | |
| *-scsi:3 | |
| physical id: 4 | |
| logical name: scsi3 | |
| capabilities: emulated | |
| *-disk | |
| description: ATA Disk | |
| product: ST2000DM006-2DM1 | |
| vendor: Seagate | |
| physical id: 0.0.0 | |
| bus info: scsi@3:0.0.0 | |
| logical name: /dev/sdd | |
| version: CC26 | |
| serial: Z4ZB7EV9 | |
| size: 1863GiB (2TB) | |
| capabilities: gpt-1.00 partitioned partitioned:gpt | |
| configuration: ansiversion=5 guid=fa606d24-f1b6-4661-ab08-41e3bae03746 logicalsectorsize=512 sectorsize=4096 | |
| ➜ ~ lsblk | |
| NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT | |
| sda 8:0 0 477G 0 disk | |
| ├─sda1 8:1 0 512M 0 part | |
| ├─sda2 8:2 0 1M 0 part | |
| ├─sda3 8:3 0 15.9G 0 part [SWAP] | |
| └─sda4 8:4 0 460.6G 0 part / | |
| sdb 8:16 0 477G 0 disk | |
| └─sdb1 8:17 0 477G 0 part | |
| sdc 8:32 0 1.8T 0 disk | |
| ├─sdc1 8:33 0 128M 0 part | |
| └─sdc2 8:34 0 1.8T 0 part | |
| sdd 8:48 0 1.8T 0 disk | |
| └─sdd1 8:49 0 1.8T 0 part /mnt/data | |
| ➜ ~ ulimit -n 4096 | |
| ➜ ~ dstat | |
| You did not select any stats, using -cdngy by default. | |
| ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- | |
| usr sys idl wai hiq siq| read writ| recv send| in out | int csw | |
| 3 1 96 1 0 0|8313k 541k| 0 0 | 0 0 |2283 14k | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 36 103 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 37 76 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 28 65 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 27 56 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 26 56 | |
| 0 0 100 0 0 0| 0 36k| 309B 0 | 0 0 | 40 85 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 23 45 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 24 59 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 24 50 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 103 368 | |
| 0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 61 283 ^C | |
| ➜ ~ cd src/LocustDB | |
| ➜ LocustDB git:(master) git rev-parse HEAD | |
| 573557575140c94e0aa3a0576ece04beb718712c | |
| ➜ LocustDB git:(master) du -h /home/clemens/ldb | |
| 80G /home/clemens/ldb | |
| ➜ LocustDB git:(master) du /home/clemens/ldb | |
| 83738212 /home/clemens/ldb | |
| ➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb --mem-limit-tables=45 --mem-lz4 | |
| ➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches" | |
| ➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb --mem-limit-tables=45 --mem-lz4 | |
| Finished release [optimized] target(s) in 0.44s | |
| Running `target/release/repl --db-path /home/clemens/ldb --mem-limit-tables=45 --mem-lz4` | |
| # Table `trips` (1464785771 rows, 192KiB) # | |
| dropoff_puma: 0.00B | |
| pickup_ntaname: 0.00B | |
| mta_tax: 0.00B | |
| pickup_ntacode: 0.00B | |
| snow_depth: 0.00B | |
| pickup_nyct2010_gid: 0.00B | |
| payment_type: 0.00B | |
| tip_amount: 0.00B | |
| pickup_borocode: 0.00B | |
| pickup_boroname: 0.00B | |
| vendor_id: 0.00B | |
| dropoff_latitude: 0.00B | |
| dropoff_nyct2010_gid: 0.00B | |
| pickup_longitude: 0.00B | |
| passenger_count: 0.00B | |
| pickup: 0.00B | |
| dropoff_boroct2010: 0.00B | |
| dropoff_ntacode: 0.00B | |
| pickup_boroct2010: 0.00B | |
| dropoff_cdeligibil: 0.00B | |
| store_and_fwd_flag: 0.00B | |
| pickup_puma: 0.00B | |
| precipitation: 0.00B | |
| fare_amount: 0.00B | |
| dropoff_longitude: 0.00B | |
| dropoff: 0.00B | |
| total_amount: 0.00B | |
| pickup_latitude: 0.00B | |
| pickup_ct2010: 0.00B | |
| dropoff_boroname: 0.00B | |
| pickup_ctlabel: 0.00B | |
| improvement_surcharge: 0.00B | |
| trip_id: 0.00B | |
| average_wind_speed: 0.00B | |
| trip_type: 0.00B | |
| rate_code_id: 0.00B | |
| pickup_datetime: 0.00B | |
| ehail_fee: 0.00B | |
| snowfall: 0.00B | |
| tolls_amount: 0.00B | |
| cab_type: 0.00B | |
| dropoff_ntaname: 0.00B | |
| dropoff_datetime: 0.00B | |
| extra: 0.00B | |
| dropoff_ctlabel: 0.00B | |
| dropoff_ct2010: 0.00B | |
| max_temperature: 0.00B | |
| dropoff_borocode: 0.00B | |
| trip_distance: 0.00B | |
| min_temperature: 0.00B | |
| pickup_cdeligibil: 0.00B | |
| locustdb> :restore | |
| Restored DB from disk in 164s | |
| trips 45.2GiB 1464785771 rows | |
| ├─ average_wind_speed 58% resident 0.86GiB 1.9% 1.1B/row | |
| ├─ cab_type fully resident 12MiB 0.03% 8.6mB/row | |
| ├─ dropoff 54% resident 8.0GiB 18% 11B/row | |
| ├─ dropoff_borocode fully resident 0.13GiB 0.28% 93mB/row | |
| ├─ dropoff_boroct2010 74% resident 0.35GiB 0.78% 0.35B/row | |
| ├─ dropoff_boroname fully resident 0.13GiB 0.29% 96mB/row | |
| ├─ dropoff_cdeligibil fully resident 0.16GiB 0.34% 0.11B/row | |
| ├─ dropoff_ct2010 73% resident 0.33GiB 0.73% 0.33B/row | |
| ├─ dropoff_ctlabel 57% resident 0.18GiB 0.40% 0.23B/row | |
| ├─ dropoff_datetime 55% resident 2.9GiB 6.4% 3.8B/row | |
| ├─ dropoff_latitude fully resident 36MiB 0.08% 26mB/row | |
| ├─ dropoff_longitude fully resident 0.11GiB 0.25% 83mB/row | |
| ├─ dropoff_ntacode fully resident 0.31GiB 0.68% 0.23B/row | |
| ├─ dropoff_ntaname fully resident 0.36GiB 0.80% 0.26B/row | |
| ├─ dropoff_nyct2010_gid 57% resident 0.19GiB 0.41% 0.24B/row | |
| ├─ dropoff_puma fully resident 0.30GiB 0.66% 0.22B/row | |
| ├─ ehail_fee fully resident 9.9MiB 0.02% 7.1mB/row | |
| ├─ extra 69% resident 0.54GiB 1.2% 0.58B/row | |
| ├─ fare_amount 59% resident 1.4GiB 3.1% 1.8B/row | |
| ├─ improvement_surcharge fully resident 19MiB 0.04% 14mB/row | |
| ├─ max_temperature 62% resident 0.69GiB 1.5% 0.81B/row | |
| ├─ min_temperature 62% resident 0.68GiB 1.5% 0.81B/row | |
| ├─ mta_tax fully resident 42MiB 0.09% 30mB/row | |
| ├─ passenger_count 69% resident 0.53GiB 1.2% 0.56B/row | |
| ├─ payment_type 68% resident 0.50GiB 1.1% 0.54B/row | |
| ├─ pickup 54% resident 7.9GiB 18% 11B/row | |
| ├─ pickup_borocode fully resident 0.28GiB 0.63% 0.21B/row | |
| ├─ pickup_boroct2010 57% resident 1.5GiB 3.4% 2.0B/row | |
| ├─ pickup_boroname fully resident 0.29GiB 0.64% 0.21B/row | |
| ├─ pickup_cdeligibil fully resident 0.43GiB 0.95% 0.32B/row | |
| ├─ pickup_ct2010 58% resident 1.6GiB 3.5% 2.0B/row | |
| ├─ pickup_ctlabel 60% resident 1.3GiB 2.9% 1.6B/row | |
| ├─ pickup_datetime 55% resident 2.6GiB 5.8% 3.5B/row | |
| ├─ pickup_latitude fully resident 44MiB 0.09% 31mB/row | |
| ├─ pickup_longitude 57% resident 0.24GiB 0.53% 0.31B/row | |
| ├─ pickup_ntacode 61% resident 0.70GiB 1.5% 0.83B/row | |
| ├─ pickup_ntaname 63% resident 0.74GiB 1.6% 0.86B/row | |
| ├─ pickup_nyct2010_gid 60% resident 1.3GiB 2.9% 1.6B/row | |
| ├─ pickup_puma 60% resident 0.65GiB 1.4% 0.80B/row | |
| ├─ precipitation 60% resident 0.54GiB 1.2% 0.65B/row | |
| ├─ rate_code_id fully resident 0.12GiB 0.26% 86mB/row | |
| ├─ snow_depth fully resident 0.21GiB 0.46% 0.15B/row | |
| ├─ snowfall fully resident 0.15GiB 0.34% 0.11B/row | |
| ├─ store_and_fwd_flag 65% resident 0.34GiB 0.75% 0.38B/row | |
| ├─ tip_amount 64% resident 1.2GiB 2.6% 1.4B/row | |
| ├─ tolls_amount fully resident 0.25GiB 0.56% 0.19B/row | |
| ├─ total_amount 55% resident 1.5GiB 3.4% 2.1B/row | |
| ├─ trip_distance 58% resident 1.9GiB 4.1% 2.3B/row | |
| ├─ trip_id fully resident 65MiB 0.14% 47mB/row | |
| ├─ trip_type fully resident 8.8MiB 0.02% 6.3mB/row | |
| └─ vendor_id 68% resident 0.50GiB 1.1% 0.54B/row | |
| locustdb> | |
| ➜ LocustDB git:(master) du -h /mnt/data/ldb | |
| 80G /mnt/data/ldb | |
| ➜ LocustDB git:(master) du /mnt/data/ldb | |
| 83736088 /mnt/data/ldb | |
| ➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read | |
| Finished release [optimized] target(s) in 0.28s | |
| Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read` | |
| ^C | |
| ➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches" | |
| ➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read | |
| Finished release [optimized] target(s) in 0.35s | |
| Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read` | |
| # Table `trips` (1464785771 rows, 192KiB) # | |
| cab_type: 0.00B | |
| dropoff_longitude: 0.00B | |
| pickup_ntacode: 0.00B | |
| pickup_nyct2010_gid: 0.00B | |
| tip_amount: 0.00B | |
| snow_depth: 0.00B | |
| improvement_surcharge: 0.00B | |
| dropoff_datetime: 0.00B | |
| pickup_ct2010: 0.00B | |
| pickup_ntaname: 0.00B | |
| pickup_boroct2010: 0.00B | |
| min_temperature: 0.00B | |
| pickup_boroname: 0.00B | |
| dropoff_latitude: 0.00B | |
| dropoff_borocode: 0.00B | |
| store_and_fwd_flag: 0.00B | |
| pickup_cdeligibil: 0.00B | |
| pickup: 0.00B | |
| pickup_ctlabel: 0.00B | |
| pickup_datetime: 0.00B | |
| average_wind_speed: 0.00B | |
| dropoff_ntacode: 0.00B | |
| dropoff_boroct2010: 0.00B | |
| pickup_puma: 0.00B | |
| ehail_fee: 0.00B | |
| dropoff_boroname: 0.00B | |
| pickup_longitude: 0.00B | |
| payment_type: 0.00B | |
| mta_tax: 0.00B | |
| passenger_count: 0.00B | |
| pickup_latitude: 0.00B | |
| dropoff: 0.00B | |
| dropoff_ctlabel: 0.00B | |
| fare_amount: 0.00B | |
| dropoff_ct2010: 0.00B | |
| precipitation: 0.00B | |
| trip_type: 0.00B | |
| trip_id: 0.00B | |
| trip_distance: 0.00B | |
| dropoff_nyct2010_gid: 0.00B | |
| dropoff_ntaname: 0.00B | |
| pickup_borocode: 0.00B | |
| snowfall: 0.00B | |
| dropoff_cdeligibil: 0.00B | |
| extra: 0.00B | |
| max_temperature: 0.00B | |
| vendor_id: 0.00B | |
| tolls_amount: 0.00B | |
| rate_code_id: 0.00B | |
| dropoff_puma: 0.00B | |
| total_amount: 0.00B | |
| locustdb> :restore | |
| Restored DB from disk in 625s | |
| trips 45.1GiB 1464785771 rows | |
| ├─ average_wind_speed 58% resident 0.86GiB 1.9% 1.1B/row | |
| ├─ cab_type fully resident 12MiB 0.03% 8.6mB/row | |
| ├─ dropoff 53% resident 7.7GiB 17% 11B/row | |
| ├─ dropoff_borocode fully resident 0.13GiB 0.28% 93mB/row | |
| ├─ dropoff_boroct2010 74% resident 0.35GiB 0.78% 0.35B/row | |
| ├─ dropoff_boroname fully resident 0.13GiB 0.29% 96mB/row | |
| ├─ dropoff_cdeligibil fully resident 0.16GiB 0.35% 0.11B/row | |
| ├─ dropoff_ct2010 73% resident 0.33GiB 0.73% 0.33B/row | |
| ├─ dropoff_ctlabel 57% resident 0.18GiB 0.40% 0.23B/row | |
| ├─ dropoff_datetime 54% resident 2.8GiB 6.3% 3.8B/row | |
| ├─ dropoff_latitude fully resident 36MiB 0.08% 26mB/row | |
| ├─ dropoff_longitude fully resident 0.11GiB 0.25% 83mB/row | |
| ├─ dropoff_ntacode fully resident 0.31GiB 0.68% 0.23B/row | |
| ├─ dropoff_ntaname fully resident 0.36GiB 0.80% 0.26B/row | |
| ├─ dropoff_nyct2010_gid 57% resident 0.19GiB 0.41% 0.24B/row | |
| ├─ dropoff_puma fully resident 0.30GiB 0.67% 0.22B/row | |
| ├─ ehail_fee fully resident 9.9MiB 0.02% 7.1mB/row | |
| ├─ extra 69% resident 0.54GiB 1.2% 0.58B/row | |
| ├─ fare_amount 59% resident 1.4GiB 3.2% 1.8B/row | |
| ├─ improvement_surcharge fully resident 19MiB 0.04% 14mB/row | |
| ├─ max_temperature 62% resident 0.69GiB 1.5% 0.81B/row | |
| ├─ min_temperature 62% resident 0.68GiB 1.5% 0.81B/row | |
| ├─ mta_tax fully resident 42MiB 0.09% 30mB/row | |
| ├─ passenger_count 69% resident 0.53GiB 1.2% 0.56B/row | |
| ├─ payment_type 68% resident 0.50GiB 1.1% 0.54B/row | |
| ├─ pickup 54% resident 7.9GiB 18% 11B/row | |
| ├─ pickup_borocode fully resident 0.28GiB 0.63% 0.21B/row | |
| ├─ pickup_boroct2010 60% resident 1.6GiB 3.6% 2.0B/row | |
| ├─ pickup_boroname fully resident 0.29GiB 0.64% 0.21B/row | |
| ├─ pickup_cdeligibil fully resident 0.43GiB 0.96% 0.32B/row | |
| ├─ pickup_ct2010 60% resident 1.6GiB 3.6% 2.0B/row | |
| ├─ pickup_ctlabel 60% resident 1.3GiB 2.9% 1.6B/row | |
| ├─ pickup_datetime 54% resident 2.6GiB 5.7% 3.5B/row | |
| ├─ pickup_latitude fully resident 44MiB 0.09% 31mB/row | |
| ├─ pickup_longitude fully resident 0.46GiB 1.0% 0.34B/row | |
| ├─ pickup_ntacode 61% resident 0.70GiB 1.5% 0.83B/row | |
| ├─ pickup_ntaname 63% resident 0.74GiB 1.6% 0.86B/row | |
| ├─ pickup_nyct2010_gid 60% resident 1.3GiB 2.9% 1.6B/row | |
| ├─ pickup_puma 60% resident 0.65GiB 1.5% 0.80B/row | |
| ├─ precipitation 55% resident 0.47GiB 1.0% 0.63B/row | |
| ├─ rate_code_id fully resident 0.12GiB 0.26% 86mB/row | |
| ├─ snow_depth fully resident 0.21GiB 0.46% 0.15B/row | |
| ├─ snowfall fully resident 0.15GiB 0.34% 0.11B/row | |
| ├─ store_and_fwd_flag 65% resident 0.34GiB 0.75% 0.38B/row | |
| ├─ tip_amount 64% resident 1.2GiB 2.6% 1.4B/row | |
| ├─ tolls_amount fully resident 0.25GiB 0.56% 0.19B/row | |
| ├─ total_amount 55% resident 1.5GiB 3.4% 2.1B/row | |
| ├─ trip_distance 58% resident 1.9GiB 4.1% 2.3B/row | |
| ├─ trip_id fully resident 65MiB 0.14% 47mB/row | |
| ├─ trip_type fully resident 8.8MiB 0.02% 6.3mB/row | |
| └─ vendor_id 68% resident 0.50GiB 1.1% 0.54B/row | |
| locustdb> | |
| ➜ LocustDB git:(master) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment