This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
➜ ~ uname -a | |
Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | |
➜ ~ sudo lshw -class disk -class storage | |
[sudo] password for clemens: | |
*-storage | |
description: SATA controller | |
product: ASM1062 Serial ATA Controller | |
vendor: ASMedia Technology Inc. | |
physical id: 0 | |
bus info: pci@0000:06:00.0 | |
version: 02 | |
width: 32 bits | |
clock: 33MHz | |
capabilities: storage msi pm pciexpress ahci_1.0 bus_master cap_list | |
configuration: driver=ahci latency=0 | |
resources: irq:43 ioport:d050(size=8) ioport:d040(size=4) ioport:d030(size=8) ioport:d020(size=4) ioport:d000(size=32) memory:fb200000-fb2001ff | |
*-storage | |
description: SATA controller | |
product: C610/X99 series chipset 6-Port SATA Controller [AHCI mode] | |
vendor: Intel Corporation | |
physical id: 1f.2 | |
bus info: pci@0000:00:1f.2 | |
version: 05 | |
width: 32 bits | |
clock: 66MHz | |
capabilities: storage msi pm ahci_1.0 bus_master cap_list | |
configuration: driver=ahci latency=0 | |
resources: irq:42 ioport:f070(size=8) ioport:f060(size=4) ioport:f050(size=8) ioport:f040(size=4) ioport:f000(size=32) memory:fb336000-fb3367ff | |
*-scsi:0 | |
physical id: 1 | |
logical name: scsi0 | |
capabilities: emulated | |
*-disk | |
description: ATA Disk | |
product: Crucial_CT512MX1 | |
physical id: 0.0.0 | |
bus info: scsi@0:0.0.0 | |
logical name: /dev/sda | |
version: MU01 | |
serial: 14340CFD629E | |
size: 476GiB (512GB) | |
capabilities: gpt-1.00 partitioned partitioned:gpt | |
configuration: ansiversion=5 guid=75bae818-9153-47eb-9cee-c01f70c5c42b logicalsectorsize=512 sectorsize=4096 | |
*-scsi:1 | |
physical id: 2 | |
logical name: scsi1 | |
capabilities: emulated | |
*-disk | |
description: ATA Disk | |
product: Crucial_CT512MX1 | |
physical id: 0.0.0 | |
bus info: scsi@1:0.0.0 | |
logical name: /dev/sdb | |
version: MU01 | |
serial: 14330CF93238 | |
size: 476GiB (512GB) | |
capabilities: gpt-1.00 partitioned partitioned:gpt | |
configuration: ansiversion=5 guid=b3e05cb9-ac11-46b2-a41b-5cf4164bcb34 logicalsectorsize=512 sectorsize=4096 | |
*-scsi:2 | |
physical id: 3 | |
logical name: scsi2 | |
capabilities: emulated | |
*-disk | |
description: ATA Disk | |
product: ST2000DM006-2DM1 | |
vendor: Seagate | |
physical id: 0.0.0 | |
bus info: scsi@2:0.0.0 | |
logical name: /dev/sdc | |
version: CC26 | |
serial: Z4Z83385 | |
size: 1863GiB (2TB) | |
capabilities: gpt-1.00 partitioned partitioned:gpt | |
configuration: ansiversion=5 guid=86fe1a86-58ce-448b-bb84-1461f00ab21f logicalsectorsize=512 sectorsize=4096 | |
*-scsi:3 | |
physical id: 4 | |
logical name: scsi3 | |
capabilities: emulated | |
*-disk | |
description: ATA Disk | |
product: ST2000DM006-2DM1 | |
vendor: Seagate | |
physical id: 0.0.0 | |
bus info: scsi@3:0.0.0 | |
logical name: /dev/sdd | |
version: CC26 | |
serial: Z4ZB7EV9 | |
size: 1863GiB (2TB) | |
capabilities: gpt-1.00 partitioned partitioned:gpt | |
configuration: ansiversion=5 guid=fa606d24-f1b6-4661-ab08-41e3bae03746 logicalsectorsize=512 sectorsize=4096 | |
➜ ~ dstat | |
You did not select any stats, using -cdngy by default. | |
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- | |
usr sys idl wai hiq siq| read writ| recv send| in out | int csw | |
7 1 90 2 0 0| 29M 6188k| 0 0 | 0 0 |2893 14k | |
0 0 100 0 0 0| 0 4096B| 309B 0 | 0 0 | 48 191 | |
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 28 60 | |
0 0 100 0 0 0| 0 0 | 472B 191B| 0 0 | 23 46 | |
0 0 100 0 0 0| 0 0 | 635B 382B| 0 0 | 100 176 | |
0 0 100 0 0 0| 0 0 | 472B 191B| 0 0 | 37 75 | |
0 0 100 0 0 0| 0 96k| 309B 0 | 0 0 | 39 173 | |
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 20 44 | |
0 0 100 0 0 0| 48k 0 | 472B 408B| 0 0 | 103 192 | |
0 0 100 0 0 0| 0 168k| 309B 217B| 0 0 | 70 311 | |
0 0 100 0 0 0| 0 0 | 309B 217B| 0 0 | 50 229 | |
0 0 100 0 0 0| 0 8192B| 399B 217B| 0 0 | 109 441 | |
0 0 100 0 0 0|4096B 0 | 309B 0 | 0 0 | 57 174 | |
1 1 98 0 0 0|1300k 124k| 309B 0 | 0 0 | 516 703 | |
0 0 100 0 0 0| 0 0 | 395B 89B| 0 0 | 50 126 | |
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 29 63 | |
0 0 100 0 0 0| 0 168k| 309B 0 | 0 0 | 76 89 | |
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 22 99 | |
0 0 100 0 0 0| 0 48k| 424B 143B| 0 0 | 40 110 | |
0 0 100 0 0 0| 0 0 |2198B 82B| 0 0 | 40 152 | |
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 70 283 ^C | |
➜ ~ lsblk | |
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT | |
sda 8:0 0 477G 0 disk | |
├─sda1 8:1 0 512M 0 part | |
├─sda2 8:2 0 1M 0 part | |
├─sda3 8:3 0 15.9G 0 part [SWAP] | |
└─sda4 8:4 0 460.6G 0 part / | |
sdb 8:16 0 477G 0 disk | |
└─sdb1 8:17 0 477G 0 part | |
sdc 8:32 0 1.8T 0 disk | |
├─sdc1 8:33 0 128M 0 part | |
└─sdc2 8:34 0 1.8T 0 part | |
sdd 8:48 0 1.8T 0 disk | |
└─sdd1 8:49 0 1.8T 0 part /mnt/data | |
➜ ~ cd src/LocustDB | |
➜ LocustDB git:(master) ulimit -n 4096 | |
➜ LocustDB git:(master) rm -r /mnt/data/ldb; RUSTFLAGS="-Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --load /mnt/data/nyc-taxi-data-denormalized/trips_x* --trips --db-path /mnt/data/ldb --table trips --mem-limit-tables 0 | |
➜ LocustDB git:(master) git rev-parse HEAD | |
fc838440384d2db4689b44caf851ef67ea5e09f7 | |
➜ LocustDB git:(master) rm -r /mnt/data/ldb; RUSTFLAGS="-Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --load /mnt/data/nyc-taxi-data-denormalized/trips_x* --trips --db-path /mnt/data/ldb --table trips --mem-limit-tables | |
Compiling locustdb v0.1.0-alpha (file:///home/clemens/src/LocustDB) | |
Finished release [optimized] target(s) in 1m 03s | |
Running `target/release/repl --load /mnt/data/nyc-taxi-data-denormalized/trips_xaa.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xab.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xac.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xad.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xae.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xag.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xah.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xai.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xak.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xal.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xam.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xan.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xao.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xap.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xar.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xas.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xat.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xau.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xav.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaw.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xax.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xay.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaz.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xba.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbb.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbc.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbd.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbe.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbg.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbh.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbi.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbk.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbl.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbm.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbn.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbo.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbp.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbr.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbs.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbt.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbu.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbv.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbw.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbx.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xby.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbz.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xca.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcb.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcc.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcd.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xce.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcg.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xch.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xci.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xck.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcl.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcm.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcn.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xco.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcp.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcr.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcs.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xct.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcu.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcv.csv.gz --trips --db-path /mnt/data/ldb --table trips --mem-limit- | |
Loading 74 files into table trips. | |
Loaded data in 8.18ks. | |
# Table `trips` (1464785771 rows, 8.00GiB) # | |
dropoff_boroname: 8.5MiB | |
dropoff_ntacode: 21MiB | |
trip_type: 0.32MiB | |
dropoff_boroct2010: 36MiB | |
dropoff_borocode: 8.1MiB | |
min_temperature: 97MiB | |
pickup_latitude: 3.5MiB | |
store_and_fwd_flag: 13MiB | |
ehail_fee: 1.7MiB | |
improvement_surcharge: 3.2MiB | |
total_amount: 0.55GiB | |
pickup_ntacode: 100MiB | |
dropoff_latitude: 2.2MiB | |
passenger_count: 0.13GiB | |
dropoff: 0.97GiB | |
dropoff_puma: 19MiB | |
pickup_longitude: 35MiB | |
snowfall: 16MiB | |
pickup_ctlabel: 0.19GiB | |
dropoff_ct2010: 33MiB | |
vendor_id: 0.13GiB | |
payment_type: 0.13GiB | |
dropoff_longitude: 7.3MiB | |
extra: 93MiB | |
pickup_nyct2010_gid: 0.19GiB | |
cab_type: 2.1MiB | |
pickup_ct2010: 0.23GiB | |
trip_id: 8.3MiB | |
dropoff_datetime: 0.90GiB | |
dropoff_ctlabel: 24MiB | |
tolls_amount: 64MiB | |
max_temperature: 97MiB | |
pickup_ntaname: 0.10GiB | |
trip_distance: 0.59GiB | |
dropoff_cdeligibil: 10.0MiB | |
precipitation: 77MiB | |
pickup_puma: 97MiB | |
tip_amount: 0.39GiB | |
pickup_boroct2010: 0.23GiB | |
dropoff_nyct2010_gid: 25MiB | |
average_wind_speed: 0.13GiB | |
dropoff_ntaname: 26MiB | |
rate_code_id: 32MiB | |
pickup_borocode: 24MiB | |
pickup_cdeligibil: 36MiB | |
pickup: 0.97GiB | |
snow_depth: 16MiB | |
fare_amount: 0.46GiB | |
mta_tax: 7.8MiB | |
pickup_boroname: 25MiB | |
pickup_datetime: 0.64GiB | |
# Table `_meta_tables` (1 rows, 8.00B) # | |
name: 0.00B | |
timestamp: 0.00B | |
locustdb> SELECT passenger_count, trip_distance, total_amount FROM trips ORDER BY total_amount DESC LIMIT 100; | |
thread '<unnamed>' panicked at 'no entry found for key', libcore/option.rs:1000:5 | |
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. | |
stack backtrace: | |
0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace | |
at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49 | |
1: std::sys_common::backtrace::print | |
at libstd/sys_common/backtrace.rs:71 | |
at libstd/sys_common/backtrace.rs:59 | |
2: std::panicking::default_hook::{{closure}} | |
at libstd/panicking.rs:211 | |
3: std::panicking::default_hook | |
at libstd/panicking.rs:227 | |
4: std::panicking::rust_panic_with_hook | |
at libstd/panicking.rs:475 | |
5: std::panicking::continue_panic_fmt | |
at libstd/panicking.rs:390 | |
6: rust_begin_unwind | |
at libstd/panicking.rs:325 | |
7: core::panicking::panic_fmt | |
at libcore/panicking.rs:77 | |
8: core::option::expect_failed | |
at libcore/option.rs:1000 | |
9: locustdb::scheduler::inner_locustdb::InnerLocustDB::restore | |
10: <locustdb::disk_store::rocksdb::RocksDB as locustdb::disk_store::interface::DiskStore>::load_column_range | |
11: locustdb::scheduler::disk_read_scheduler::DiskReadScheduler::service_reads | |
12: <locustdb::scheduler::task::FnTask<F, T> as locustdb::scheduler::task::Task>::execute | |
^C | |
➜ LocustDB git:(master) | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment