locustdb-ingestion-hdd

➜  ~ uname -a
Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
➜  ~ sudo lshw -class disk -class storage
[sudo] password for clemens: 
  *-storage               
       description: SATA controller
       product: ASM1062 Serial ATA Controller
       vendor: ASMedia Technology Inc.
       physical id: 0
       bus info: pci@0000:06:00.0
       version: 02
       width: 32 bits
       clock: 33MHz
       capabilities: storage msi pm pciexpress ahci_1.0 bus_master cap_list
       configuration: driver=ahci latency=0
       resources: irq:43 ioport:d050(size=8) ioport:d040(size=4) ioport:d030(size=8) ioport:d020(size=4) ioport:d000(size=32) memory:fb200000-fb2001ff
  *-storage
       description: SATA controller
       product: C610/X99 series chipset 6-Port SATA Controller [AHCI mode]
       vendor: Intel Corporation
       physical id: 1f.2
       bus info: pci@0000:00:1f.2
       version: 05
       width: 32 bits
       clock: 66MHz
       capabilities: storage msi pm ahci_1.0 bus_master cap_list
       configuration: driver=ahci latency=0
       resources: irq:42 ioport:f070(size=8) ioport:f060(size=4) ioport:f050(size=8) ioport:f040(size=4) ioport:f000(size=32) memory:fb336000-fb3367ff
  *-scsi:0
       physical id: 1
       logical name: scsi0
       capabilities: emulated
     *-disk
          description: ATA Disk
          product: Crucial_CT512MX1
          physical id: 0.0.0
          bus info: scsi@0:0.0.0
          logical name: /dev/sda
          version: MU01
          serial: 14340CFD629E
          size: 476GiB (512GB)
          capabilities: gpt-1.00 partitioned partitioned:gpt
          configuration: ansiversion=5 guid=75bae818-9153-47eb-9cee-c01f70c5c42b logicalsectorsize=512 sectorsize=4096
  *-scsi:1
       physical id: 2
       logical name: scsi1
       capabilities: emulated
     *-disk
          description: ATA Disk
          product: Crucial_CT512MX1
          physical id: 0.0.0
          bus info: scsi@1:0.0.0
          logical name: /dev/sdb
          version: MU01
          serial: 14330CF93238
          size: 476GiB (512GB)
          capabilities: gpt-1.00 partitioned partitioned:gpt
          configuration: ansiversion=5 guid=b3e05cb9-ac11-46b2-a41b-5cf4164bcb34 logicalsectorsize=512 sectorsize=4096
  *-scsi:2
       physical id: 3
       logical name: scsi2
       capabilities: emulated
     *-disk
          description: ATA Disk
          product: ST2000DM006-2DM1
          vendor: Seagate
          physical id: 0.0.0
          bus info: scsi@2:0.0.0
          logical name: /dev/sdc
          version: CC26
          serial: Z4Z83385
          size: 1863GiB (2TB)
          capabilities: gpt-1.00 partitioned partitioned:gpt
          configuration: ansiversion=5 guid=86fe1a86-58ce-448b-bb84-1461f00ab21f logicalsectorsize=512 sectorsize=4096
  *-scsi:3
       physical id: 4
       logical name: scsi3
       capabilities: emulated
     *-disk
          description: ATA Disk
          product: ST2000DM006-2DM1
          vendor: Seagate
          physical id: 0.0.0
          bus info: scsi@3:0.0.0
          logical name: /dev/sdd
          version: CC26
          serial: Z4ZB7EV9
          size: 1863GiB (2TB)
          capabilities: gpt-1.00 partitioned partitioned:gpt
          configuration: ansiversion=5 guid=fa606d24-f1b6-4661-ab08-41e3bae03746 logicalsectorsize=512 sectorsize=4096
➜  ~ dstat                               
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  7   1  90   2   0   0|  29M 6188k|   0     0 |   0     0 |2893    14k
  0   0 100   0   0   0|   0  4096B| 309B    0 |   0     0 |  48   191 
  0   0 100   0   0   0|   0     0 | 309B    0 |   0     0 |  28    60 
  0   0 100   0   0   0|   0     0 | 472B  191B|   0     0 |  23    46 
  0   0 100   0   0   0|   0     0 | 635B  382B|   0     0 | 100   176 
  0   0 100   0   0   0|   0     0 | 472B  191B|   0     0 |  37    75 
  0   0 100   0   0   0|   0    96k| 309B    0 |   0     0 |  39   173 
  0   0 100   0   0   0|   0     0 | 309B    0 |   0     0 |  20    44 
  0   0 100   0   0   0|  48k    0 | 472B  408B|   0     0 | 103   192 
  0   0 100   0   0   0|   0   168k| 309B  217B|   0     0 |  70   311 
  0   0 100   0   0   0|   0     0 | 309B  217B|   0     0 |  50   229 
  0   0 100   0   0   0|   0  8192B| 399B  217B|   0     0 | 109   441 
  0   0 100   0   0   0|4096B    0 | 309B    0 |   0     0 |  57   174 
  1   1  98   0   0   0|1300k  124k| 309B    0 |   0     0 | 516   703 
  0   0 100   0   0   0|   0     0 | 395B   89B|   0     0 |  50   126 
  0   0 100   0   0   0|   0     0 | 309B    0 |   0     0 |  29    63 
  0   0 100   0   0   0|   0   168k| 309B    0 |   0     0 |  76    89 
  0   0 100   0   0   0|   0     0 | 309B    0 |   0     0 |  22    99 
  0   0 100   0   0   0|   0    48k| 424B  143B|   0     0 |  40   110 
  0   0 100   0   0   0|   0     0 |2198B   82B|   0     0 |  40   152 
  0   0 100   0   0   0|   0     0 | 309B    0 |   0     0 |  70   283 ^C
➜  ~ lsblk                               
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   477G  0 disk 
├─sda1   8:1    0   512M  0 part 
├─sda2   8:2    0     1M  0 part 
├─sda3   8:3    0  15.9G  0 part [SWAP]
└─sda4   8:4    0 460.6G  0 part /
sdb      8:16   0   477G  0 disk 
└─sdb1   8:17   0   477G  0 part 
sdc      8:32   0   1.8T  0 disk 
├─sdc1   8:33   0   128M  0 part 
└─sdc2   8:34   0   1.8T  0 part 
sdd      8:48   0   1.8T  0 disk 
└─sdd1   8:49   0   1.8T  0 part /mnt/data
➜  ~ cd src/LocustDB 
➜  LocustDB git:(master) ulimit -n 4096
➜  LocustDB git:(master) rm -r /mnt/data/ldb; RUSTFLAGS="-Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --load /mnt/data/nyc-taxi-data-denormalized/trips_x* --trips --db-path /mnt/data/ldb --table trips --mem-limit-tables 0            
➜  LocustDB git:(master) git rev-parse HEAD
fc838440384d2db4689b44caf851ef67ea5e09f7
➜  LocustDB git:(master) rm -r /mnt/data/ldb; RUSTFLAGS="-Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --load /mnt/data/nyc-taxi-data-denormalized/trips_x* --trips --db-path /mnt/data/ldb --table trips --mem-limit-tables              
   Compiling locustdb v0.1.0-alpha (file:///home/clemens/src/LocustDB)                                          
    Finished release [optimized] target(s) in 1m 03s
     Running `target/release/repl --load /mnt/data/nyc-taxi-data-denormalized/trips_xaa.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xab.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xac.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xad.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xae.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xag.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xah.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xai.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xak.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xal.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xam.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xan.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xao.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xap.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xar.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xas.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xat.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xau.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xav.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaw.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xax.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xay.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaz.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xba.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbb.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbc.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbd.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbe.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbg.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbh.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbi.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbk.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbl.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbm.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbn.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbo.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbp.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbr.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbs.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbt.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbu.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbv.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbw.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbx.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xby.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbz.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xca.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcb.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcc.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcd.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xce.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcg.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xch.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xci.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xck.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcl.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcm.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcn.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xco.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcp.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcr.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcs.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xct.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcu.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcv.csv.gz --trips --db-path /mnt/data/ldb --table trips --mem-limit-
Loading 74 files into table trips.
Loaded data in 8.18ks.

# Table `trips` (1464785771 rows, 8.00GiB) #
dropoff_boroname: 8.5MiB
dropoff_ntacode: 21MiB
trip_type: 0.32MiB
dropoff_boroct2010: 36MiB
dropoff_borocode: 8.1MiB
min_temperature: 97MiB
pickup_latitude: 3.5MiB
store_and_fwd_flag: 13MiB
ehail_fee: 1.7MiB
improvement_surcharge: 3.2MiB
total_amount: 0.55GiB
pickup_ntacode: 100MiB
dropoff_latitude: 2.2MiB
passenger_count: 0.13GiB
dropoff: 0.97GiB
dropoff_puma: 19MiB
pickup_longitude: 35MiB
snowfall: 16MiB
pickup_ctlabel: 0.19GiB
dropoff_ct2010: 33MiB
vendor_id: 0.13GiB
payment_type: 0.13GiB
dropoff_longitude: 7.3MiB
extra: 93MiB
pickup_nyct2010_gid: 0.19GiB
cab_type: 2.1MiB
pickup_ct2010: 0.23GiB
trip_id: 8.3MiB
dropoff_datetime: 0.90GiB
dropoff_ctlabel: 24MiB
tolls_amount: 64MiB
max_temperature: 97MiB
pickup_ntaname: 0.10GiB
trip_distance: 0.59GiB
dropoff_cdeligibil: 10.0MiB
precipitation: 77MiB
pickup_puma: 97MiB
tip_amount: 0.39GiB
pickup_boroct2010: 0.23GiB
dropoff_nyct2010_gid: 25MiB
average_wind_speed: 0.13GiB
dropoff_ntaname: 26MiB
rate_code_id: 32MiB
pickup_borocode: 24MiB
pickup_cdeligibil: 36MiB
pickup: 0.97GiB
snow_depth: 16MiB
fare_amount: 0.46GiB
mta_tax: 7.8MiB
pickup_boroname: 25MiB
pickup_datetime: 0.64GiB

# Table `_meta_tables` (1 rows, 8.00B) #
name: 0.00B
timestamp: 0.00B
locustdb> SELECT passenger_count, trip_distance, total_amount FROM trips ORDER BY total_amount DESC LIMIT 100;
thread '<unnamed>' panicked at 'no entry found for key', libcore/option.rs:1000:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:227
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:475
   5: std::panicking::continue_panic_fmt
             at libstd/panicking.rs:390
   6: rust_begin_unwind
             at libstd/panicking.rs:325
   7: core::panicking::panic_fmt
             at libcore/panicking.rs:77
   8: core::option::expect_failed
             at libcore/option.rs:1000
   9: locustdb::scheduler::inner_locustdb::InnerLocustDB::restore
  10: <locustdb::disk_store::rocksdb::RocksDB as locustdb::disk_store::interface::DiskStore>::load_column_range
  11: locustdb::scheduler::disk_read_scheduler::DiskReadScheduler::service_reads
  12: <locustdb::scheduler::task::FnTask<F, T> as locustdb::scheduler::task::Task>::execute
^C
➜  LocustDB git:(master)