Skip to content

Instantly share code, notes, and snippets.

@cswinter

cswinter/locustdb-ingestion-hdd

Created August 12, 2018 17:49
Star
Embed
What would you like to do?
Learn more about clone URLs
Download ZIP
Raw
locustdb-ingestion-hdd
➜ ~ uname -a
Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
➜ ~ sudo lshw -class disk -class storage
[sudo] password for clemens:
*-storage
description: SATA controller
product: ASM1062 Serial ATA Controller
vendor: ASMedia Technology Inc.
physical id: 0
bus info: pci@0000:06:00.0
version: 02
width: 32 bits
clock: 33MHz
capabilities: storage msi pm pciexpress ahci_1.0 bus_master cap_list
configuration: driver=ahci latency=0
resources: irq:43 ioport:d050(size=8) ioport:d040(size=4) ioport:d030(size=8) ioport:d020(size=4) ioport:d000(size=32) memory:fb200000-fb2001ff
*-storage
description: SATA controller
product: C610/X99 series chipset 6-Port SATA Controller [AHCI mode]
vendor: Intel Corporation
physical id: 1f.2
bus info: pci@0000:00:1f.2
version: 05
width: 32 bits
clock: 66MHz
capabilities: storage msi pm ahci_1.0 bus_master cap_list
configuration: driver=ahci latency=0
resources: irq:42 ioport:f070(size=8) ioport:f060(size=4) ioport:f050(size=8) ioport:f040(size=4) ioport:f000(size=32) memory:fb336000-fb3367ff
*-scsi:0
physical id: 1
logical name: scsi0
capabilities: emulated
*-disk
description: ATA Disk
product: Crucial_CT512MX1
physical id: 0.0.0
bus info: scsi@0:0.0.0
logical name: /dev/sda
version: MU01
serial: 14340CFD629E
size: 476GiB (512GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=75bae818-9153-47eb-9cee-c01f70c5c42b logicalsectorsize=512 sectorsize=4096
*-scsi:1
physical id: 2
logical name: scsi1
capabilities: emulated
*-disk
description: ATA Disk
product: Crucial_CT512MX1
physical id: 0.0.0
bus info: scsi@1:0.0.0
logical name: /dev/sdb
version: MU01
serial: 14330CF93238
size: 476GiB (512GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=b3e05cb9-ac11-46b2-a41b-5cf4164bcb34 logicalsectorsize=512 sectorsize=4096
*-scsi:2
physical id: 3
logical name: scsi2
capabilities: emulated
*-disk
description: ATA Disk
product: ST2000DM006-2DM1
vendor: Seagate
physical id: 0.0.0
bus info: scsi@2:0.0.0
logical name: /dev/sdc
version: CC26
serial: Z4Z83385
size: 1863GiB (2TB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=86fe1a86-58ce-448b-bb84-1461f00ab21f logicalsectorsize=512 sectorsize=4096
*-scsi:3
physical id: 4
logical name: scsi3
capabilities: emulated
*-disk
description: ATA Disk
product: ST2000DM006-2DM1
vendor: Seagate
physical id: 0.0.0
bus info: scsi@3:0.0.0
logical name: /dev/sdd
version: CC26
serial: Z4ZB7EV9
size: 1863GiB (2TB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=fa606d24-f1b6-4661-ab08-41e3bae03746 logicalsectorsize=512 sectorsize=4096
➜ ~ dstat
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
7 1 90 2 0 0| 29M 6188k| 0 0 | 0 0 |2893 14k
0 0 100 0 0 0| 0 4096B| 309B 0 | 0 0 | 48 191
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 28 60
0 0 100 0 0 0| 0 0 | 472B 191B| 0 0 | 23 46
0 0 100 0 0 0| 0 0 | 635B 382B| 0 0 | 100 176
0 0 100 0 0 0| 0 0 | 472B 191B| 0 0 | 37 75
0 0 100 0 0 0| 0 96k| 309B 0 | 0 0 | 39 173
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 20 44
0 0 100 0 0 0| 48k 0 | 472B 408B| 0 0 | 103 192
0 0 100 0 0 0| 0 168k| 309B 217B| 0 0 | 70 311
0 0 100 0 0 0| 0 0 | 309B 217B| 0 0 | 50 229
0 0 100 0 0 0| 0 8192B| 399B 217B| 0 0 | 109 441
0 0 100 0 0 0|4096B 0 | 309B 0 | 0 0 | 57 174
1 1 98 0 0 0|1300k 124k| 309B 0 | 0 0 | 516 703
0 0 100 0 0 0| 0 0 | 395B 89B| 0 0 | 50 126
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 29 63
0 0 100 0 0 0| 0 168k| 309B 0 | 0 0 | 76 89
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 22 99
0 0 100 0 0 0| 0 48k| 424B 143B| 0 0 | 40 110
0 0 100 0 0 0| 0 0 |2198B 82B| 0 0 | 40 152
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 70 283 ^C
➜ ~ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 477G 0 disk
├─sda1 8:1 0 512M 0 part
├─sda2 8:2 0 1M 0 part
├─sda3 8:3 0 15.9G 0 part [SWAP]
└─sda4 8:4 0 460.6G 0 part /
sdb 8:16 0 477G 0 disk
└─sdb1 8:17 0 477G 0 part
sdc 8:32 0 1.8T 0 disk
├─sdc1 8:33 0 128M 0 part
└─sdc2 8:34 0 1.8T 0 part
sdd 8:48 0 1.8T 0 disk
└─sdd1 8:49 0 1.8T 0 part /mnt/data
➜ ~ cd src/LocustDB
➜ LocustDB git:(master) ulimit -n 4096
➜ LocustDB git:(master) rm -r /mnt/data/ldb; RUSTFLAGS="-Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --load /mnt/data/nyc-taxi-data-denormalized/trips_x* --trips --db-path /mnt/data/ldb --table trips --mem-limit-tables 0
➜ LocustDB git:(master) git rev-parse HEAD
fc838440384d2db4689b44caf851ef67ea5e09f7
➜ LocustDB git:(master) rm -r /mnt/data/ldb; RUSTFLAGS="-Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --load /mnt/data/nyc-taxi-data-denormalized/trips_x* --trips --db-path /mnt/data/ldb --table trips --mem-limit-tables
Compiling locustdb v0.1.0-alpha (file:///home/clemens/src/LocustDB)
Finished release [optimized] target(s) in 1m 03s
Running `target/release/repl --load /mnt/data/nyc-taxi-data-denormalized/trips_xaa.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xab.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xac.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xad.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xae.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xag.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xah.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xai.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xak.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xal.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xam.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xan.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xao.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xap.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xar.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xas.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xat.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xau.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xav.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaw.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xax.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xay.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xaz.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xba.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbb.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbc.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbd.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbe.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbg.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbh.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbi.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbk.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbl.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbm.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbn.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbo.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbp.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbr.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbs.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbt.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbu.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbv.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbw.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbx.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xby.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xbz.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xca.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcb.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcc.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcd.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xce.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcf.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcg.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xch.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xci.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcj.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xck.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcl.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcm.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcn.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xco.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcp.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcq.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcr.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcs.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xct.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcu.csv.gz /mnt/data/nyc-taxi-data-denormalized/trips_xcv.csv.gz --trips --db-path /mnt/data/ldb --table trips --mem-limit-
Loading 74 files into table trips.
Loaded data in 8.18ks.
# Table `trips` (1464785771 rows, 8.00GiB) #
dropoff_boroname: 8.5MiB
dropoff_ntacode: 21MiB
trip_type: 0.32MiB
dropoff_boroct2010: 36MiB
dropoff_borocode: 8.1MiB
min_temperature: 97MiB
pickup_latitude: 3.5MiB
store_and_fwd_flag: 13MiB
ehail_fee: 1.7MiB
improvement_surcharge: 3.2MiB
total_amount: 0.55GiB
pickup_ntacode: 100MiB
dropoff_latitude: 2.2MiB
passenger_count: 0.13GiB
dropoff: 0.97GiB
dropoff_puma: 19MiB
pickup_longitude: 35MiB
snowfall: 16MiB
pickup_ctlabel: 0.19GiB
dropoff_ct2010: 33MiB
vendor_id: 0.13GiB
payment_type: 0.13GiB
dropoff_longitude: 7.3MiB
extra: 93MiB
pickup_nyct2010_gid: 0.19GiB
cab_type: 2.1MiB
pickup_ct2010: 0.23GiB
trip_id: 8.3MiB
dropoff_datetime: 0.90GiB
dropoff_ctlabel: 24MiB
tolls_amount: 64MiB
max_temperature: 97MiB
pickup_ntaname: 0.10GiB
trip_distance: 0.59GiB
dropoff_cdeligibil: 10.0MiB
precipitation: 77MiB
pickup_puma: 97MiB
tip_amount: 0.39GiB
pickup_boroct2010: 0.23GiB
dropoff_nyct2010_gid: 25MiB
average_wind_speed: 0.13GiB
dropoff_ntaname: 26MiB
rate_code_id: 32MiB
pickup_borocode: 24MiB
pickup_cdeligibil: 36MiB
pickup: 0.97GiB
snow_depth: 16MiB
fare_amount: 0.46GiB
mta_tax: 7.8MiB
pickup_boroname: 25MiB
pickup_datetime: 0.64GiB
# Table `_meta_tables` (1 rows, 8.00B) #
name: 0.00B
timestamp: 0.00B
locustdb> SELECT passenger_count, trip_distance, total_amount FROM trips ORDER BY total_amount DESC LIMIT 100;
thread '<unnamed>' panicked at 'no entry found for key', libcore/option.rs:1000:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
1: std::sys_common::backtrace::print
at libstd/sys_common/backtrace.rs:71
at libstd/sys_common/backtrace.rs:59
2: std::panicking::default_hook::{{closure}}
at libstd/panicking.rs:211
3: std::panicking::default_hook
at libstd/panicking.rs:227
4: std::panicking::rust_panic_with_hook
at libstd/panicking.rs:475
5: std::panicking::continue_panic_fmt
at libstd/panicking.rs:390
6: rust_begin_unwind
at libstd/panicking.rs:325
7: core::panicking::panic_fmt
at libcore/panicking.rs:77
8: core::option::expect_failed
at libcore/option.rs:1000
9: locustdb::scheduler::inner_locustdb::InnerLocustDB::restore
10: <locustdb::disk_store::rocksdb::RocksDB as locustdb::disk_store::interface::DiskStore>::load_column_range
11: locustdb::scheduler::disk_read_scheduler::DiskReadScheduler::service_reads
12: <locustdb::scheduler::task::FnTask<F, T> as locustdb::scheduler::task::Task>::execute
^C
➜ LocustDB git:(master)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment