Skip to content

Instantly share code, notes, and snippets.

@cswinter

cswinter/locustdb-cold-query-hdd

Created August 12, 2018 17:42
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save cswinter/c8838d5951da85feb85a13a77ceaa3b0 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save cswinter/c8838d5951da85feb85a13a77ceaa3b0 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
locustdb-cold-query-hdd
➜ ~ uname -a
Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
➜ ~ sudo lshw -class disk -class storage
[sudo] password for clemens:
*-storage
description: SATA controller
product: ASM1062 Serial ATA Controller
vendor: ASMedia Technology Inc.
physical id: 0
bus info: pci@0000:06:00.0
version: 02
width: 32 bits
clock: 33MHz
capabilities: storage msi pm pciexpress ahci_1.0 bus_master cap_list
configuration: driver=ahci latency=0
resources: irq:43 ioport:d050(size=8) ioport:d040(size=4) ioport:d030(size=8) ioport:d020(size=4) ioport:d000(size=32) memory:fb200000-fb2001ff
*-storage
description: SATA controller
product: C610/X99 series chipset 6-Port SATA Controller [AHCI mode]
vendor: Intel Corporation
physical id: 1f.2
bus info: pci@0000:00:1f.2
version: 05
width: 32 bits
clock: 66MHz
capabilities: storage msi pm ahci_1.0 bus_master cap_list
configuration: driver=ahci latency=0
resources: irq:42 ioport:f070(size=8) ioport:f060(size=4) ioport:f050(size=8) ioport:f040(size=4) ioport:f000(size=32) memory:fb336000-fb3367ff
*-scsi:0
physical id: 1
logical name: scsi0
capabilities: emulated
*-disk
description: ATA Disk
product: Crucial_CT512MX1
physical id: 0.0.0
bus info: scsi@0:0.0.0
logical name: /dev/sda
version: MU01
serial: 14340CFD629E
size: 476GiB (512GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=75bae818-9153-47eb-9cee-c01f70c5c42b logicalsectorsize=512 sectorsize=4096
*-scsi:1
physical id: 2
logical name: scsi1
capabilities: emulated
*-disk
description: ATA Disk
product: Crucial_CT512MX1
physical id: 0.0.0
bus info: scsi@1:0.0.0
logical name: /dev/sdb
version: MU01
serial: 14330CF93238
size: 476GiB (512GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=b3e05cb9-ac11-46b2-a41b-5cf4164bcb34 logicalsectorsize=512 sectorsize=4096
*-scsi:2
physical id: 3
logical name: scsi2
capabilities: emulated
*-disk
description: ATA Disk
product: ST2000DM006-2DM1
vendor: Seagate
physical id: 0.0.0
bus info: scsi@2:0.0.0
logical name: /dev/sdc
version: CC26
serial: Z4Z83385
size: 1863GiB (2TB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=86fe1a86-58ce-448b-bb84-1461f00ab21f logicalsectorsize=512 sectorsize=4096
*-scsi:3
physical id: 4
logical name: scsi3
capabilities: emulated
*-disk
description: ATA Disk
product: ST2000DM006-2DM1
vendor: Seagate
physical id: 0.0.0
bus info: scsi@3:0.0.0
logical name: /dev/sdd
version: CC26
serial: Z4ZB7EV9
size: 1863GiB (2TB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=fa606d24-f1b6-4661-ab08-41e3bae03746 logicalsectorsize=512 sectorsize=4096
➜ ~ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 477G 0 disk
├─sda1 8:1 0 512M 0 part
├─sda2 8:2 0 1M 0 part
├─sda3 8:3 0 15.9G 0 part [SWAP]
└─sda4 8:4 0 460.6G 0 part /
sdb 8:16 0 477G 0 disk
└─sdb1 8:17 0 477G 0 part
sdc 8:32 0 1.8T 0 disk
├─sdc1 8:33 0 128M 0 part
└─sdc2 8:34 0 1.8T 0 part
sdd 8:48 0 1.8T 0 disk
└─sdd1 8:49 0 1.8T 0 part /mnt/data
➜ ~ ulimit -n 4096
➜ ~ dstat
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
5 1 86 8 0 0| 109M 32k| 0 0 | 0 0 |1776 2957
0 0 100 0 0 0| 0 36k| 309B 0 | 0 0 | 27 71
0 0 100 0 0 0| 0 0 | 309B 64B| 0 0 | 38 76
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 17 36
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 22 55
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 22 50
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 29 60
0 0 100 0 0 0| 0 0 |2108B 0 | 0 0 | 20 42
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 29 127
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 30 106
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 33 84 ^C
➜ ~ cd src/LocustDB
➜ LocustDB git:(master) git rev-parse HEAD
573557575140c94e0aa3a0576ece04beb718712c
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.43s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
pickup_latitude: 0.00B
trip_distance: 0.00B
snow_depth: 0.00B
ehail_fee: 0.00B
max_temperature: 0.00B
dropoff_ctlabel: 0.00B
passenger_count: 0.00B
pickup: 0.00B
dropoff_borocode: 0.00B
improvement_surcharge: 0.00B
dropoff_boroname: 0.00B
dropoff_nyct2010_gid: 0.00B
dropoff_boroct2010: 0.00B
pickup_cdeligibil: 0.00B
trip_id: 0.00B
dropoff_cdeligibil: 0.00B
dropoff: 0.00B
pickup_puma: 0.00B
precipitation: 0.00B
pickup_datetime: 0.00B
rate_code_id: 0.00B
dropoff_ntaname: 0.00B
pickup_borocode: 0.00B
pickup_longitude: 0.00B
dropoff_puma: 0.00B
pickup_ct2010: 0.00B
pickup_ctlabel: 0.00B
dropoff_longitude: 0.00B
mta_tax: 0.00B
payment_type: 0.00B
pickup_boroct2010: 0.00B
extra: 0.00B
total_amount: 0.00B
dropoff_ct2010: 0.00B
average_wind_speed: 0.00B
pickup_nyct2010_gid: 0.00B
dropoff_datetime: 0.00B
dropoff_latitude: 0.00B
dropoff_ntacode: 0.00B
pickup_ntacode: 0.00B
pickup_ntaname: 0.00B
cab_type: 0.00B
tolls_amount: 0.00B
trip_type: 0.00B
fare_amount: 0.00B
store_and_fwd_flag: 0.00B
pickup_boroname: 0.00B
min_temperature: 0.00B
vendor_id: 0.00B
tip_amount: 0.00B
snowfall: 0.00B
locustdb> SELECT cab_type, count(0) FROM trips;
Scanned 1.46 billion rows in 784ms (1.9 billion rows/s)!
cab_type | count_0
---------+------------
"green" | 64407776
"yellow" | 1400377995
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.37s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
total_amount: 0.00B
min_temperature: 0.00B
pickup_ntaname: 0.00B
dropoff_puma: 0.00B
tolls_amount: 0.00B
dropoff: 0.00B
pickup_longitude: 0.00B
pickup_ntacode: 0.00B
trip_id: 0.00B
max_temperature: 0.00B
pickup_ct2010: 0.00B
pickup_borocode: 0.00B
fare_amount: 0.00B
pickup_puma: 0.00B
trip_distance: 0.00B
average_wind_speed: 0.00B
ehail_fee: 0.00B
trip_type: 0.00B
precipitation: 0.00B
dropoff_boroct2010: 0.00B
pickup: 0.00B
dropoff_ntacode: 0.00B
dropoff_ntaname: 0.00B
pickup_ctlabel: 0.00B
dropoff_borocode: 0.00B
store_and_fwd_flag: 0.00B
payment_type: 0.00B
dropoff_nyct2010_gid: 0.00B
rate_code_id: 0.00B
dropoff_datetime: 0.00B
pickup_cdeligibil: 0.00B
vendor_id: 0.00B
dropoff_latitude: 0.00B
dropoff_boroname: 0.00B
improvement_surcharge: 0.00B
extra: 0.00B
tip_amount: 0.00B
dropoff_cdeligibil: 0.00B
pickup_boroct2010: 0.00B
pickup_datetime: 0.00B
mta_tax: 0.00B
pickup_nyct2010_gid: 0.00B
snow_depth: 0.00B
snowfall: 0.00B
passenger_count: 0.00B
cab_type: 0.00B
pickup_boroname: 0.00B
dropoff_longitude: 0.00B
dropoff_ctlabel: 0.00B
dropoff_ct2010: 0.00B
pickup_latitude: 0.00B
locustdb> SELECT cab_type, count(0) FROM trips;
Scanned 1.46 billion rows in 798ms (1.8 billion rows/s)!
cab_type | count_0
---------+------------
"green" | 64407776
"yellow" | 1400377995
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
pickup_puma: 0.00B
dropoff_ntaname: 0.00B
pickup_datetime: 0.00B
pickup_cdeligibil: 0.00B
pickup_borocode: 0.00B
ehail_fee: 0.00B
payment_type: 0.00B
dropoff_ntacode: 0.00B
pickup_nyct2010_gid: 0.00B
pickup_boroname: 0.00B
dropoff_cdeligibil: 0.00B
dropoff_datetime: 0.00B
mta_tax: 0.00B
trip_type: 0.00B
dropoff_borocode: 0.00B
pickup_ntacode: 0.00B
dropoff_boroname: 0.00B
max_temperature: 0.00B
pickup_ntaname: 0.00B
trip_id: 0.00B
pickup_ctlabel: 0.00B
improvement_surcharge: 0.00B
vendor_id: 0.00B
fare_amount: 0.00B
cab_type: 0.00B
tolls_amount: 0.00B
pickup_latitude: 0.00B
min_temperature: 0.00B
total_amount: 0.00B
pickup_ct2010: 0.00B
dropoff_latitude: 0.00B
pickup_boroct2010: 0.00B
snow_depth: 0.00B
dropoff_boroct2010: 0.00B
store_and_fwd_flag: 0.00B
pickup: 0.00B
tip_amount: 0.00B
rate_code_id: 0.00B
dropoff_nyct2010_gid: 0.00B
snowfall: 0.00B
extra: 0.00B
dropoff_puma: 0.00B
dropoff_ctlabel: 0.00B
average_wind_speed: 0.00B
dropoff: 0.00B
passenger_count: 0.00B
trip_distance: 0.00B
precipitation: 0.00B
pickup_longitude: 0.00B
dropoff_ct2010: 0.00B
dropoff_longitude: 0.00B
locustdb> SELECT cab_type, count(0) FROM trips;
Scanned 1.46 billion rows in 773ms (1.9 billion rows/s)!
cab_type | count_0
---------+------------
"green" | 64407776
"yellow" | 1400377995
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
fare_amount: 0.00B
dropoff_cdeligibil: 0.00B
rate_code_id: 0.00B
dropoff_boroct2010: 0.00B
tolls_amount: 0.00B
cab_type: 0.00B
pickup_boroname: 0.00B
mta_tax: 0.00B
pickup_ntaname: 0.00B
snow_depth: 0.00B
dropoff_nyct2010_gid: 0.00B
pickup: 0.00B
trip_type: 0.00B
dropoff_boroname: 0.00B
dropoff_ctlabel: 0.00B
pickup_borocode: 0.00B
trip_id: 0.00B
pickup_longitude: 0.00B
store_and_fwd_flag: 0.00B
improvement_surcharge: 0.00B
pickup_ctlabel: 0.00B
dropoff_borocode: 0.00B
dropoff_ntacode: 0.00B
dropoff_latitude: 0.00B
dropoff_puma: 0.00B
total_amount: 0.00B
pickup_ntacode: 0.00B
pickup_puma: 0.00B
pickup_datetime: 0.00B
snowfall: 0.00B
extra: 0.00B
tip_amount: 0.00B
pickup_boroct2010: 0.00B
dropoff_ntaname: 0.00B
vendor_id: 0.00B
max_temperature: 0.00B
min_temperature: 0.00B
trip_distance: 0.00B
pickup_latitude: 0.00B
pickup_cdeligibil: 0.00B
ehail_fee: 0.00B
dropoff_datetime: 0.00B
passenger_count: 0.00B
dropoff: 0.00B
precipitation: 0.00B
dropoff_longitude: 0.00B
dropoff_ct2010: 0.00B
average_wind_speed: 0.00B
pickup_ct2010: 0.00B
payment_type: 0.00B
pickup_nyct2010_gid: 0.00B
locustdb> SELECT passenger_count, count(0), sum(total_amount) FROM trips;
Scanned 1.46 billion rows in 26.7s (55 million rows/s)!
passenger_count | count_0 | sum_1
----------------+------------+---------------
0 | 4107443 | 4509842206
1 | 1028187597 | 1425303818216
2 | 210577025 | 304718353489
3 | 62345389 | 87788492212
4 | 29994987 | 42514355874
5 | 95184622 | 130181002616
6 | 34380600 | 51063741303
7 | 2595 | 8693795
8 | 2525 | 8272055
9 | 1367 | 7196690
10 | 17 | 75470
13 | 1 | 3150
15 | 2 | 2410
17 | 1 | 3995
19 | 1 | 500
25 | 1 | 760
33 | 2 | 1719
34 | 1 | 1680
36 | 1 | 6155
37 | 1 | 1420
38 | 1 | 730
47 | 1 | 900
49 | 26 | 8440
53 | 1 | 730
58 | 2 | 5188
61 | 1 | 3134
65 | 3 | 7010
66 | 1 | 1930
69 | 1 | 580
70 | 1 | 1070
84 | 1 | 4384
97 | 1 | 990
113 | 1 | 1330
125 | 1 | 1660
129 | 7 | 6150
133 | 1 | 1030
134 | 1 | 5514
137 | 1 | 5964
141 | 1 | 1894
155 | 1 | 9024
158 | 1 | 1444
160 | 1 | 1534
163 | 1 | 1554
164 | 1 | 6214
165 | 1 | 1214
177 | 1 | 1700
192 | 1 | 953
193 | 1 | 750
208 | 1508 | 1062534
211 | 1 | 700
213 | 4 | 1000
223 | 1 | 950
225 | 1 | 1600
247 | 1 | 1944
249 | 1 | 950
250 | 3 | 3770
254 | 1 | 650
255 | 10 | 17990
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
rate_code_id: 0.00B
trip_distance: 0.00B
tolls_amount: 0.00B
store_and_fwd_flag: 0.00B
ehail_fee: 0.00B
dropoff_ct2010: 0.00B
pickup_ctlabel: 0.00B
extra: 0.00B
pickup: 0.00B
dropoff_nyct2010_gid: 0.00B
pickup_datetime: 0.00B
dropoff: 0.00B
pickup_nyct2010_gid: 0.00B
pickup_longitude: 0.00B
dropoff_ntaname: 0.00B
dropoff_borocode: 0.00B
pickup_puma: 0.00B
dropoff_latitude: 0.00B
snow_depth: 0.00B
dropoff_puma: 0.00B
average_wind_speed: 0.00B
tip_amount: 0.00B
dropoff_ctlabel: 0.00B
pickup_borocode: 0.00B
pickup_cdeligibil: 0.00B
dropoff_boroname: 0.00B
passenger_count: 0.00B
trip_type: 0.00B
pickup_ct2010: 0.00B
snowfall: 0.00B
fare_amount: 0.00B
pickup_boroname: 0.00B
pickup_latitude: 0.00B
max_temperature: 0.00B
mta_tax: 0.00B
pickup_boroct2010: 0.00B
improvement_surcharge: 0.00B
vendor_id: 0.00B
min_temperature: 0.00B
total_amount: 0.00B
pickup_ntacode: 0.00B
precipitation: 0.00B
pickup_ntaname: 0.00B
dropoff_datetime: 0.00B
cab_type: 0.00B
trip_id: 0.00B
dropoff_longitude: 0.00B
payment_type: 0.00B
dropoff_cdeligibil: 0.00B
dropoff_ntacode: 0.00B
dropoff_boroct2010: 0.00B
locustdb> SELECT passenger_count, count(0), sum(total_amount) FROM trips;
Scanned 1.46 billion rows in 26.5s (55 million rows/s)!
passenger_count | count_0 | sum_1
----------------+------------+---------------
0 | 4107443 | 4509842206
1 | 1028187597 | 1425303818216
2 | 210577025 | 304718353489
3 | 62345389 | 87788492212
4 | 29994987 | 42514355874
5 | 95184622 | 130181002616
6 | 34380600 | 51063741303
7 | 2595 | 8693795
8 | 2525 | 8272055
9 | 1367 | 7196690
10 | 17 | 75470
13 | 1 | 3150
15 | 2 | 2410
17 | 1 | 3995
19 | 1 | 500
25 | 1 | 760
33 | 2 | 1719
34 | 1 | 1680
36 | 1 | 6155
37 | 1 | 1420
38 | 1 | 730
47 | 1 | 900
49 | 26 | 8440
53 | 1 | 730
58 | 2 | 5188
61 | 1 | 3134
65 | 3 | 7010
66 | 1 | 1930
69 | 1 | 580
70 | 1 | 1070
84 | 1 | 4384
97 | 1 | 990
113 | 1 | 1330
125 | 1 | 1660
129 | 7 | 6150
133 | 1 | 1030
134 | 1 | 5514
137 | 1 | 5964
141 | 1 | 1894
155 | 1 | 9024
158 | 1 | 1444
160 | 1 | 1534
163 | 1 | 1554
164 | 1 | 6214
165 | 1 | 1214
177 | 1 | 1700
192 | 1 | 953
193 | 1 | 750
208 | 1508 | 1062534
211 | 1 | 700
213 | 4 | 1000
223 | 1 | 950
225 | 1 | 1600
247 | 1 | 1944
249 | 1 | 950
250 | 3 | 3770
254 | 1 | 650
255 | 10 | 17990
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.37s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
max_temperature: 0.00B
dropoff_ctlabel: 0.00B
dropoff_longitude: 0.00B
dropoff_ntaname: 0.00B
ehail_fee: 0.00B
dropoff_borocode: 0.00B
trip_id: 0.00B
pickup_ntaname: 0.00B
dropoff_puma: 0.00B
pickup_cdeligibil: 0.00B
dropoff_ntacode: 0.00B
vendor_id: 0.00B
dropoff_boroct2010: 0.00B
snow_depth: 0.00B
dropoff_cdeligibil: 0.00B
min_temperature: 0.00B
pickup_boroct2010: 0.00B
mta_tax: 0.00B
cab_type: 0.00B
dropoff_datetime: 0.00B
tolls_amount: 0.00B
trip_distance: 0.00B
pickup_borocode: 0.00B
pickup_latitude: 0.00B
pickup_ct2010: 0.00B
pickup_datetime: 0.00B
rate_code_id: 0.00B
passenger_count: 0.00B
fare_amount: 0.00B
pickup_ntacode: 0.00B
pickup_nyct2010_gid: 0.00B
snowfall: 0.00B
trip_type: 0.00B
pickup: 0.00B
precipitation: 0.00B
pickup_ctlabel: 0.00B
pickup_longitude: 0.00B
dropoff_latitude: 0.00B
payment_type: 0.00B
pickup_puma: 0.00B
dropoff_boroname: 0.00B
dropoff_nyct2010_gid: 0.00B
dropoff: 0.00B
tip_amount: 0.00B
average_wind_speed: 0.00B
improvement_surcharge: 0.00B
total_amount: 0.00B
store_and_fwd_flag: 0.00B
dropoff_ct2010: 0.00B
extra: 0.00B
pickup_boroname: 0.00B
locustdb> SELECT passenger_count, count(0), sum(total_amount) FROM trips;
Scanned 1.46 billion rows in 26.7s (55 million rows/s)!
passenger_count | count_0 | sum_1
----------------+------------+---------------
0 | 4107443 | 4509842206
1 | 1028187597 | 1425303818216
2 | 210577025 | 304718353489
3 | 62345389 | 87788492212
4 | 29994987 | 42514355874
5 | 95184622 | 130181002616
6 | 34380600 | 51063741303
7 | 2595 | 8693795
8 | 2525 | 8272055
9 | 1367 | 7196690
10 | 17 | 75470
13 | 1 | 3150
15 | 2 | 2410
17 | 1 | 3995
19 | 1 | 500
25 | 1 | 760
33 | 2 | 1719
34 | 1 | 1680
36 | 1 | 6155
37 | 1 | 1420
38 | 1 | 730
47 | 1 | 900
49 | 26 | 8440
53 | 1 | 730
58 | 2 | 5188
61 | 1 | 3134
65 | 3 | 7010
66 | 1 | 1930
69 | 1 | 580
70 | 1 | 1070
84 | 1 | 4384
97 | 1 | 990
113 | 1 | 1330
125 | 1 | 1660
129 | 7 | 6150
133 | 1 | 1030
134 | 1 | 5514
137 | 1 | 5964
141 | 1 | 1894
155 | 1 | 9024
158 | 1 | 1444
160 | 1 | 1534
163 | 1 | 1554
164 | 1 | 6214
165 | 1 | 1214
177 | 1 | 1700
192 | 1 | 953
193 | 1 | 750
208 | 1508 | 1062534
211 | 1 | 700
213 | 4 | 1000
223 | 1 | 950
225 | 1 | 1600
247 | 1 | 1944
249 | 1 | 950
250 | 3 | 3770
254 | 1 | 650
255 | 10 | 17990
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
trip_type: 0.00B
pickup_boroct2010: 0.00B
pickup_ntacode: 0.00B
dropoff_ctlabel: 0.00B
min_temperature: 0.00B
dropoff_borocode: 0.00B
dropoff: 0.00B
pickup_ntaname: 0.00B
precipitation: 0.00B
dropoff_boroname: 0.00B
pickup_datetime: 0.00B
dropoff_cdeligibil: 0.00B
extra: 0.00B
pickup_cdeligibil: 0.00B
improvement_surcharge: 0.00B
pickup_longitude: 0.00B
pickup_puma: 0.00B
vendor_id: 0.00B
rate_code_id: 0.00B
dropoff_longitude: 0.00B
cab_type: 0.00B
ehail_fee: 0.00B
snow_depth: 0.00B
max_temperature: 0.00B
pickup_nyct2010_gid: 0.00B
passenger_count: 0.00B
dropoff_nyct2010_gid: 0.00B
tolls_amount: 0.00B
dropoff_datetime: 0.00B
tip_amount: 0.00B
dropoff_puma: 0.00B
pickup_ct2010: 0.00B
store_and_fwd_flag: 0.00B
average_wind_speed: 0.00B
pickup_borocode: 0.00B
pickup_latitude: 0.00B
pickup_ctlabel: 0.00B
dropoff_boroct2010: 0.00B
payment_type: 0.00B
trip_distance: 0.00B
trip_id: 0.00B
fare_amount: 0.00B
dropoff_ntacode: 0.00B
total_amount: 0.00B
mta_tax: 0.00B
pickup: 0.00B
snowfall: 0.00B
dropoff_latitude: 0.00B
pickup_boroname: 0.00B
dropoff_ct2010: 0.00B
dropoff_ntaname: 0.00B
locustdb> SELECT passenger_count, to_year(pickup_datetime), count(0) FROM trips;
Scanned 1.46 billion rows in 44.3s (33 million rows/s)!
passenger_count | col_0 | count_0
----------------+-------+-----------
0 | 2009 | 1126
0 | 2010 | 35487
0 | 2011 | 2547937
0 | 2012 | 1274858
0 | 2013 | 5476
0 | 2014 | 9848
0 | 2015 | 46953
0 | 2016 | 17829
0 | 2017 | 167929
1 | 2001 | 3
1 | 2002 | 1
1 | 2003 | 4
1 | 2008 | 33
1 | 2009 | 116050070
1 | 2010 | 98991938
1 | 2011 | 119969450
1 | 2012 | 123955717
1 | 2013 | 122918578
1 | 2014 | 129404151
1 | 2015 | 119150088
1 | 2016 | 106826192
1 | 2017 | 90921221
1 | 2018 | 150
1 | 2041 | 1
2 | 2009 | 27534617
2 | 2010 | 23053740
2 | 2011 | 27132349
2 | 2012 | 24900623
2 | 2013 | 23610698
2 | 2014 | 24221283
2 | 2015 | 22349143
2 | 2016 | 20290051
2 | 2017 | 17484514
2 | 2018 | 7
3 | 2008 | 1
3 | 2009 | 7860551
3 | 2010 | 6700468
3 | 2011 | 8015005
3 | 2012 | 7716364
3 | 2013 | 7343603
3 | 2014 | 7348686
3 | 2015 | 6580430
3 | 2016 | 5810400
3 | 2017 | 4969879
3 | 2018 | 1
3 | 2053 | 1
4 | 2009 | 3829136
4 | 2010 | 3282494
4 | 2011 | 3819715
4 | 2012 | 3752549
4 | 2013 | 3593459
4 | 2014 | 3512831
4 | 2015 | 3105718
4 | 2016 | 2770237
4 | 2017 | 2328848
5 | 2008 | 7
5 | 2009 | 14885624
5 | 2010 | 12311339
5 | 2011 | 13876061
5 | 2012 | 12063339
5 | 2013 | 10139454
5 | 2014 | 10056870
5 | 2015 | 8691048
5 | 2016 | 7337348
5 | 2017 | 5823513
5 | 2018 | 19
6 | 2003 | 1
6 | 2008 | 1
6 | 2009 | 734341
6 | 2010 | 595332
6 | 2011 | 1536489
6 | 2012 | 4880708
6 | 2013 | 6779135
6 | 2014 | 6396494
6 | 2015 | 5421796
6 | 2016 | 4496670
6 | 2017 | 3539630
6 | 2018 | 3
7 | 2009 | 7
7 | 2010 | 4
7 | 2011 | 8
7 | 2012 | 17
7 | 2013 | 65
7 | 2014 | 504
7 | 2015 | 599
7 | 2016 | 765
7 | 2017 | 626
8 | 2009 | 2
8 | 2010 | 1
8 | 2011 | 2
8 | 2012 | 13
8 | 2013 | 43
8 | 2014 | 464
8 | 2015 | 668
8 | 2016 | 707
8 | 2017 | 625
9 | 2009 | 1
9 | 2010 | 3
9 | 2011 | 8
9 | 2012 | 25
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.37s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
snowfall: 0.00B
trip_type: 0.00B
rate_code_id: 0.00B
dropoff_ctlabel: 0.00B
precipitation: 0.00B
dropoff_nyct2010_gid: 0.00B
total_amount: 0.00B
payment_type: 0.00B
pickup_nyct2010_gid: 0.00B
pickup_ctlabel: 0.00B
pickup_ntaname: 0.00B
tip_amount: 0.00B
pickup: 0.00B
mta_tax: 0.00B
pickup_borocode: 0.00B
pickup_ct2010: 0.00B
pickup_cdeligibil: 0.00B
pickup_ntacode: 0.00B
dropoff_datetime: 0.00B
vendor_id: 0.00B
pickup_datetime: 0.00B
dropoff_ntacode: 0.00B
snow_depth: 0.00B
tolls_amount: 0.00B
dropoff_puma: 0.00B
ehail_fee: 0.00B
dropoff_borocode: 0.00B
passenger_count: 0.00B
extra: 0.00B
pickup_latitude: 0.00B
average_wind_speed: 0.00B
pickup_boroct2010: 0.00B
dropoff_longitude: 0.00B
dropoff_boroct2010: 0.00B
trip_distance: 0.00B
trip_id: 0.00B
cab_type: 0.00B
min_temperature: 0.00B
dropoff_cdeligibil: 0.00B
max_temperature: 0.00B
improvement_surcharge: 0.00B
dropoff_latitude: 0.00B
fare_amount: 0.00B
dropoff_boroname: 0.00B
pickup_puma: 0.00B
store_and_fwd_flag: 0.00B
dropoff_ntaname: 0.00B
dropoff: 0.00B
pickup_boroname: 0.00B
dropoff_ct2010: 0.00B
pickup_longitude: 0.00B
locustdb> SELECT passenger_count, to_year(pickup_datetime), count(0) FROM trips;
Scanned 1.46 billion rows in 44.2s (33 million rows/s)!
passenger_count | col_0 | count_0
----------------+-------+-----------
0 | 2009 | 1126
0 | 2010 | 35487
0 | 2011 | 2547937
0 | 2012 | 1274858
0 | 2013 | 5476
0 | 2014 | 9848
0 | 2015 | 46953
0 | 2016 | 17829
0 | 2017 | 167929
1 | 2001 | 3
1 | 2002 | 1
1 | 2003 | 4
1 | 2008 | 33
1 | 2009 | 116050070
1 | 2010 | 98991938
1 | 2011 | 119969450
1 | 2012 | 123955717
1 | 2013 | 122918578
1 | 2014 | 129404151
1 | 2015 | 119150088
1 | 2016 | 106826192
1 | 2017 | 90921221
1 | 2018 | 150
1 | 2041 | 1
2 | 2009 | 27534617
2 | 2010 | 23053740
2 | 2011 | 27132349
2 | 2012 | 24900623
2 | 2013 | 23610698
2 | 2014 | 24221283
2 | 2015 | 22349143
2 | 2016 | 20290051
2 | 2017 | 17484514
2 | 2018 | 7
3 | 2008 | 1
3 | 2009 | 7860551
3 | 2010 | 6700468
3 | 2011 | 8015005
3 | 2012 | 7716364
3 | 2013 | 7343603
3 | 2014 | 7348686
3 | 2015 | 6580430
3 | 2016 | 5810400
3 | 2017 | 4969879
3 | 2018 | 1
3 | 2053 | 1
4 | 2009 | 3829136
4 | 2010 | 3282494
4 | 2011 | 3819715
4 | 2012 | 3752549
4 | 2013 | 3593459
4 | 2014 | 3512831
4 | 2015 | 3105718
4 | 2016 | 2770237
4 | 2017 | 2328848
5 | 2008 | 7
5 | 2009 | 14885624
5 | 2010 | 12311339
5 | 2011 | 13876061
5 | 2012 | 12063339
5 | 2013 | 10139454
5 | 2014 | 10056870
5 | 2015 | 8691048
5 | 2016 | 7337348
5 | 2017 | 5823513
5 | 2018 | 19
6 | 2003 | 1
6 | 2008 | 1
6 | 2009 | 734341
6 | 2010 | 595332
6 | 2011 | 1536489
6 | 2012 | 4880708
6 | 2013 | 6779135
6 | 2014 | 6396494
6 | 2015 | 5421796
6 | 2016 | 4496670
6 | 2017 | 3539630
6 | 2018 | 3
7 | 2009 | 7
7 | 2010 | 4
7 | 2011 | 8
7 | 2012 | 17
7 | 2013 | 65
7 | 2014 | 504
7 | 2015 | 599
7 | 2016 | 765
7 | 2017 | 626
8 | 2009 | 2
8 | 2010 | 1
8 | 2011 | 2
8 | 2012 | 13
8 | 2013 | 43
8 | 2014 | 464
8 | 2015 | 668
8 | 2016 | 707
8 | 2017 | 625
9 | 2009 | 1
9 | 2010 | 3
9 | 2011 | 8
9 | 2012 | 25
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
pickup_datetime: 0.00B
mta_tax: 0.00B
tip_amount: 0.00B
dropoff_latitude: 0.00B
pickup_cdeligibil: 0.00B
dropoff_longitude: 0.00B
pickup: 0.00B
pickup_boroct2010: 0.00B
dropoff_puma: 0.00B
trip_type: 0.00B
dropoff_borocode: 0.00B
pickup_ntaname: 0.00B
trip_id: 0.00B
trip_distance: 0.00B
dropoff: 0.00B
pickup_latitude: 0.00B
pickup_boroname: 0.00B
min_temperature: 0.00B
payment_type: 0.00B
cab_type: 0.00B
snow_depth: 0.00B
ehail_fee: 0.00B
pickup_puma: 0.00B
dropoff_ct2010: 0.00B
tolls_amount: 0.00B
average_wind_speed: 0.00B
improvement_surcharge: 0.00B
precipitation: 0.00B
pickup_ntacode: 0.00B
dropoff_datetime: 0.00B
pickup_nyct2010_gid: 0.00B
dropoff_cdeligibil: 0.00B
pickup_ct2010: 0.00B
dropoff_boroct2010: 0.00B
snowfall: 0.00B
dropoff_ntacode: 0.00B
vendor_id: 0.00B
dropoff_ctlabel: 0.00B
pickup_ctlabel: 0.00B
dropoff_boroname: 0.00B
rate_code_id: 0.00B
extra: 0.00B
pickup_borocode: 0.00B
total_amount: 0.00B
max_temperature: 0.00B
dropoff_nyct2010_gid: 0.00B
store_and_fwd_flag: 0.00B
fare_amount: 0.00B
passenger_count: 0.00B
pickup_longitude: 0.00B
dropoff_ntaname: 0.00B
locustdb> SELECT passenger_count, to_year(pickup_datetime), count(0) FROM trips;
Scanned 1.46 billion rows in 44.1s (33 million rows/s)!
passenger_count | col_0 | count_0
----------------+-------+-----------
0 | 2009 | 1126
0 | 2010 | 35487
0 | 2011 | 2547937
0 | 2012 | 1274858
0 | 2013 | 5476
0 | 2014 | 9848
0 | 2015 | 46953
0 | 2016 | 17829
0 | 2017 | 167929
1 | 2001 | 3
1 | 2002 | 1
1 | 2003 | 4
1 | 2008 | 33
1 | 2009 | 116050070
1 | 2010 | 98991938
1 | 2011 | 119969450
1 | 2012 | 123955717
1 | 2013 | 122918578
1 | 2014 | 129404151
1 | 2015 | 119150088
1 | 2016 | 106826192
1 | 2017 | 90921221
1 | 2018 | 150
1 | 2041 | 1
2 | 2009 | 27534617
2 | 2010 | 23053740
2 | 2011 | 27132349
2 | 2012 | 24900623
2 | 2013 | 23610698
2 | 2014 | 24221283
2 | 2015 | 22349143
2 | 2016 | 20290051
2 | 2017 | 17484514
2 | 2018 | 7
3 | 2008 | 1
3 | 2009 | 7860551
3 | 2010 | 6700468
3 | 2011 | 8015005
3 | 2012 | 7716364
3 | 2013 | 7343603
3 | 2014 | 7348686
3 | 2015 | 6580430
3 | 2016 | 5810400
3 | 2017 | 4969879
3 | 2018 | 1
3 | 2053 | 1
4 | 2009 | 3829136
4 | 2010 | 3282494
4 | 2011 | 3819715
4 | 2012 | 3752549
4 | 2013 | 3593459
4 | 2014 | 3512831
4 | 2015 | 3105718
4 | 2016 | 2770237
4 | 2017 | 2328848
5 | 2008 | 7
5 | 2009 | 14885624
5 | 2010 | 12311339
5 | 2011 | 13876061
5 | 2012 | 12063339
5 | 2013 | 10139454
5 | 2014 | 10056870
5 | 2015 | 8691048
5 | 2016 | 7337348
5 | 2017 | 5823513
5 | 2018 | 19
6 | 2003 | 1
6 | 2008 | 1
6 | 2009 | 734341
6 | 2010 | 595332
6 | 2011 | 1536489
6 | 2012 | 4880708
6 | 2013 | 6779135
6 | 2014 | 6396494
6 | 2015 | 5421796
6 | 2016 | 4496670
6 | 2017 | 3539630
6 | 2018 | 3
7 | 2009 | 7
7 | 2010 | 4
7 | 2011 | 8
7 | 2012 | 17
7 | 2013 | 65
7 | 2014 | 504
7 | 2015 | 599
7 | 2016 | 765
7 | 2017 | 626
8 | 2009 | 2
8 | 2010 | 1
8 | 2011 | 2
8 | 2012 | 13
8 | 2013 | 43
8 | 2014 | 464
8 | 2015 | 668
8 | 2016 | 707
8 | 2017 | 625
9 | 2009 | 1
9 | 2010 | 3
9 | 2011 | 8
9 | 2012 | 25
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
pickup_ctlabel: 0.00B
dropoff_boroname: 0.00B
dropoff: 0.00B
improvement_surcharge: 0.00B
trip_id: 0.00B
dropoff_boroct2010: 0.00B
extra: 0.00B
mta_tax: 0.00B
dropoff_puma: 0.00B
trip_type: 0.00B
pickup_boroname: 0.00B
pickup_cdeligibil: 0.00B
snowfall: 0.00B
rate_code_id: 0.00B
dropoff_latitude: 0.00B
pickup_borocode: 0.00B
cab_type: 0.00B
max_temperature: 0.00B
tip_amount: 0.00B
store_and_fwd_flag: 0.00B
min_temperature: 0.00B
pickup_nyct2010_gid: 0.00B
passenger_count: 0.00B
dropoff_cdeligibil: 0.00B
pickup_ct2010: 0.00B
tolls_amount: 0.00B
dropoff_datetime: 0.00B
dropoff_ntacode: 0.00B
vendor_id: 0.00B
pickup_ntaname: 0.00B
dropoff_ctlabel: 0.00B
fare_amount: 0.00B
pickup_longitude: 0.00B
dropoff_borocode: 0.00B
ehail_fee: 0.00B
snow_depth: 0.00B
dropoff_ntaname: 0.00B
pickup_ntacode: 0.00B
average_wind_speed: 0.00B
payment_type: 0.00B
precipitation: 0.00B
pickup: 0.00B
total_amount: 0.00B
dropoff_ct2010: 0.00B
pickup_datetime: 0.00B
pickup_puma: 0.00B
dropoff_nyct2010_gid: 0.00B
pickup_boroct2010: 0.00B
pickup_latitude: 0.00B
dropoff_longitude: 0.00B
trip_distance: 0.00B
locustdb> SELECT passenger_count, to_year(pickup_datetime), trip_distance / 1000, count(0) FROM trips;
Scanned 1.46 billion rows in 68.1s (22 million rows/s)!
passenger_count | col_0 | col_1 | count_0
----------------+-------+-------+---------
0 | 2009 | 0 | 570
0 | 2009 | 1 | 267
0 | 2009 | 2 | 115
0 | 2009 | 3 | 52
0 | 2009 | 4 | 28
0 | 2009 | 5 | 28
0 | 2009 | 6 | 13
0 | 2009 | 7 | 11
0 | 2009 | 8 | 8
0 | 2009 | 9 | 5
0 | 2009 | 10 | 4
0 | 2009 | 11 | 6
0 | 2009 | 12 | 2
0 | 2009 | 13 | 2
0 | 2009 | 14 | 2
0 | 2009 | 17 | 3
0 | 2009 | 18 | 2
0 | 2009 | 19 | 2
0 | 2009 | 20 | 1
0 | 2009 | 21 | 2
0 | 2009 | 22 | 2
0 | 2009 | 31 | 1
0 | 2010 | 0 | 16834
0 | 2010 | 1 | 10264
0 | 2010 | 2 | 3858
0 | 2010 | 3 | 1797
0 | 2010 | 4 | 831
0 | 2010 | 5 | 643
0 | 2010 | 6 | 418
0 | 2010 | 7 | 207
0 | 2010 | 8 | 108
0 | 2010 | 9 | 100
0 | 2010 | 10 | 109
0 | 2010 | 11 | 118
0 | 2010 | 12 | 58
0 | 2010 | 13 | 30
0 | 2010 | 14 | 20
0 | 2010 | 15 | 17
0 | 2010 | 16 | 13
0 | 2010 | 17 | 10
0 | 2010 | 18 | 3
0 | 2010 | 19 | 8
0 | 2010 | 20 | 8
0 | 2010 | 21 | 5
0 | 2010 | 22 | 2
0 | 2010 | 23 | 3
0 | 2010 | 24 | 2
0 | 2010 | 26 | 3
0 | 2010 | 27 | 1
0 | 2010 | 28 | 1
0 | 2010 | 29 | 1
0 | 2010 | 32 | 1
0 | 2010 | 33 | 1
0 | 2010 | 36 | 2
0 | 2010 | 37 | 2
0 | 2010 | 38 | 1
0 | 2010 | 43 | 1
0 | 2010 | 47 | 1
0 | 2010 | 53 | 1
0 | 2010 | 54 | 1
0 | 2010 | 70 | 1
0 | 2010 | 76 | 1
0 | 2010 | 77 | 1
0 | 2010 | 86 | 1
0 | 2011 | 0 | 679047
0 | 2011 | 1 | 884997
0 | 2011 | 2 | 430866
0 | 2011 | 3 | 211773
0 | 2011 | 4 | 112552
0 | 2011 | 5 | 69700
0 | 2011 | 6 | 43498
0 | 2011 | 7 | 26703
0 | 2011 | 8 | 16962
0 | 2011 | 9 | 13569
0 | 2011 | 10 | 10974
0 | 2011 | 11 | 8656
0 | 2011 | 12 | 6488
0 | 2011 | 13 | 4422
0 | 2011 | 14 | 3534
0 | 2011 | 15 | 3470
0 | 2011 | 16 | 4541
0 | 2011 | 17 | 4617
0 | 2011 | 18 | 3805
0 | 2011 | 19 | 2574
0 | 2011 | 20 | 1710
0 | 2011 | 21 | 965
0 | 2011 | 22 | 581
0 | 2011 | 23 | 340
0 | 2011 | 24 | 214
0 | 2011 | 25 | 189
0 | 2011 | 26 | 153
0 | 2011 | 27 | 138
0 | 2011 | 28 | 114
0 | 2011 | 29 | 63
0 | 2011 | 30 | 73
0 | 2011 | 31 | 58
0 | 2011 | 32 | 72
0 | 2011 | 33 | 39
0 | 2011 | 34 | 46
0 | 2011 | 35 | 39
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
pickup_boroct2010: 0.00B
pickup_ctlabel: 0.00B
dropoff: 0.00B
trip_id: 0.00B
dropoff_borocode: 0.00B
pickup_longitude: 0.00B
min_temperature: 0.00B
pickup_datetime: 0.00B
trip_distance: 0.00B
dropoff_boroct2010: 0.00B
pickup_ntaname: 0.00B
extra: 0.00B
pickup_cdeligibil: 0.00B
dropoff_ntacode: 0.00B
trip_type: 0.00B
dropoff_puma: 0.00B
pickup_latitude: 0.00B
fare_amount: 0.00B
average_wind_speed: 0.00B
cab_type: 0.00B
dropoff_ctlabel: 0.00B
improvement_surcharge: 0.00B
dropoff_boroname: 0.00B
payment_type: 0.00B
precipitation: 0.00B
dropoff_ntaname: 0.00B
dropoff_ct2010: 0.00B
mta_tax: 0.00B
pickup_nyct2010_gid: 0.00B
store_and_fwd_flag: 0.00B
dropoff_cdeligibil: 0.00B
pickup_puma: 0.00B
snowfall: 0.00B
tolls_amount: 0.00B
pickup_borocode: 0.00B
pickup_ntacode: 0.00B
rate_code_id: 0.00B
passenger_count: 0.00B
pickup_boroname: 0.00B
snow_depth: 0.00B
tip_amount: 0.00B
ehail_fee: 0.00B
vendor_id: 0.00B
pickup_ct2010: 0.00B
dropoff_longitude: 0.00B
pickup: 0.00B
dropoff_nyct2010_gid: 0.00B
total_amount: 0.00B
dropoff_datetime: 0.00B
dropoff_latitude: 0.00B
max_temperature: 0.00B
locustdb> SELECT passenger_count, to_year(pickup_datetime), trip_distance / 1000, count(0) FROM trips;
Scanned 1.46 billion rows in 68.2s (21 million rows/s)!
passenger_count | col_0 | col_1 | count_0
----------------+-------+-------+---------
0 | 2009 | 0 | 570
0 | 2009 | 1 | 267
0 | 2009 | 2 | 115
0 | 2009 | 3 | 52
0 | 2009 | 4 | 28
0 | 2009 | 5 | 28
0 | 2009 | 6 | 13
0 | 2009 | 7 | 11
0 | 2009 | 8 | 8
0 | 2009 | 9 | 5
0 | 2009 | 10 | 4
0 | 2009 | 11 | 6
0 | 2009 | 12 | 2
0 | 2009 | 13 | 2
0 | 2009 | 14 | 2
0 | 2009 | 17 | 3
0 | 2009 | 18 | 2
0 | 2009 | 19 | 2
0 | 2009 | 20 | 1
0 | 2009 | 21 | 2
0 | 2009 | 22 | 2
0 | 2009 | 31 | 1
0 | 2010 | 0 | 16834
0 | 2010 | 1 | 10264
0 | 2010 | 2 | 3858
0 | 2010 | 3 | 1797
0 | 2010 | 4 | 831
0 | 2010 | 5 | 643
0 | 2010 | 6 | 418
0 | 2010 | 7 | 207
0 | 2010 | 8 | 108
0 | 2010 | 9 | 100
0 | 2010 | 10 | 109
0 | 2010 | 11 | 118
0 | 2010 | 12 | 58
0 | 2010 | 13 | 30
0 | 2010 | 14 | 20
0 | 2010 | 15 | 17
0 | 2010 | 16 | 13
0 | 2010 | 17 | 10
0 | 2010 | 18 | 3
0 | 2010 | 19 | 8
0 | 2010 | 20 | 8
0 | 2010 | 21 | 5
0 | 2010 | 22 | 2
0 | 2010 | 23 | 3
0 | 2010 | 24 | 2
0 | 2010 | 26 | 3
0 | 2010 | 27 | 1
0 | 2010 | 28 | 1
0 | 2010 | 29 | 1
0 | 2010 | 32 | 1
0 | 2010 | 33 | 1
0 | 2010 | 36 | 2
0 | 2010 | 37 | 2
0 | 2010 | 38 | 1
0 | 2010 | 43 | 1
0 | 2010 | 47 | 1
0 | 2010 | 53 | 1
0 | 2010 | 54 | 1
0 | 2010 | 70 | 1
0 | 2010 | 76 | 1
0 | 2010 | 77 | 1
0 | 2010 | 86 | 1
0 | 2011 | 0 | 679047
0 | 2011 | 1 | 884997
0 | 2011 | 2 | 430866
0 | 2011 | 3 | 211773
0 | 2011 | 4 | 112552
0 | 2011 | 5 | 69700
0 | 2011 | 6 | 43498
0 | 2011 | 7 | 26703
0 | 2011 | 8 | 16962
0 | 2011 | 9 | 13569
0 | 2011 | 10 | 10974
0 | 2011 | 11 | 8656
0 | 2011 | 12 | 6488
0 | 2011 | 13 | 4422
0 | 2011 | 14 | 3534
0 | 2011 | 15 | 3470
0 | 2011 | 16 | 4541
0 | 2011 | 17 | 4617
0 | 2011 | 18 | 3805
0 | 2011 | 19 | 2574
0 | 2011 | 20 | 1710
0 | 2011 | 21 | 965
0 | 2011 | 22 | 581
0 | 2011 | 23 | 340
0 | 2011 | 24 | 214
0 | 2011 | 25 | 189
0 | 2011 | 26 | 153
0 | 2011 | 27 | 138
0 | 2011 | 28 | 114
0 | 2011 | 29 | 63
0 | 2011 | 30 | 73
0 | 2011 | 31 | 58
0 | 2011 | 32 | 72
0 | 2011 | 33 | 39
0 | 2011 | 34 | 46
0 | 2011 | 35 | 39
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
pickup_cdeligibil: 0.00B
pickup: 0.00B
max_temperature: 0.00B
dropoff_datetime: 0.00B
dropoff_boroname: 0.00B
trip_distance: 0.00B
ehail_fee: 0.00B
dropoff_boroct2010: 0.00B
dropoff_latitude: 0.00B
tip_amount: 0.00B
trip_id: 0.00B
fare_amount: 0.00B
tolls_amount: 0.00B
pickup_borocode: 0.00B
dropoff_nyct2010_gid: 0.00B
snowfall: 0.00B
pickup_datetime: 0.00B
pickup_boroct2010: 0.00B
pickup_ctlabel: 0.00B
pickup_latitude: 0.00B
average_wind_speed: 0.00B
pickup_ntacode: 0.00B
pickup_nyct2010_gid: 0.00B
total_amount: 0.00B
precipitation: 0.00B
pickup_boroname: 0.00B
dropoff_borocode: 0.00B
cab_type: 0.00B
dropoff_cdeligibil: 0.00B
extra: 0.00B
rate_code_id: 0.00B
pickup_ntaname: 0.00B
pickup_ct2010: 0.00B
dropoff_longitude: 0.00B
trip_type: 0.00B
dropoff_ntaname: 0.00B
store_and_fwd_flag: 0.00B
vendor_id: 0.00B
passenger_count: 0.00B
min_temperature: 0.00B
dropoff_ctlabel: 0.00B
dropoff_puma: 0.00B
payment_type: 0.00B
dropoff_ct2010: 0.00B
dropoff: 0.00B
mta_tax: 0.00B
pickup_longitude: 0.00B
dropoff_ntacode: 0.00B
pickup_puma: 0.00B
snow_depth: 0.00B
improvement_surcharge: 0.00B
locustdb> SELECT passenger_count, to_year(pickup_datetime), trip_distance / 1000, count(0) FROM trips;
Scanned 1.46 billion rows in 68.1s (22 million rows/s)!
passenger_count | col_0 | col_1 | count_0
----------------+-------+-------+---------
0 | 2009 | 0 | 570
0 | 2009 | 1 | 267
0 | 2009 | 2 | 115
0 | 2009 | 3 | 52
0 | 2009 | 4 | 28
0 | 2009 | 5 | 28
0 | 2009 | 6 | 13
0 | 2009 | 7 | 11
0 | 2009 | 8 | 8
0 | 2009 | 9 | 5
0 | 2009 | 10 | 4
0 | 2009 | 11 | 6
0 | 2009 | 12 | 2
0 | 2009 | 13 | 2
0 | 2009 | 14 | 2
0 | 2009 | 17 | 3
0 | 2009 | 18 | 2
0 | 2009 | 19 | 2
0 | 2009 | 20 | 1
0 | 2009 | 21 | 2
0 | 2009 | 22 | 2
0 | 2009 | 31 | 1
0 | 2010 | 0 | 16834
0 | 2010 | 1 | 10264
0 | 2010 | 2 | 3858
0 | 2010 | 3 | 1797
0 | 2010 | 4 | 831
0 | 2010 | 5 | 643
0 | 2010 | 6 | 418
0 | 2010 | 7 | 207
0 | 2010 | 8 | 108
0 | 2010 | 9 | 100
0 | 2010 | 10 | 109
0 | 2010 | 11 | 118
0 | 2010 | 12 | 58
0 | 2010 | 13 | 30
0 | 2010 | 14 | 20
0 | 2010 | 15 | 17
0 | 2010 | 16 | 13
0 | 2010 | 17 | 10
0 | 2010 | 18 | 3
0 | 2010 | 19 | 8
0 | 2010 | 20 | 8
0 | 2010 | 21 | 5
0 | 2010 | 22 | 2
0 | 2010 | 23 | 3
0 | 2010 | 24 | 2
0 | 2010 | 26 | 3
0 | 2010 | 27 | 1
0 | 2010 | 28 | 1
0 | 2010 | 29 | 1
0 | 2010 | 32 | 1
0 | 2010 | 33 | 1
0 | 2010 | 36 | 2
0 | 2010 | 37 | 2
0 | 2010 | 38 | 1
0 | 2010 | 43 | 1
0 | 2010 | 47 | 1
0 | 2010 | 53 | 1
0 | 2010 | 54 | 1
0 | 2010 | 70 | 1
0 | 2010 | 76 | 1
0 | 2010 | 77 | 1
0 | 2010 | 86 | 1
0 | 2011 | 0 | 679047
0 | 2011 | 1 | 884997
0 | 2011 | 2 | 430866
0 | 2011 | 3 | 211773
0 | 2011 | 4 | 112552
0 | 2011 | 5 | 69700
0 | 2011 | 6 | 43498
0 | 2011 | 7 | 26703
0 | 2011 | 8 | 16962
0 | 2011 | 9 | 13569
0 | 2011 | 10 | 10974
0 | 2011 | 11 | 8656
0 | 2011 | 12 | 6488
0 | 2011 | 13 | 4422
0 | 2011 | 14 | 3534
0 | 2011 | 15 | 3470
0 | 2011 | 16 | 4541
0 | 2011 | 17 | 4617
0 | 2011 | 18 | 3805
0 | 2011 | 19 | 2574
0 | 2011 | 20 | 1710
0 | 2011 | 21 | 965
0 | 2011 | 22 | 581
0 | 2011 | 23 | 340
0 | 2011 | 24 | 214
0 | 2011 | 25 | 189
0 | 2011 | 26 | 153
0 | 2011 | 27 | 138
0 | 2011 | 28 | 114
0 | 2011 | 29 | 63
0 | 2011 | 30 | 73
0 | 2011 | 31 | 58
0 | 2011 | 32 | 72
0 | 2011 | 33 | 39
0 | 2011 | 34 | 46
0 | 2011 | 35 | 39
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.37s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
pickup_longitude: 0.00B
dropoff_cdeligibil: 0.00B
dropoff_ctlabel: 0.00B
pickup_datetime: 0.00B
pickup_ntaname: 0.00B
dropoff_boroname: 0.00B
store_and_fwd_flag: 0.00B
trip_type: 0.00B
rate_code_id: 0.00B
dropoff_nyct2010_gid: 0.00B
dropoff_borocode: 0.00B
pickup_ctlabel: 0.00B
dropoff_puma: 0.00B
min_temperature: 0.00B
total_amount: 0.00B
cab_type: 0.00B
trip_distance: 0.00B
trip_id: 0.00B
max_temperature: 0.00B
dropoff_datetime: 0.00B
vendor_id: 0.00B
extra: 0.00B
pickup_ct2010: 0.00B
improvement_surcharge: 0.00B
dropoff_ntaname: 0.00B
pickup_cdeligibil: 0.00B
pickup_boroname: 0.00B
dropoff: 0.00B
precipitation: 0.00B
passenger_count: 0.00B
pickup: 0.00B
dropoff_boroct2010: 0.00B
dropoff_ntacode: 0.00B
pickup_ntacode: 0.00B
snow_depth: 0.00B
pickup_latitude: 0.00B
dropoff_longitude: 0.00B
fare_amount: 0.00B
pickup_puma: 0.00B
pickup_nyct2010_gid: 0.00B
ehail_fee: 0.00B
dropoff_latitude: 0.00B
payment_type: 0.00B
pickup_borocode: 0.00B
snowfall: 0.00B
tolls_amount: 0.00B
mta_tax: 0.00B
dropoff_ct2010: 0.00B
tip_amount: 0.00B
average_wind_speed: 0.00B
pickup_boroct2010: 0.00B
locustdb> SELECT trip_id FROM trips WHERE (passenger_count = 5) AND (vendor_id = "CMT") AND (total_amount < 500)
AND (store_and_fwd_flag = "1") LIMIT 100;
Scanned 1.46 billion rows in 37.5s (39 million rows/s)!
trip_id
----------
182124046
166444891
207059403
208926304
173799760
175280990
247696432
176394104
310082405
85701057
244996620
174687424
154865525
237523596
237547051
138080491
261961948
293214555
122800644
152363765
195778208
195827261
136472715
303171377
161072070
121789437
142031633
102569125
163106718
104895642
106309860
241317351
190937722
130612502
232337094
212714740
174182597
203089568
164837200
84895468
191366388
97545508
259106549
144660880
136644714
250557151
263677077
202610735
106048938
112520068
245281086
303600048
257680549
299069329
80845388
201367926
83385404
103186939
148254686
210730145
91804982
195132906
237368106
177367538
228237325
233908085
194235323
156105032
138089339
181868551
190953060
193071605
155088848
237961482
137556872
139768745
121970701
122012280
203844240
147380973
208072881
208684890
189096372
243127062
113478507
135130551
255415775
298515659
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
snow_depth: 0.00B
passenger_count: 0.00B
pickup_longitude: 0.00B
mta_tax: 0.00B
dropoff_ntacode: 0.00B
pickup_ct2010: 0.00B
snowfall: 0.00B
min_temperature: 0.00B
dropoff_ctlabel: 0.00B
vendor_id: 0.00B
pickup_datetime: 0.00B
rate_code_id: 0.00B
cab_type: 0.00B
dropoff_longitude: 0.00B
pickup_cdeligibil: 0.00B
dropoff_datetime: 0.00B
fare_amount: 0.00B
pickup_ntaname: 0.00B
trip_distance: 0.00B
pickup_ctlabel: 0.00B
dropoff_boroname: 0.00B
extra: 0.00B
precipitation: 0.00B
ehail_fee: 0.00B
payment_type: 0.00B
store_and_fwd_flag: 0.00B
pickup_ntacode: 0.00B
pickup_boroct2010: 0.00B
dropoff_boroct2010: 0.00B
pickup_latitude: 0.00B
average_wind_speed: 0.00B
pickup_nyct2010_gid: 0.00B
dropoff_borocode: 0.00B
dropoff_ntaname: 0.00B
pickup_puma: 0.00B
dropoff_puma: 0.00B
dropoff_cdeligibil: 0.00B
pickup: 0.00B
trip_type: 0.00B
tolls_amount: 0.00B
tip_amount: 0.00B
improvement_surcharge: 0.00B
pickup_borocode: 0.00B
pickup_boroname: 0.00B
dropoff_ct2010: 0.00B
dropoff: 0.00B
dropoff_nyct2010_gid: 0.00B
total_amount: 0.00B
trip_id: 0.00B
max_temperature: 0.00B
dropoff_latitude: 0.00B
locustdb> SELECT trip_id FROM trips WHERE (passenger_count = 5) AND (vendor_id = "CMT") AND (total_amount < 500) AND (store_and_fwd_flag = "1") LIMIT 100;
Scanned 1.46 billion rows in 36.9s (40 million rows/s)!
trip_id
----------
182124046
174182597
155088848
175280990
237368106
121970701
122012280
122800644
106048938
166444891
189096372
244996620
194235323
137556872
190953060
191366388
136472715
139768745
303600048
181868551
121789437
85701057
228237325
112520068
195132906
156105032
136644714
202610735
203089568
164837200
148254686
190937722
210730145
243127062
232337094
233908085
237523596
237547051
261961948
255415775
259106549
293214555
208926304
91804982
193071605
176394104
97545508
250557151
257680549
299069329
163106718
144660880
174687424
237961482
303171377
310082405
201367926
83385404
84895468
106309860
207059403
147380973
241317351
212714740
154865525
195778208
195827261
177367538
138089339
80845388
102569125
103186939
203844240
208072881
208684890
152363765
113478507
173799760
138080491
263677077
161072070
142031633
130612502
135130551
247696432
104895642
245281086
298515659
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
trip_id: 0.00B
dropoff_longitude: 0.00B
pickup_longitude: 0.00B
pickup_puma: 0.00B
dropoff_puma: 0.00B
passenger_count: 0.00B
vendor_id: 0.00B
pickup_cdeligibil: 0.00B
min_temperature: 0.00B
fare_amount: 0.00B
pickup_ntacode: 0.00B
extra: 0.00B
pickup_nyct2010_gid: 0.00B
pickup_datetime: 0.00B
dropoff_boroname: 0.00B
dropoff_ntaname: 0.00B
pickup_latitude: 0.00B
snowfall: 0.00B
dropoff_boroct2010: 0.00B
pickup_ntaname: 0.00B
store_and_fwd_flag: 0.00B
ehail_fee: 0.00B
pickup: 0.00B
trip_type: 0.00B
dropoff_nyct2010_gid: 0.00B
pickup_ct2010: 0.00B
dropoff_ct2010: 0.00B
dropoff_ctlabel: 0.00B
dropoff_borocode: 0.00B
pickup_ctlabel: 0.00B
total_amount: 0.00B
snow_depth: 0.00B
pickup_boroname: 0.00B
pickup_boroct2010: 0.00B
trip_distance: 0.00B
pickup_borocode: 0.00B
max_temperature: 0.00B
dropoff_latitude: 0.00B
dropoff: 0.00B
tolls_amount: 0.00B
dropoff_datetime: 0.00B
dropoff_ntacode: 0.00B
tip_amount: 0.00B
mta_tax: 0.00B
cab_type: 0.00B
precipitation: 0.00B
rate_code_id: 0.00B
average_wind_speed: 0.00B
payment_type: 0.00B
improvement_surcharge: 0.00B
dropoff_cdeligibil: 0.00B
locustdb> SELECT trip_id FROM trips WHERE (passenger_count = 5) AND (vendor_id = "CMT") AND (total_amount < 500) AND (store_and_fwd_flag = "1") LIMIT 100;
Scanned 1.46 billion rows in 37.6s (39 million rows/s)!
trip_id
----------
121789437
144660880
130612502
191366388
243127062
97545508
139768745
259106549
161072070
208926304
244996620
154865525
237523596
237547051
263677077
201367926
202610735
208684890
232337094
135130551
195778208
136472715
250557151
257680549
298515659
299069329
142031633
203844240
147380973
113478507
155088848
247696432
293214555
83385404
103186939
106048938
152363765
175280990
255415775
80845388
163106718
164837200
85701057
148254686
190937722
177367538
303600048
121970701
122012280
84895468
106309860
228237325
91804982
212714740
233908085
174182597
174687424
136644714
138080491
122800644
190953060
210730145
112520068
245281086
156105032
189096372
241317351
193071605
173799760
195827261
237368106
137556872
138089339
303171377
181868551
182124046
102569125
166444891
207059403
261961948
203089568
104895642
208072881
194235323
195132906
176394104
237961482
310082405
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
dropoff_boroct2010: 0.00B
snow_depth: 0.00B
dropoff_datetime: 0.00B
snowfall: 0.00B
pickup_nyct2010_gid: 0.00B
pickup_borocode: 0.00B
dropoff: 0.00B
trip_id: 0.00B
pickup_ctlabel: 0.00B
dropoff_ct2010: 0.00B
dropoff_cdeligibil: 0.00B
pickup_ct2010: 0.00B
pickup_latitude: 0.00B
cab_type: 0.00B
pickup_datetime: 0.00B
average_wind_speed: 0.00B
pickup_cdeligibil: 0.00B
improvement_surcharge: 0.00B
max_temperature: 0.00B
vendor_id: 0.00B
extra: 0.00B
dropoff_borocode: 0.00B
dropoff_ntacode: 0.00B
fare_amount: 0.00B
pickup: 0.00B
min_temperature: 0.00B
mta_tax: 0.00B
pickup_longitude: 0.00B
total_amount: 0.00B
passenger_count: 0.00B
trip_distance: 0.00B
tolls_amount: 0.00B
precipitation: 0.00B
pickup_boroname: 0.00B
dropoff_ctlabel: 0.00B
dropoff_puma: 0.00B
dropoff_nyct2010_gid: 0.00B
pickup_ntaname: 0.00B
tip_amount: 0.00B
dropoff_boroname: 0.00B
dropoff_ntaname: 0.00B
ehail_fee: 0.00B
pickup_puma: 0.00B
dropoff_longitude: 0.00B
payment_type: 0.00B
dropoff_latitude: 0.00B
pickup_boroct2010: 0.00B
trip_type: 0.00B
rate_code_id: 0.00B
store_and_fwd_flag: 0.00B
pickup_ntacode: 0.00B
locustdb> SELECT passenger_count, trip_distance, total_amount FROM trips ORDER BY total_amount DESC LIMIT 100;
Scanned 1.46 billion rows in 50.5s (29 million rows/s)!
passenger_count | trip_distance | total_amount
----------------+---------------+--------------
1 | 5320 | 395061160
1 | 180 | 98997039
2 | 9600 | 86161105
1 | 700 | 86160324
1 | 0 | 82603997
1 | 0 | 82603997
1 | 11620 | 68590810
1 | 1160 | 65026285
2 | 0 | 64756743
1 | 2200 | 63112198
1 | 800 | 63046312
1 | 0 | 62903378
1 | 0 | 62903378
3 | 0 | 62590160
5 | 1170 | 54846335
5 | 1800 | 54143256
1 | 0 | 53858000
1 | 0 | 53848268
2 | 0 | 50530481
1 | 1300 | 50332633
1 | 0 | 42956225
1 | 4300 | 42781990
1 | 0 | 41026766
1 | 0 | 40848698
1 | 1178600 | 40411466
1 | 0 | 39847996
1 | 0 | 39846670
2 | 0 | 39846620
1 | 0 | 39322232
1 | 1000 | 39191358
1 | 0 | 37918374
3 | 2200 | 36177202
1 | 0 | 33886334
2 | 1100 | 33554521
1 | 700 | 33541449
1 | 0 | 33340289
1 | 258800 | 32549603
1 | 0 | 32274382
1 | 45600 | 30704518
2 | 0 | 26908936
1 | 1200 | 24861943
1 | 0 | 21831709
1 | 0 | 21052379
1 | 0 | 20214955
1 | 1000 | 20054751
1 | 6800 | 19864438
2 | 2700 | 18744576
1 | 2100 | 18744426
2 | 1800 | 18744226
1 | 1800 | 18613254
2 | 3700 | 17186358
1 | 600 | 17186246
1 | 2800 | 17185699
1 | 0 | 16868839
1 | 0 | 16868789
1 | 0 | 16606012
2 | 12500 | 15900116
1 | 298914200 | 15483214
1 | 151694000 | 15329622
1 | 1714056400 | 15327117
6 | 300 | 15243838
1 | 4400 | 15099774
1 | 0 | 13690599
3 | 166331600 | 13313120
2 | 166086300 | 13312500
1 | 158600 | 13306368
2 | 131091400 | 12636658
1 | 1000 | 12451927
1 | 0 | 11605382
1 | 900 | 11127165
2 | 500 | 10778818
3 | 0 | 10744509
3 | 0 | 10108073
1 | 7500 | 9397810
2 | 1500 | 9396416
1 | 198623013600 | 9395569
1 | 536862699 | 9395270
1 | 390288800 | 9150945
1 | 13100 | 8897786
2 | 1180 | 8806493
6 | 1190 | 8633055
2 | 522133000 | 8224197
1 | 3140000000 | 7579732
1 | 1530 | 6995607
1 | 288703900 | 6155303
1 | 2200 | 5119200
1 | 100 | 3736032
1 | 357894700 | 3153439
1 | 0 | 2361976
1 | 5240 | 2254411
1 | 234410600 | 2245986
1 | 400 | 2097476
1 | 71732700 | 2004407
1 | 0 | 1401030
2 | 733400 | 1289125
4 | 0 | 1275534
3 | 0 | 1248541
1 | 1524200 | 1018021
5 | 0 | 1000099
1 | 1514910 | 1000099
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
pickup_ntacode: 0.00B
min_temperature: 0.00B
pickup_boroct2010: 0.00B
dropoff_ctlabel: 0.00B
pickup_datetime: 0.00B
pickup_boroname: 0.00B
dropoff_borocode: 0.00B
dropoff_ntacode: 0.00B
tip_amount: 0.00B
mta_tax: 0.00B
dropoff_boroct2010: 0.00B
vendor_id: 0.00B
fare_amount: 0.00B
pickup_longitude: 0.00B
dropoff_latitude: 0.00B
pickup_ctlabel: 0.00B
ehail_fee: 0.00B
dropoff_boroname: 0.00B
dropoff_longitude: 0.00B
total_amount: 0.00B
snowfall: 0.00B
precipitation: 0.00B
dropoff_datetime: 0.00B
pickup_cdeligibil: 0.00B
trip_distance: 0.00B
extra: 0.00B
store_and_fwd_flag: 0.00B
average_wind_speed: 0.00B
dropoff_puma: 0.00B
improvement_surcharge: 0.00B
dropoff_ntaname: 0.00B
dropoff: 0.00B
cab_type: 0.00B
dropoff_cdeligibil: 0.00B
pickup_ct2010: 0.00B
trip_id: 0.00B
pickup_nyct2010_gid: 0.00B
passenger_count: 0.00B
dropoff_nyct2010_gid: 0.00B
pickup_borocode: 0.00B
max_temperature: 0.00B
pickup_latitude: 0.00B
pickup_puma: 0.00B
trip_type: 0.00B
rate_code_id: 0.00B
pickup_ntaname: 0.00B
payment_type: 0.00B
pickup: 0.00B
dropoff_ct2010: 0.00B
snow_depth: 0.00B
tolls_amount: 0.00B
locustdb>
Failed to parser query: Position(Alt, [59])
locustdb> SELECT passenger_count, trip_distance, total_amount FROM trips ORDER BY total_amount DESC LIMIT 100;
Scanned 1.46 billion rows in 50.4s (29 million rows/s)!
passenger_count | trip_distance | total_amount
----------------+---------------+--------------
1 | 5320 | 395061160
1 | 180 | 98997039
2 | 9600 | 86161105
1 | 700 | 86160324
1 | 0 | 82603997
1 | 0 | 82603997
1 | 11620 | 68590810
1 | 1160 | 65026285
2 | 0 | 64756743
1 | 2200 | 63112198
1 | 800 | 63046312
1 | 0 | 62903378
1 | 0 | 62903378
3 | 0 | 62590160
5 | 1170 | 54846335
5 | 1800 | 54143256
1 | 0 | 53858000
1 | 0 | 53848268
2 | 0 | 50530481
1 | 1300 | 50332633
1 | 0 | 42956225
1 | 4300 | 42781990
1 | 0 | 41026766
1 | 0 | 40848698
1 | 1178600 | 40411466
1 | 0 | 39847996
1 | 0 | 39846670
2 | 0 | 39846620
1 | 0 | 39322232
1 | 1000 | 39191358
1 | 0 | 37918374
3 | 2200 | 36177202
1 | 0 | 33886334
2 | 1100 | 33554521
1 | 700 | 33541449
1 | 0 | 33340289
1 | 258800 | 32549603
1 | 0 | 32274382
1 | 45600 | 30704518
2 | 0 | 26908936
1 | 1200 | 24861943
1 | 0 | 21831709
1 | 0 | 21052379
1 | 0 | 20214955
1 | 1000 | 20054751
1 | 6800 | 19864438
2 | 2700 | 18744576
1 | 2100 | 18744426
2 | 1800 | 18744226
1 | 1800 | 18613254
2 | 3700 | 17186358
1 | 600 | 17186246
1 | 2800 | 17185699
1 | 0 | 16868839
1 | 0 | 16868789
1 | 0 | 16606012
2 | 12500 | 15900116
1 | 298914200 | 15483214
1 | 151694000 | 15329622
1 | 1714056400 | 15327117
6 | 300 | 15243838
1 | 4400 | 15099774
1 | 0 | 13690599
3 | 166331600 | 13313120
2 | 166086300 | 13312500
1 | 158600 | 13306368
2 | 131091400 | 12636658
1 | 1000 | 12451927
1 | 0 | 11605382
1 | 900 | 11127165
2 | 500 | 10778818
3 | 0 | 10744509
3 | 0 | 10108073
1 | 7500 | 9397810
2 | 1500 | 9396416
1 | 198623013600 | 9395569
1 | 536862699 | 9395270
1 | 390288800 | 9150945
1 | 13100 | 8897786
2 | 1180 | 8806493
6 | 1190 | 8633055
2 | 522133000 | 8224197
1 | 3140000000 | 7579732
1 | 1530 | 6995607
1 | 288703900 | 6155303
1 | 2200 | 5119200
1 | 100 | 3736032
1 | 357894700 | 3153439
1 | 0 | 2361976
1 | 5240 | 2254411
1 | 234410600 | 2245986
1 | 400 | 2097476
1 | 71732700 | 2004407
1 | 0 | 1401030
2 | 733400 | 1289125
4 | 0 | 1275534
3 | 0 | 1248541
1 | 1524200 | 1018021
1 | 1514910 | 1000099
5 | 0 | 1000099
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
dropoff_cdeligibil: 0.00B
tolls_amount: 0.00B
dropoff_latitude: 0.00B
dropoff_borocode: 0.00B
dropoff_ct2010: 0.00B
dropoff: 0.00B
pickup_borocode: 0.00B
improvement_surcharge: 0.00B
dropoff_datetime: 0.00B
rate_code_id: 0.00B
pickup_longitude: 0.00B
tip_amount: 0.00B
dropoff_puma: 0.00B
pickup_boroct2010: 0.00B
total_amount: 0.00B
pickup_nyct2010_gid: 0.00B
max_temperature: 0.00B
dropoff_ntacode: 0.00B
dropoff_boroct2010: 0.00B
passenger_count: 0.00B
ehail_fee: 0.00B
pickup_ntaname: 0.00B
dropoff_ntaname: 0.00B
snow_depth: 0.00B
dropoff_longitude: 0.00B
pickup_ctlabel: 0.00B
pickup_boroname: 0.00B
trip_type: 0.00B
payment_type: 0.00B
pickup_cdeligibil: 0.00B
fare_amount: 0.00B
pickup_ntacode: 0.00B
trip_distance: 0.00B
snowfall: 0.00B
dropoff_boroname: 0.00B
trip_id: 0.00B
mta_tax: 0.00B
extra: 0.00B
dropoff_ctlabel: 0.00B
cab_type: 0.00B
vendor_id: 0.00B
pickup: 0.00B
average_wind_speed: 0.00B
pickup_puma: 0.00B
precipitation: 0.00B
pickup_ct2010: 0.00B
dropoff_nyct2010_gid: 0.00B
pickup_latitude: 0.00B
pickup_datetime: 0.00B
min_temperature: 0.00B
store_and_fwd_flag: 0.00B
locustdb> SELECT passenger_count, trip_distance, total_amount FROM trips ORDER BY total_amount DESC LIMIT 100;
Scanned 1.46 billion rows in 50.5s (29 million rows/s)!
passenger_count | trip_distance | total_amount
----------------+---------------+--------------
1 | 5320 | 395061160
1 | 180 | 98997039
2 | 9600 | 86161105
1 | 700 | 86160324
1 | 0 | 82603997
1 | 0 | 82603997
1 | 11620 | 68590810
1 | 1160 | 65026285
2 | 0 | 64756743
1 | 2200 | 63112198
1 | 800 | 63046312
1 | 0 | 62903378
1 | 0 | 62903378
3 | 0 | 62590160
5 | 1170 | 54846335
5 | 1800 | 54143256
1 | 0 | 53858000
1 | 0 | 53848268
2 | 0 | 50530481
1 | 1300 | 50332633
1 | 0 | 42956225
1 | 4300 | 42781990
1 | 0 | 41026766
1 | 0 | 40848698
1 | 1178600 | 40411466
1 | 0 | 39847996
1 | 0 | 39846670
2 | 0 | 39846620
1 | 0 | 39322232
1 | 1000 | 39191358
1 | 0 | 37918374
3 | 2200 | 36177202
1 | 0 | 33886334
2 | 1100 | 33554521
1 | 700 | 33541449
1 | 0 | 33340289
1 | 258800 | 32549603
1 | 0 | 32274382
1 | 45600 | 30704518
2 | 0 | 26908936
1 | 1200 | 24861943
1 | 0 | 21831709
1 | 0 | 21052379
1 | 0 | 20214955
1 | 1000 | 20054751
1 | 6800 | 19864438
2 | 2700 | 18744576
1 | 2100 | 18744426
2 | 1800 | 18744226
1 | 1800 | 18613254
2 | 3700 | 17186358
1 | 600 | 17186246
1 | 2800 | 17185699
1 | 0 | 16868839
1 | 0 | 16868789
1 | 0 | 16606012
2 | 12500 | 15900116
1 | 298914200 | 15483214
1 | 151694000 | 15329622
1 | 1714056400 | 15327117
6 | 300 | 15243838
1 | 4400 | 15099774
1 | 0 | 13690599
3 | 166331600 | 13313120
2 | 166086300 | 13312500
1 | 158600 | 13306368
2 | 131091400 | 12636658
1 | 1000 | 12451927
1 | 0 | 11605382
1 | 900 | 11127165
2 | 500 | 10778818
3 | 0 | 10744509
3 | 0 | 10108073
1 | 7500 | 9397810
2 | 1500 | 9396416
1 | 198623013600 | 9395569
1 | 536862699 | 9395270
1 | 390288800 | 9150945
1 | 13100 | 8897786
2 | 1180 | 8806493
6 | 1190 | 8633055
2 | 522133000 | 8224197
1 | 3140000000 | 7579732
1 | 1530 | 6995607
1 | 288703900 | 6155303
1 | 2200 | 5119200
1 | 100 | 3736032
1 | 357894700 | 3153439
1 | 0 | 2361976
1 | 5240 | 2254411
1 | 234410600 | 2245986
1 | 400 | 2097476
1 | 71732700 | 2004407
1 | 0 | 1401030
2 | 733400 | 1289125
4 | 0 | 1275534
3 | 0 | 1248541
1 | 1524200 | 1018021
5 | 0 | 1000099
1 | 1514910 | 1000099
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
cab_type: 0.00B
pickup_nyct2010_gid: 0.00B
pickup_latitude: 0.00B
rate_code_id: 0.00B
pickup_borocode: 0.00B
extra: 0.00B
min_temperature: 0.00B
improvement_surcharge: 0.00B
dropoff_puma: 0.00B
dropoff_borocode: 0.00B
pickup_ntaname: 0.00B
dropoff_cdeligibil: 0.00B
pickup_ctlabel: 0.00B
vendor_id: 0.00B
fare_amount: 0.00B
snowfall: 0.00B
trip_type: 0.00B
ehail_fee: 0.00B
pickup_puma: 0.00B
dropoff_boroname: 0.00B
dropoff_boroct2010: 0.00B
tip_amount: 0.00B
total_amount: 0.00B
pickup_boroname: 0.00B
pickup_boroct2010: 0.00B
dropoff: 0.00B
snow_depth: 0.00B
dropoff_ntaname: 0.00B
dropoff_longitude: 0.00B
pickup_cdeligibil: 0.00B
dropoff_latitude: 0.00B
max_temperature: 0.00B
trip_id: 0.00B
pickup: 0.00B
tolls_amount: 0.00B
dropoff_nyct2010_gid: 0.00B
average_wind_speed: 0.00B
pickup_datetime: 0.00B
pickup_ct2010: 0.00B
passenger_count: 0.00B
payment_type: 0.00B
mta_tax: 0.00B
pickup_ntacode: 0.00B
precipitation: 0.00B
dropoff_ctlabel: 0.00B
pickup_longitude: 0.00B
trip_distance: 0.00B
dropoff_ntacode: 0.00B
dropoff_datetime: 0.00B
store_and_fwd_flag: 0.00B
dropoff_ct2010: 0.00B
locustdb> SELECT pickup_puma, dropoff_puma, passenger_count, count(0) from trips;
Scanned 1.46 billion rows in 16.8s (87 million rows/s)!
pickup_puma | dropoff_puma | passenger_count | count_0
------------+--------------+-----------------+-----------
0 | 0 | 0 | 239636
0 | 0 | 1 | 158504430
0 | 0 | 2 | 30589460
0 | 0 | 3 | 8775405
0 | 0 | 4 | 4182591
0 | 0 | 5 | 11051643
0 | 0 | 6 | 6082326
0 | 0 | 7 | 1168
0 | 0 | 8 | 1107
0 | 0 | 9 | 703
0 | 0 | 10 | 14
0 | 0 | 36 | 1
0 | 0 | 47 | 1
0 | 0 | 49 | 26
0 | 0 | 58 | 2
0 | 0 | 70 | 1
0 | 0 | 84 | 1
0 | 0 | 113 | 1
0 | 0 | 134 | 1
0 | 0 | 137 | 1
0 | 0 | 141 | 1
0 | 0 | 155 | 1
0 | 0 | 158 | 1
0 | 0 | 160 | 1
0 | 0 | 163 | 1
0 | 0 | 164 | 1
0 | 0 | 165 | 1
0 | 0 | 192 | 1
0 | 0 | 208 | 812
0 | 0 | 213 | 4
0 | 0 | 225 | 1
0 | 0 | 247 | 1
0 | 0 | 255 | 6
0 | 3701 | 0 | 13
0 | 3701 | 1 | 3593
0 | 3701 | 2 | 760
0 | 3701 | 3 | 209
0 | 3701 | 4 | 69
0 | 3701 | 5 | 417
0 | 3701 | 6 | 56
0 | 3701 | 7 | 2
0 | 3701 | 8 | 2
0 | 3701 | 9 | 1
0 | 3702 | 0 | 12
0 | 3702 | 1 | 2706
0 | 3702 | 2 | 511
0 | 3702 | 3 | 131
0 | 3702 | 4 | 56
0 | 3702 | 5 | 291
0 | 3702 | 6 | 58
0 | 3702 | 8 | 1
0 | 3703 | 0 | 5
0 | 3703 | 1 | 3673
0 | 3703 | 2 | 796
0 | 3703 | 3 | 237
0 | 3703 | 4 | 95
0 | 3703 | 5 | 614
0 | 3703 | 6 | 47
0 | 3703 | 8 | 1
0 | 3704 | 0 | 13
0 | 3704 | 1 | 1850
0 | 3704 | 2 | 390
0 | 3704 | 3 | 81
0 | 3704 | 4 | 43
0 | 3704 | 5 | 164
0 | 3704 | 6 | 23
0 | 3704 | 8 | 2
0 | 3705 | 0 | 13
0 | 3705 | 1 | 2913
0 | 3705 | 2 | 594
0 | 3705 | 3 | 186
0 | 3705 | 4 | 69
0 | 3705 | 5 | 317
0 | 3705 | 6 | 43
0 | 3705 | 8 | 1
0 | 3706 | 0 | 9
0 | 3706 | 1 | 2551
0 | 3706 | 2 | 469
0 | 3706 | 3 | 122
0 | 3706 | 4 | 39
0 | 3706 | 5 | 211
0 | 3706 | 6 | 28
0 | 3706 | 8 | 4
0 | 3706 | 9 | 1
0 | 3707 | 0 | 8
0 | 3707 | 1 | 2695
0 | 3707 | 2 | 523
0 | 3707 | 3 | 167
0 | 3707 | 4 | 67
0 | 3707 | 5 | 255
0 | 3707 | 6 | 34
0 | 3707 | 8 | 1
0 | 3708 | 0 | 16
0 | 3708 | 1 | 4899
0 | 3708 | 2 | 1045
0 | 3708 | 3 | 310
0 | 3708 | 4 | 128
0 | 3708 | 5 | 768
0 | 3708 | 6 | 50
0 | 3708 | 7 | 3
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
trip_distance: 0.00B
dropoff: 0.00B
pickup_cdeligibil: 0.00B
extra: 0.00B
pickup: 0.00B
min_temperature: 0.00B
tolls_amount: 0.00B
dropoff_ct2010: 0.00B
snow_depth: 0.00B
passenger_count: 0.00B
pickup_boroname: 0.00B
pickup_longitude: 0.00B
pickup_boroct2010: 0.00B
precipitation: 0.00B
average_wind_speed: 0.00B
dropoff_boroname: 0.00B
rate_code_id: 0.00B
trip_type: 0.00B
dropoff_boroct2010: 0.00B
max_temperature: 0.00B
pickup_latitude: 0.00B
pickup_ntaname: 0.00B
dropoff_cdeligibil: 0.00B
vendor_id: 0.00B
fare_amount: 0.00B
snowfall: 0.00B
dropoff_ctlabel: 0.00B
pickup_ntacode: 0.00B
trip_id: 0.00B
dropoff_ntacode: 0.00B
pickup_puma: 0.00B
cab_type: 0.00B
pickup_ct2010: 0.00B
ehail_fee: 0.00B
dropoff_ntaname: 0.00B
dropoff_puma: 0.00B
payment_type: 0.00B
dropoff_longitude: 0.00B
total_amount: 0.00B
pickup_ctlabel: 0.00B
dropoff_datetime: 0.00B
pickup_nyct2010_gid: 0.00B
store_and_fwd_flag: 0.00B
dropoff_nyct2010_gid: 0.00B
dropoff_borocode: 0.00B
mta_tax: 0.00B
dropoff_latitude: 0.00B
pickup_borocode: 0.00B
pickup_datetime: 0.00B
tip_amount: 0.00B
improvement_surcharge: 0.00B
locustdb> SELECT pickup_puma, dropoff_puma, passenger_count, count(0) from trips;
Scanned 1.46 billion rows in 17.0s (86 million rows/s)!
pickup_puma | dropoff_puma | passenger_count | count_0
------------+--------------+-----------------+-----------
0 | 0 | 0 | 239636
0 | 0 | 1 | 158504430
0 | 0 | 2 | 30589460
0 | 0 | 3 | 8775405
0 | 0 | 4 | 4182591
0 | 0 | 5 | 11051643
0 | 0 | 6 | 6082326
0 | 0 | 7 | 1168
0 | 0 | 8 | 1107
0 | 0 | 9 | 703
0 | 0 | 10 | 14
0 | 0 | 36 | 1
0 | 0 | 47 | 1
0 | 0 | 49 | 26
0 | 0 | 58 | 2
0 | 0 | 70 | 1
0 | 0 | 84 | 1
0 | 0 | 113 | 1
0 | 0 | 134 | 1
0 | 0 | 137 | 1
0 | 0 | 141 | 1
0 | 0 | 155 | 1
0 | 0 | 158 | 1
0 | 0 | 160 | 1
0 | 0 | 163 | 1
0 | 0 | 164 | 1
0 | 0 | 165 | 1
0 | 0 | 192 | 1
0 | 0 | 208 | 812
0 | 0 | 213 | 4
0 | 0 | 225 | 1
0 | 0 | 247 | 1
0 | 0 | 255 | 6
0 | 3701 | 0 | 13
0 | 3701 | 1 | 3593
0 | 3701 | 2 | 760
0 | 3701 | 3 | 209
0 | 3701 | 4 | 69
0 | 3701 | 5 | 417
0 | 3701 | 6 | 56
0 | 3701 | 7 | 2
0 | 3701 | 8 | 2
0 | 3701 | 9 | 1
0 | 3702 | 0 | 12
0 | 3702 | 1 | 2706
0 | 3702 | 2 | 511
0 | 3702 | 3 | 131
0 | 3702 | 4 | 56
0 | 3702 | 5 | 291
0 | 3702 | 6 | 58
0 | 3702 | 8 | 1
0 | 3703 | 0 | 5
0 | 3703 | 1 | 3673
0 | 3703 | 2 | 796
0 | 3703 | 3 | 237
0 | 3703 | 4 | 95
0 | 3703 | 5 | 614
0 | 3703 | 6 | 47
0 | 3703 | 8 | 1
0 | 3704 | 0 | 13
0 | 3704 | 1 | 1850
0 | 3704 | 2 | 390
0 | 3704 | 3 | 81
0 | 3704 | 4 | 43
0 | 3704 | 5 | 164
0 | 3704 | 6 | 23
0 | 3704 | 8 | 2
0 | 3705 | 0 | 13
0 | 3705 | 1 | 2913
0 | 3705 | 2 | 594
0 | 3705 | 3 | 186
0 | 3705 | 4 | 69
0 | 3705 | 5 | 317
0 | 3705 | 6 | 43
0 | 3705 | 8 | 1
0 | 3706 | 0 | 9
0 | 3706 | 1 | 2551
0 | 3706 | 2 | 469
0 | 3706 | 3 | 122
0 | 3706 | 4 | 39
0 | 3706 | 5 | 211
0 | 3706 | 6 | 28
0 | 3706 | 8 | 4
0 | 3706 | 9 | 1
0 | 3707 | 0 | 8
0 | 3707 | 1 | 2695
0 | 3707 | 2 | 523
0 | 3707 | 3 | 167
0 | 3707 | 4 | 67
0 | 3707 | 5 | 255
0 | 3707 | 6 | 34
0 | 3707 | 8 | 1
0 | 3708 | 0 | 16
0 | 3708 | 1 | 4899
0 | 3708 | 2 | 1045
0 | 3708 | 3 | 310
0 | 3708 | 4 | 128
0 | 3708 | 5 | 768
0 | 3708 | 6 | 50
0 | 3708 | 7 | 3
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /mnt/data/ldb --mem-limit-tables=45 --mem-lz4 --seq-disk-read`
# Table `trips` (1464785771 rows, 192KiB) #
pickup_ct2010: 0.00B
fare_amount: 0.00B
dropoff_ntaname: 0.00B
pickup_longitude: 0.00B
vendor_id: 0.00B
pickup_datetime: 0.00B
tolls_amount: 0.00B
tip_amount: 0.00B
dropoff_ct2010: 0.00B
dropoff_latitude: 0.00B
passenger_count: 0.00B
trip_type: 0.00B
pickup_ntacode: 0.00B
pickup_puma: 0.00B
snow_depth: 0.00B
dropoff_ctlabel: 0.00B
dropoff_boroname: 0.00B
rate_code_id: 0.00B
dropoff_longitude: 0.00B
payment_type: 0.00B
pickup_nyct2010_gid: 0.00B
average_wind_speed: 0.00B
store_and_fwd_flag: 0.00B
total_amount: 0.00B
pickup_latitude: 0.00B
pickup_ntaname: 0.00B
min_temperature: 0.00B
cab_type: 0.00B
pickup_cdeligibil: 0.00B
snowfall: 0.00B
dropoff_cdeligibil: 0.00B
extra: 0.00B
trip_id: 0.00B
max_temperature: 0.00B
ehail_fee: 0.00B
dropoff: 0.00B
pickup_ctlabel: 0.00B
pickup_boroname: 0.00B
pickup_boroct2010: 0.00B
dropoff_borocode: 0.00B
dropoff_ntacode: 0.00B
dropoff_datetime: 0.00B
improvement_surcharge: 0.00B
dropoff_nyct2010_gid: 0.00B
precipitation: 0.00B
pickup_borocode: 0.00B
dropoff_boroct2010: 0.00B
trip_distance: 0.00B
dropoff_puma: 0.00B
pickup: 0.00B
mta_tax: 0.00B
locustdb> SELECT pickup_puma, dropoff_puma, passenger_count, count(0) from trips;
Scanned 1.46 billion rows in 17.0s (86 million rows/s)!
pickup_puma | dropoff_puma | passenger_count | count_0
------------+--------------+-----------------+-----------
0 | 0 | 0 | 239636
0 | 0 | 1 | 158504430
0 | 0 | 2 | 30589460
0 | 0 | 3 | 8775405
0 | 0 | 4 | 4182591
0 | 0 | 5 | 11051643
0 | 0 | 6 | 6082326
0 | 0 | 7 | 1168
0 | 0 | 8 | 1107
0 | 0 | 9 | 703
0 | 0 | 10 | 14
0 | 0 | 36 | 1
0 | 0 | 47 | 1
0 | 0 | 49 | 26
0 | 0 | 58 | 2
0 | 0 | 70 | 1
0 | 0 | 84 | 1
0 | 0 | 113 | 1
0 | 0 | 134 | 1
0 | 0 | 137 | 1
0 | 0 | 141 | 1
0 | 0 | 155 | 1
0 | 0 | 158 | 1
0 | 0 | 160 | 1
0 | 0 | 163 | 1
0 | 0 | 164 | 1
0 | 0 | 165 | 1
0 | 0 | 192 | 1
0 | 0 | 208 | 812
0 | 0 | 213 | 4
0 | 0 | 225 | 1
0 | 0 | 247 | 1
0 | 0 | 255 | 6
0 | 3701 | 0 | 13
0 | 3701 | 1 | 3593
0 | 3701 | 2 | 760
0 | 3701 | 3 | 209
0 | 3701 | 4 | 69
0 | 3701 | 5 | 417
0 | 3701 | 6 | 56
0 | 3701 | 7 | 2
0 | 3701 | 8 | 2
0 | 3701 | 9 | 1
0 | 3702 | 0 | 12
0 | 3702 | 1 | 2706
0 | 3702 | 2 | 511
0 | 3702 | 3 | 131
0 | 3702 | 4 | 56
0 | 3702 | 5 | 291
0 | 3702 | 6 | 58
0 | 3702 | 8 | 1
0 | 3703 | 0 | 5
0 | 3703 | 1 | 3673
0 | 3703 | 2 | 796
0 | 3703 | 3 | 237
0 | 3703 | 4 | 95
0 | 3703 | 5 | 614
0 | 3703 | 6 | 47
0 | 3703 | 8 | 1
0 | 3704 | 0 | 13
0 | 3704 | 1 | 1850
0 | 3704 | 2 | 390
0 | 3704 | 3 | 81
0 | 3704 | 4 | 43
0 | 3704 | 5 | 164
0 | 3704 | 6 | 23
0 | 3704 | 8 | 2
0 | 3705 | 0 | 13
0 | 3705 | 1 | 2913
0 | 3705 | 2 | 594
0 | 3705 | 3 | 186
0 | 3705 | 4 | 69
0 | 3705 | 5 | 317
0 | 3705 | 6 | 43
0 | 3705 | 8 | 1
0 | 3706 | 0 | 9
0 | 3706 | 1 | 2551
0 | 3706 | 2 | 469
0 | 3706 | 3 | 122
0 | 3706 | 4 | 39
0 | 3706 | 5 | 211
0 | 3706 | 6 | 28
0 | 3706 | 8 | 4
0 | 3706 | 9 | 1
0 | 3707 | 0 | 8
0 | 3707 | 1 | 2695
0 | 3707 | 2 | 523
0 | 3707 | 3 | 167
0 | 3707 | 4 | 67
0 | 3707 | 5 | 255
0 | 3707 | 6 | 34
0 | 3707 | 8 | 1
0 | 3708 | 0 | 16
0 | 3708 | 1 | 4899
0 | 3708 | 2 | 1045
0 | 3708 | 3 | 310
0 | 3708 | 4 | 128
0 | 3708 | 5 | 768
0 | 3708 | 6 | 50
0 | 3708 | 7 | 3
locustdb>
➜ LocustDB git:(master) lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 477G 0 disk
├─sda1 8:1 0 512M 0 part
├─sda2 8:2 0 1M 0 part
├─sda3 8:3 0 15.9G 0 part [SWAP]
└─sda4 8:4 0 460.6G 0 part /
sdb 8:16 0 477G 0 disk
└─sdb1 8:17 0 477G 0 part
sdc 8:32 0 1.8T 0 disk
├─sdc1 8:33 0 128M 0 part
└─sdc2 8:34 0 1.8T 0 part
sdd 8:48 0 1.8T 0 disk
└─sdd1 8:49 0 1.8T 0 part /mnt/data
➜ LocustDB git:(master) sudo hdparm -t /dev/sdd1
/dev/sdd1:
Timing buffered disk reads: 538 MB in 3.00 seconds = 179.19 MB/sec
➜ LocustDB git:(master) sudo hdparm -t /dev/sdd1
/dev/sdd1:
Timing buffered disk reads: 536 MB in 3.00 seconds = 178.58 MB/sec
➜ LocustDB git:(master) sudo hdparm -t /dev/sdd1
/dev/sdd1:
Timing buffered disk reads: 538 MB in 3.01 seconds = 179.00 MB/sec
➜ LocustDB git:(master) fio --name TEST --eta-newline=1s --filename=/mnt/data/fio --rw=read --size=2g --blocksize=1024k --ioengine=libaio --iodepth=32 --direct=1 --numjobs=1 --group_reporting
TEST: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=32
fio-2.2.10
Starting 1 process
TEST: Laying out IO file(s) (1 file(s) / 2048MB)
Jobs: 1 (f=1): [R(1)] [33.3% done] [189.0MB/0KB/0KB /s] [189/0/0 iops] [eta 00m:08s]
Jobs: 1 (f=1): [R(1)] [50.0% done] [183.0MB/0KB/0KB /s] [183/0/0 iops] [eta 00m:06s]
Jobs: 1 (f=1): [R(1)] [66.7% done] [183.0MB/0KB/0KB /s] [183/0/0 iops] [eta 00m:04s]
Jobs: 1 (f=1): [R(1)] [83.3% done] [169.0MB/0KB/0KB /s] [169/0/0 iops] [eta 00m:02s]
Jobs: 1 (f=1): [R(1)] [100.0% done] [149.0MB/0KB/0KB /s] [149/0/0 iops] [eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=5406: Sun Aug 12 00:27:51 2018
read : io=2048.0MB, bw=179183KB/s, iops=174, runt= 11704msec
slat (usec): min=95, max=724, avg=200.66, stdev=40.90
clat (msec): min=12, max=1526, avg=182.46, stdev=163.50
lat (msec): min=12, max=1526, avg=182.66, stdev=163.50
clat percentiles (msec):
| 1.00th=[ 38], 5.00th=[ 42], 10.00th=[ 52], 20.00th=[ 98],
| 30.00th=[ 120], 40.00th=[ 137], 50.00th=[ 159], 60.00th=[ 178],
| 70.00th=[ 194], 80.00th=[ 210], 90.00th=[ 245], 95.00th=[ 457],
| 99.00th=[ 1020], 99.50th=[ 1319], 99.90th=[ 1516], 99.95th=[ 1516],
| 99.99th=[ 1532]
bw (KB /s): min=99096, max=205206, per=99.07%, avg=177519.83, stdev=23824.02
lat (msec) : 20=0.15%, 50=9.42%, 100=11.87%, 250=69.14%, 500=5.13%
lat (msec) : 750=2.73%, 1000=0.39%, 2000=1.17%
cpu : usr=0.27%, sys=3.86%, ctx=2048, majf=0, minf=538
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=2048/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: io=2048.0MB, aggrb=179182KB/s, minb=179182KB/s, maxb=179182KB/s, mint=11704msec, maxt=11704msec
Disk stats (read/write):
sdd: ios=2045/2, merge=0/1, ticks=370092/376, in_queue=371068, util=99.23%
➜ LocustDB git:(master)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment