Skip to content

Instantly share code, notes, and snippets.

@cswinter

cswinter/locustdb-cold-query-ssd

Created August 12, 2018 17:45
Star
Embed
What would you like to do?
Learn more about clone URLs
Download ZIP
Raw
locustdb-cold-query-ssd
➜ ~ uname -a
Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
➜ ~ sudo lshw -class disk -class storage
[sudo] password for clemens:
*-storage
description: SATA controller
product: ASM1062 Serial ATA Controller
vendor: ASMedia Technology Inc.
physical id: 0
bus info: pci@0000:06:00.0
version: 02
width: 32 bits
clock: 33MHz
capabilities: storage msi pm pciexpress ahci_1.0 bus_master cap_list
configuration: driver=ahci latency=0
resources: irq:43 ioport:d050(size=8) ioport:d040(size=4) ioport:d030(size=8) ioport:d020(size=4) ioport:d000(size=32) memory:fb200000-fb2001ff
*-storage
description: SATA controller
product: C610/X99 series chipset 6-Port SATA Controller [AHCI mode]
vendor: Intel Corporation
physical id: 1f.2
bus info: pci@0000:00:1f.2
version: 05
width: 32 bits
clock: 66MHz
capabilities: storage msi pm ahci_1.0 bus_master cap_list
configuration: driver=ahci latency=0
resources: irq:42 ioport:f070(size=8) ioport:f060(size=4) ioport:f050(size=8) ioport:f040(size=4) ioport:f000(size=32) memory:fb336000-fb3367ff
*-scsi:0
physical id: 1
logical name: scsi0
capabilities: emulated
*-disk
description: ATA Disk
product: Crucial_CT512MX1
physical id: 0.0.0
bus info: scsi@0:0.0.0
logical name: /dev/sda
version: MU01
serial: 14340CFD629E
size: 476GiB (512GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=75bae818-9153-47eb-9cee-c01f70c5c42b logicalsectorsize=512 sectorsize=4096
*-scsi:1
physical id: 2
logical name: scsi1
capabilities: emulated
*-disk
description: ATA Disk
product: Crucial_CT512MX1
physical id: 0.0.0
bus info: scsi@1:0.0.0
logical name: /dev/sdb
version: MU01
serial: 14330CF93238
size: 476GiB (512GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=b3e05cb9-ac11-46b2-a41b-5cf4164bcb34 logicalsectorsize=512 sectorsize=4096
*-scsi:2
physical id: 3
logical name: scsi2
capabilities: emulated
*-disk
description: ATA Disk
product: ST2000DM006-2DM1
vendor: Seagate
physical id: 0.0.0
bus info: scsi@2:0.0.0
logical name: /dev/sdc
version: CC26
serial: Z4Z83385
size: 1863GiB (2TB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=86fe1a86-58ce-448b-bb84-1461f00ab21f logicalsectorsize=512 sectorsize=4096
*-scsi:3
physical id: 4
logical name: scsi3
capabilities: emulated
*-disk
description: ATA Disk
product: ST2000DM006-2DM1
vendor: Seagate
physical id: 0.0.0
bus info: scsi@3:0.0.0
logical name: /dev/sdd
version: CC26
serial: Z4ZB7EV9
size: 1863GiB (2TB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=fa606d24-f1b6-4661-ab08-41e3bae03746 logicalsectorsize=512 sectorsize=4096
➜ ~ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 477G 0 disk
├─sda1 8:1 0 512M 0 part
├─sda2 8:2 0 1M 0 part
├─sda3 8:3 0 15.9G 0 part [SWAP]
└─sda4 8:4 0 460.6G 0 part /
sdb 8:16 0 477G 0 disk
└─sdb1 8:17 0 477G 0 part
sdc 8:32 0 1.8T 0 disk
├─sdc1 8:33 0 128M 0 part
└─sdc2 8:34 0 1.8T 0 part
sdd 8:48 0 1.8T 0 disk
└─sdd1 8:49 0 1.8T 0 part /mnt/data
➜ ~ ulimit -n 4096
➜ ~ dstat
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
10 0 89 1 0 0|8071k 3516k| 0 0 | 0 0 |1784 3824
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 34 87
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 54 172
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 25 92
0 0 100 0 0 0| 0 80k| 309B 0 | 0 0 | 26 64
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 30 148
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 34 83
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 20 44
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 23 142
0 0 100 0 0 0| 0 8192B| 309B 0 | 0 0 | 18 42
0 0 100 0 0 0| 0 0 | 309B 0 | 0 0 | 36 70
0 0 100 0 0 0| 0 0 |2108B 0 | 0 0 | 39 173 ^C
➜ ~ cd src/LocustDB
➜ LocustDB git:(master) git rev-parse HEAD
6bcbfcba3093c98ce2a062c493a855eff45acacb
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
dropoff_ntacode: 0.00B
dropoff_cdeligibil: 0.00B
dropoff_datetime: 0.00B
dropoff_boroname: 0.00B
dropoff_puma: 0.00B
tolls_amount: 0.00B
pickup_nyct2010_gid: 0.00B
store_and_fwd_flag: 0.00B
trip_distance: 0.00B
vendor_id: 0.00B
min_temperature: 0.00B
dropoff_latitude: 0.00B
pickup_boroct2010: 0.00B
payment_type: 0.00B
pickup_cdeligibil: 0.00B
dropoff_longitude: 0.00B
trip_id: 0.00B
pickup_ctlabel: 0.00B
max_temperature: 0.00B
dropoff_ctlabel: 0.00B
trip_type: 0.00B
passenger_count: 0.00B
dropoff: 0.00B
improvement_surcharge: 0.00B
dropoff_boroct2010: 0.00B
pickup_boroname: 0.00B
pickup_ntaname: 0.00B
pickup_latitude: 0.00B
total_amount: 0.00B
pickup_longitude: 0.00B
snowfall: 0.00B
dropoff_borocode: 0.00B
tip_amount: 0.00B
dropoff_ct2010: 0.00B
mta_tax: 0.00B
rate_code_id: 0.00B
extra: 0.00B
pickup_puma: 0.00B
pickup_borocode: 0.00B
ehail_fee: 0.00B
cab_type: 0.00B
pickup_ntacode: 0.00B
dropoff_ntaname: 0.00B
average_wind_speed: 0.00B
dropoff_nyct2010_gid: 0.00B
pickup_ct2010: 0.00B
precipitation: 0.00B
snow_depth: 0.00B
fare_amount: 0.00B
pickup: 0.00B
pickup_datetime: 0.00B
locustdb> SELECT cab_type, count(0) FROM trips;
Scanned 1.46 billion rows in 668ms (2.2 billion rows/s)!
cab_type | count_0
---------+------------
"green" | 64407776
"yellow" | 1400377995
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
total_amount: 0.00B
pickup_ntaname: 0.00B
dropoff: 0.00B
trip_type: 0.00B
max_temperature: 0.00B
pickup_ct2010: 0.00B
dropoff_boroct2010: 0.00B
precipitation: 0.00B
pickup: 0.00B
ehail_fee: 0.00B
pickup_longitude: 0.00B
pickup_boroname: 0.00B
pickup_cdeligibil: 0.00B
pickup_puma: 0.00B
dropoff_ct2010: 0.00B
pickup_datetime: 0.00B
tolls_amount: 0.00B
pickup_borocode: 0.00B
dropoff_latitude: 0.00B
rate_code_id: 0.00B
pickup_nyct2010_gid: 0.00B
average_wind_speed: 0.00B
trip_distance: 0.00B
dropoff_nyct2010_gid: 0.00B
vendor_id: 0.00B
snowfall: 0.00B
pickup_boroct2010: 0.00B
snow_depth: 0.00B
dropoff_datetime: 0.00B
pickup_latitude: 0.00B
dropoff_ntaname: 0.00B
tip_amount: 0.00B
trip_id: 0.00B
dropoff_boroname: 0.00B
store_and_fwd_flag: 0.00B
dropoff_ctlabel: 0.00B
pickup_ctlabel: 0.00B
dropoff_puma: 0.00B
cab_type: 0.00B
dropoff_cdeligibil: 0.00B
improvement_surcharge: 0.00B
fare_amount: 0.00B
min_temperature: 0.00B
dropoff_borocode: 0.00B
dropoff_longitude: 0.00B
payment_type: 0.00B
pickup_ntacode: 0.00B
mta_tax: 0.00B
dropoff_ntacode: 0.00B
passenger_count: 0.00B
extra: 0.00B
locustdb> SELECT cab_type, count(0) FROM trips;
Scanned 1.46 billion rows in 660ms (2.2 billion rows/s)!
cab_type | count_0
---------+------------
"green" | 64407776
"yellow" | 1400377995
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
dropoff_ct2010: 0.00B
pickup_ct2010: 0.00B
dropoff_ntacode: 0.00B
dropoff_nyct2010_gid: 0.00B
pickup_latitude: 0.00B
trip_id: 0.00B
total_amount: 0.00B
dropoff_longitude: 0.00B
pickup_nyct2010_gid: 0.00B
vendor_id: 0.00B
store_and_fwd_flag: 0.00B
pickup: 0.00B
pickup_ntacode: 0.00B
cab_type: 0.00B
passenger_count: 0.00B
tolls_amount: 0.00B
snowfall: 0.00B
snow_depth: 0.00B
pickup_ntaname: 0.00B
trip_type: 0.00B
min_temperature: 0.00B
pickup_borocode: 0.00B
dropoff_cdeligibil: 0.00B
dropoff_borocode: 0.00B
dropoff_boroct2010: 0.00B
max_temperature: 0.00B
ehail_fee: 0.00B
rate_code_id: 0.00B
dropoff_ntaname: 0.00B
fare_amount: 0.00B
extra: 0.00B
pickup_longitude: 0.00B
mta_tax: 0.00B
pickup_ctlabel: 0.00B
pickup_boroct2010: 0.00B
pickup_cdeligibil: 0.00B
dropoff_latitude: 0.00B
trip_distance: 0.00B
payment_type: 0.00B
tip_amount: 0.00B
dropoff_datetime: 0.00B
pickup_puma: 0.00B
dropoff_boroname: 0.00B
pickup_datetime: 0.00B
precipitation: 0.00B
dropoff_puma: 0.00B
pickup_boroname: 0.00B
dropoff_ctlabel: 0.00B
dropoff: 0.00B
improvement_surcharge: 0.00B
average_wind_speed: 0.00B
locustdb> SELECT cab_type, count(0) FROM trips;
Scanned 1.46 billion rows in 662ms (2.2 billion rows/s)!
cab_type | count_0
---------+------------
"green" | 64407776
"yellow" | 1400377995
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.35s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
pickup_ntaname: 0.00B
dropoff_borocode: 0.00B
cab_type: 0.00B
snow_depth: 0.00B
pickup_puma: 0.00B
dropoff_ntacode: 0.00B
vendor_id: 0.00B
pickup_ctlabel: 0.00B
store_and_fwd_flag: 0.00B
payment_type: 0.00B
extra: 0.00B
pickup_nyct2010_gid: 0.00B
pickup: 0.00B
total_amount: 0.00B
improvement_surcharge: 0.00B
dropoff_nyct2010_gid: 0.00B
pickup_boroname: 0.00B
dropoff_latitude: 0.00B
pickup_borocode: 0.00B
pickup_cdeligibil: 0.00B
dropoff_boroname: 0.00B
pickup_ntacode: 0.00B
pickup_longitude: 0.00B
average_wind_speed: 0.00B
dropoff_ct2010: 0.00B
dropoff_puma: 0.00B
trip_type: 0.00B
tip_amount: 0.00B
trip_id: 0.00B
pickup_datetime: 0.00B
ehail_fee: 0.00B
pickup_boroct2010: 0.00B
dropoff_boroct2010: 0.00B
snowfall: 0.00B
rate_code_id: 0.00B
mta_tax: 0.00B
dropoff_ctlabel: 0.00B
dropoff_longitude: 0.00B
precipitation: 0.00B
trip_distance: 0.00B
passenger_count: 0.00B
max_temperature: 0.00B
tolls_amount: 0.00B
dropoff_cdeligibil: 0.00B
dropoff_datetime: 0.00B
fare_amount: 0.00B
pickup_latitude: 0.00B
dropoff_ntaname: 0.00B
min_temperature: 0.00B
pickup_ct2010: 0.00B
dropoff: 0.00B
locustdb> SELECT passenger_count, count(0), sum(total_amount) FROM trips;
Scanned 1.46 billion rows in 6.84s (0.21 billion rows/s)!
passenger_count | count_0 | sum_1
----------------+------------+---------------
0 | 4107443 | 4509842206
1 | 1028187597 | 1425303818216
2 | 210577025 | 304718353489
3 | 62345389 | 87788492212
4 | 29994987 | 42514355874
5 | 95184622 | 130181002616
6 | 34380600 | 51063741303
7 | 2595 | 8693795
8 | 2525 | 8272055
9 | 1367 | 7196690
10 | 17 | 75470
13 | 1 | 3150
15 | 2 | 2410
17 | 1 | 3995
19 | 1 | 500
25 | 1 | 760
33 | 2 | 1719
34 | 1 | 1680
36 | 1 | 6155
37 | 1 | 1420
38 | 1 | 730
47 | 1 | 900
49 | 26 | 8440
53 | 1 | 730
58 | 2 | 5188
61 | 1 | 3134
65 | 3 | 7010
66 | 1 | 1930
69 | 1 | 580
70 | 1 | 1070
84 | 1 | 4384
97 | 1 | 990
113 | 1 | 1330
125 | 1 | 1660
129 | 7 | 6150
133 | 1 | 1030
134 | 1 | 5514
137 | 1 | 5964
141 | 1 | 1894
155 | 1 | 9024
158 | 1 | 1444
160 | 1 | 1534
163 | 1 | 1554
164 | 1 | 6214
165 | 1 | 1214
177 | 1 | 1700
192 | 1 | 953
193 | 1 | 750
208 | 1508 | 1062534
211 | 1 | 700
213 | 4 | 1000
223 | 1 | 950
225 | 1 | 1600
247 | 1 | 1944
249 | 1 | 950
250 | 3 | 3770
254 | 1 | 650
255 | 10 | 17990
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.35s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
dropoff_ntaname: 0.00B
extra: 0.00B
pickup_ntacode: 0.00B
dropoff_puma: 0.00B
vendor_id: 0.00B
passenger_count: 0.00B
trip_distance: 0.00B
store_and_fwd_flag: 0.00B
fare_amount: 0.00B
tip_amount: 0.00B
pickup_puma: 0.00B
pickup_longitude: 0.00B
dropoff_longitude: 0.00B
mta_tax: 0.00B
improvement_surcharge: 0.00B
pickup_nyct2010_gid: 0.00B
ehail_fee: 0.00B
pickup_datetime: 0.00B
cab_type: 0.00B
dropoff_ct2010: 0.00B
pickup_boroct2010: 0.00B
dropoff_ctlabel: 0.00B
dropoff_boroct2010: 0.00B
pickup_boroname: 0.00B
average_wind_speed: 0.00B
dropoff_ntacode: 0.00B
tolls_amount: 0.00B
precipitation: 0.00B
trip_id: 0.00B
pickup_cdeligibil: 0.00B
max_temperature: 0.00B
trip_type: 0.00B
pickup_ctlabel: 0.00B
total_amount: 0.00B
pickup_borocode: 0.00B
rate_code_id: 0.00B
dropoff_cdeligibil: 0.00B
snowfall: 0.00B
dropoff_borocode: 0.00B
min_temperature: 0.00B
dropoff: 0.00B
payment_type: 0.00B
pickup_ct2010: 0.00B
pickup_ntaname: 0.00B
dropoff_latitude: 0.00B
pickup_latitude: 0.00B
pickup: 0.00B
snow_depth: 0.00B
dropoff_datetime: 0.00B
dropoff_nyct2010_gid: 0.00B
dropoff_boroname: 0.00B
locustdb> SELECT passenger_count, count(0), sum(total_amount) FROM trips;
Scanned 1.46 billion rows in 6.82s (0.21 billion rows/s)!
passenger_count | count_0 | sum_1
----------------+------------+---------------
0 | 4107443 | 4509842206
1 | 1028187597 | 1425303818216
2 | 210577025 | 304718353489
3 | 62345389 | 87788492212
4 | 29994987 | 42514355874
5 | 95184622 | 130181002616
6 | 34380600 | 51063741303
7 | 2595 | 8693795
8 | 2525 | 8272055
9 | 1367 | 7196690
10 | 17 | 75470
13 | 1 | 3150
15 | 2 | 2410
17 | 1 | 3995
19 | 1 | 500
25 | 1 | 760
33 | 2 | 1719
34 | 1 | 1680
36 | 1 | 6155
37 | 1 | 1420
38 | 1 | 730
47 | 1 | 900
49 | 26 | 8440
53 | 1 | 730
58 | 2 | 5188
61 | 1 | 3134
65 | 3 | 7010
66 | 1 | 1930
69 | 1 | 580
70 | 1 | 1070
84 | 1 | 4384
97 | 1 | 990
113 | 1 | 1330
125 | 1 | 1660
129 | 7 | 6150
133 | 1 | 1030
134 | 1 | 5514
137 | 1 | 5964
141 | 1 | 1894
155 | 1 | 9024
158 | 1 | 1444
160 | 1 | 1534
163 | 1 | 1554
164 | 1 | 6214
165 | 1 | 1214
177 | 1 | 1700
192 | 1 | 953
193 | 1 | 750
208 | 1508 | 1062534
211 | 1 | 700
213 | 4 | 1000
223 | 1 | 950
225 | 1 | 1600
247 | 1 | 1944
249 | 1 | 950
250 | 3 | 3770
254 | 1 | 650
255 | 10 | 17990
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.35s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
min_temperature: 0.00B
dropoff_ctlabel: 0.00B
trip_type: 0.00B
pickup_boroct2010: 0.00B
pickup_ntaname: 0.00B
ehail_fee: 0.00B
dropoff_ntacode: 0.00B
extra: 0.00B
pickup_puma: 0.00B
pickup_ntacode: 0.00B
vendor_id: 0.00B
dropoff_cdeligibil: 0.00B
dropoff_nyct2010_gid: 0.00B
trip_id: 0.00B
mta_tax: 0.00B
dropoff_longitude: 0.00B
pickup: 0.00B
dropoff: 0.00B
pickup_ct2010: 0.00B
fare_amount: 0.00B
dropoff_puma: 0.00B
average_wind_speed: 0.00B
pickup_cdeligibil: 0.00B
passenger_count: 0.00B
max_temperature: 0.00B
total_amount: 0.00B
tolls_amount: 0.00B
trip_distance: 0.00B
snow_depth: 0.00B
improvement_surcharge: 0.00B
dropoff_ct2010: 0.00B
cab_type: 0.00B
pickup_boroname: 0.00B
pickup_nyct2010_gid: 0.00B
store_and_fwd_flag: 0.00B
dropoff_boroct2010: 0.00B
tip_amount: 0.00B
rate_code_id: 0.00B
dropoff_latitude: 0.00B
dropoff_borocode: 0.00B
precipitation: 0.00B
dropoff_boroname: 0.00B
payment_type: 0.00B
pickup_longitude: 0.00B
pickup_ctlabel: 0.00B
dropoff_ntaname: 0.00B
pickup_datetime: 0.00B
dropoff_datetime: 0.00B
snowfall: 0.00B
pickup_borocode: 0.00B
pickup_latitude: 0.00B
locustdb> SELECT passenger_count, count(0), sum(total_amount) FROM trips;
Scanned 1.46 billion rows in 6.83s (0.21 billion rows/s)!
passenger_count | count_0 | sum_1
----------------+------------+---------------
0 | 4107443 | 4509842206
1 | 1028187597 | 1425303818216
2 | 210577025 | 304718353489
3 | 62345389 | 87788492212
4 | 29994987 | 42514355874
5 | 95184622 | 130181002616
6 | 34380600 | 51063741303
7 | 2595 | 8693795
8 | 2525 | 8272055
9 | 1367 | 7196690
10 | 17 | 75470
13 | 1 | 3150
15 | 2 | 2410
17 | 1 | 3995
19 | 1 | 500
25 | 1 | 760
33 | 2 | 1719
34 | 1 | 1680
36 | 1 | 6155
37 | 1 | 1420
38 | 1 | 730
47 | 1 | 900
49 | 26 | 8440
53 | 1 | 730
58 | 2 | 5188
61 | 1 | 3134
65 | 3 | 7010
66 | 1 | 1930
69 | 1 | 580
70 | 1 | 1070
84 | 1 | 4384
97 | 1 | 990
113 | 1 | 1330
125 | 1 | 1660
129 | 7 | 6150
133 | 1 | 1030
134 | 1 | 5514
137 | 1 | 5964
141 | 1 | 1894
155 | 1 | 9024
158 | 1 | 1444
160 | 1 | 1534
163 | 1 | 1554
164 | 1 | 6214
165 | 1 | 1214
177 | 1 | 1700
192 | 1 | 953
193 | 1 | 750
208 | 1508 | 1062534
211 | 1 | 700
213 | 4 | 1000
223 | 1 | 950
225 | 1 | 1600
247 | 1 | 1944
249 | 1 | 950
250 | 3 | 3770
254 | 1 | 650
255 | 10 | 17990
locustdb> :memtree
trips 3.67GiB 1464785771 rows
├─ average_wind_speed nonresident
├─ cab_type nonresident
├─ dropoff nonresident
├─ dropoff_borocode nonresident
├─ dropoff_boroct2010 nonresident
├─ dropoff_boroname nonresident
├─ dropoff_cdeligibil nonresident
├─ dropoff_ct2010 nonresident
├─ dropoff_ctlabel nonresident
├─ dropoff_datetime nonresident
├─ dropoff_latitude nonresident
├─ dropoff_longitude nonresident
├─ dropoff_ntacode nonresident
├─ dropoff_ntaname nonresident
├─ dropoff_nyct2010_gid nonresident
├─ dropoff_puma nonresident
├─ ehail_fee nonresident
├─ extra nonresident
├─ fare_amount nonresident
├─ improvement_surcharge nonresident
├─ max_temperature nonresident
├─ min_temperature nonresident
├─ mta_tax nonresident
├─ passenger_count fully resident 0.73GiB 20% 0.53B/row
├─ payment_type nonresident
├─ pickup nonresident
├─ pickup_borocode nonresident
├─ pickup_boroct2010 nonresident
├─ pickup_boroname nonresident
├─ pickup_cdeligibil nonresident
├─ pickup_ct2010 nonresident
├─ pickup_ctlabel nonresident
├─ pickup_datetime nonresident
├─ pickup_latitude nonresident
├─ pickup_longitude nonresident
├─ pickup_ntacode nonresident
├─ pickup_ntaname nonresident
├─ pickup_nyct2010_gid nonresident
├─ pickup_puma nonresident
├─ precipitation nonresident
├─ rate_code_id nonresident
├─ snow_depth nonresident
├─ snowfall nonresident
├─ store_and_fwd_flag nonresident
├─ tip_amount nonresident
├─ tolls_amount nonresident
├─ total_amount fully resident 2.9GiB 80% 2.2B/row
├─ trip_distance nonresident
├─ trip_id nonresident
├─ trip_type nonresident
└─ vendor_id nonresident
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
dropoff_longitude: 0.00B
pickup_nyct2010_gid: 0.00B
dropoff_ctlabel: 0.00B
pickup: 0.00B
snowfall: 0.00B
tolls_amount: 0.00B
cab_type: 0.00B
dropoff: 0.00B
max_temperature: 0.00B
passenger_count: 0.00B
ehail_fee: 0.00B
trip_id: 0.00B
dropoff_latitude: 0.00B
mta_tax: 0.00B
snow_depth: 0.00B
trip_type: 0.00B
min_temperature: 0.00B
pickup_cdeligibil: 0.00B
payment_type: 0.00B
pickup_boroname: 0.00B
dropoff_cdeligibil: 0.00B
pickup_ntacode: 0.00B
tip_amount: 0.00B
dropoff_borocode: 0.00B
pickup_longitude: 0.00B
pickup_ntaname: 0.00B
dropoff_ntaname: 0.00B
pickup_puma: 0.00B
pickup_latitude: 0.00B
vendor_id: 0.00B
dropoff_ntacode: 0.00B
fare_amount: 0.00B
improvement_surcharge: 0.00B
pickup_ctlabel: 0.00B
pickup_boroct2010: 0.00B
pickup_borocode: 0.00B
dropoff_boroname: 0.00B
extra: 0.00B
dropoff_boroct2010: 0.00B
total_amount: 0.00B
store_and_fwd_flag: 0.00B
pickup_ct2010: 0.00B
average_wind_speed: 0.00B
dropoff_datetime: 0.00B
dropoff_nyct2010_gid: 0.00B
rate_code_id: 0.00B
precipitation: 0.00B
dropoff_ct2010: 0.00B
dropoff_puma: 0.00B
pickup_datetime: 0.00B
trip_distance: 0.00B
locustdb> SELECT passenger_count, to_year(pickup_datetime), count(0) FROM trips;
Scanned 1.46 billion rows in 11.2s (0.13 billion rows/s)!
passenger_count | col_0 | count_0
----------------+-------+-----------
0 | 2009 | 1126
0 | 2010 | 35487
0 | 2011 | 2547937
0 | 2012 | 1274858
0 | 2013 | 5476
0 | 2014 | 9848
0 | 2015 | 46953
0 | 2016 | 17829
0 | 2017 | 167929
1 | 2001 | 3
1 | 2002 | 1
1 | 2003 | 4
1 | 2008 | 33
1 | 2009 | 116050070
1 | 2010 | 98991938
1 | 2011 | 119969450
1 | 2012 | 123955717
1 | 2013 | 122918578
1 | 2014 | 129404151
1 | 2015 | 119150088
1 | 2016 | 106826192
1 | 2017 | 90921221
1 | 2018 | 150
1 | 2041 | 1
2 | 2009 | 27534617
2 | 2010 | 23053740
2 | 2011 | 27132349
2 | 2012 | 24900623
2 | 2013 | 23610698
2 | 2014 | 24221283
2 | 2015 | 22349143
2 | 2016 | 20290051
2 | 2017 | 17484514
2 | 2018 | 7
3 | 2008 | 1
3 | 2009 | 7860551
3 | 2010 | 6700468
3 | 2011 | 8015005
3 | 2012 | 7716364
3 | 2013 | 7343603
3 | 2014 | 7348686
3 | 2015 | 6580430
3 | 2016 | 5810400
3 | 2017 | 4969879
3 | 2018 | 1
3 | 2053 | 1
4 | 2009 | 3829136
4 | 2010 | 3282494
4 | 2011 | 3819715
4 | 2012 | 3752549
4 | 2013 | 3593459
4 | 2014 | 3512831
4 | 2015 | 3105718
4 | 2016 | 2770237
4 | 2017 | 2328848
5 | 2008 | 7
5 | 2009 | 14885624
5 | 2010 | 12311339
5 | 2011 | 13876061
5 | 2012 | 12063339
5 | 2013 | 10139454
5 | 2014 | 10056870
5 | 2015 | 8691048
5 | 2016 | 7337348
5 | 2017 | 5823513
5 | 2018 | 19
6 | 2003 | 1
6 | 2008 | 1
6 | 2009 | 734341
6 | 2010 | 595332
6 | 2011 | 1536489
6 | 2012 | 4880708
6 | 2013 | 6779135
6 | 2014 | 6396494
6 | 2015 | 5421796
6 | 2016 | 4496670
6 | 2017 | 3539630
6 | 2018 | 3
7 | 2009 | 7
7 | 2010 | 4
7 | 2011 | 8
7 | 2012 | 17
7 | 2013 | 65
7 | 2014 | 504
7 | 2015 | 599
7 | 2016 | 765
7 | 2017 | 626
8 | 2009 | 2
8 | 2010 | 1
8 | 2011 | 2
8 | 2012 | 13
8 | 2013 | 43
8 | 2014 | 464
8 | 2015 | 668
8 | 2016 | 707
8 | 2017 | 625
9 | 2009 | 1
9 | 2010 | 3
9 | 2011 | 8
9 | 2012 | 25
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.35s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
pickup_puma: 0.00B
dropoff_nyct2010_gid: 0.00B
passenger_count: 0.00B
max_temperature: 0.00B
dropoff_boroname: 0.00B
pickup_longitude: 0.00B
tip_amount: 0.00B
cab_type: 0.00B
pickup: 0.00B
pickup_nyct2010_gid: 0.00B
snow_depth: 0.00B
trip_distance: 0.00B
fare_amount: 0.00B
improvement_surcharge: 0.00B
dropoff_ctlabel: 0.00B
dropoff_ntacode: 0.00B
dropoff_cdeligibil: 0.00B
trip_id: 0.00B
payment_type: 0.00B
store_and_fwd_flag: 0.00B
min_temperature: 0.00B
trip_type: 0.00B
tolls_amount: 0.00B
dropoff_latitude: 0.00B
pickup_ct2010: 0.00B
pickup_datetime: 0.00B
mta_tax: 0.00B
total_amount: 0.00B
pickup_cdeligibil: 0.00B
dropoff_ntaname: 0.00B
dropoff: 0.00B
average_wind_speed: 0.00B
pickup_boroname: 0.00B
pickup_ntaname: 0.00B
extra: 0.00B
rate_code_id: 0.00B
dropoff_datetime: 0.00B
vendor_id: 0.00B
dropoff_longitude: 0.00B
precipitation: 0.00B
dropoff_ct2010: 0.00B
pickup_latitude: 0.00B
pickup_ctlabel: 0.00B
dropoff_boroct2010: 0.00B
pickup_borocode: 0.00B
ehail_fee: 0.00B
snowfall: 0.00B
pickup_ntacode: 0.00B
dropoff_puma: 0.00B
dropoff_borocode: 0.00B
pickup_boroct2010: 0.00B
locustdb> SELECT passenger_count, to_year(pickup_datetime), count(0) FROM trips;
Scanned 1.46 billion rows in 11.2s (0.13 billion rows/s)!
passenger_count | col_0 | count_0
----------------+-------+-----------
0 | 2009 | 1126
0 | 2010 | 35487
0 | 2011 | 2547937
0 | 2012 | 1274858
0 | 2013 | 5476
0 | 2014 | 9848
0 | 2015 | 46953
0 | 2016 | 17829
0 | 2017 | 167929
1 | 2001 | 3
1 | 2002 | 1
1 | 2003 | 4
1 | 2008 | 33
1 | 2009 | 116050070
1 | 2010 | 98991938
1 | 2011 | 119969450
1 | 2012 | 123955717
1 | 2013 | 122918578
1 | 2014 | 129404151
1 | 2015 | 119150088
1 | 2016 | 106826192
1 | 2017 | 90921221
1 | 2018 | 150
1 | 2041 | 1
2 | 2009 | 27534617
2 | 2010 | 23053740
2 | 2011 | 27132349
2 | 2012 | 24900623
2 | 2013 | 23610698
2 | 2014 | 24221283
2 | 2015 | 22349143
2 | 2016 | 20290051
2 | 2017 | 17484514
2 | 2018 | 7
3 | 2008 | 1
3 | 2009 | 7860551
3 | 2010 | 6700468
3 | 2011 | 8015005
3 | 2012 | 7716364
3 | 2013 | 7343603
3 | 2014 | 7348686
3 | 2015 | 6580430
3 | 2016 | 5810400
3 | 2017 | 4969879
3 | 2018 | 1
3 | 2053 | 1
4 | 2009 | 3829136
4 | 2010 | 3282494
4 | 2011 | 3819715
4 | 2012 | 3752549
4 | 2013 | 3593459
4 | 2014 | 3512831
4 | 2015 | 3105718
4 | 2016 | 2770237
4 | 2017 | 2328848
5 | 2008 | 7
5 | 2009 | 14885624
5 | 2010 | 12311339
5 | 2011 | 13876061
5 | 2012 | 12063339
5 | 2013 | 10139454
5 | 2014 | 10056870
5 | 2015 | 8691048
5 | 2016 | 7337348
5 | 2017 | 5823513
5 | 2018 | 19
6 | 2003 | 1
6 | 2008 | 1
6 | 2009 | 734341
6 | 2010 | 595332
6 | 2011 | 1536489
6 | 2012 | 4880708
6 | 2013 | 6779135
6 | 2014 | 6396494
6 | 2015 | 5421796
6 | 2016 | 4496670
6 | 2017 | 3539630
6 | 2018 | 3
7 | 2009 | 7
7 | 2010 | 4
7 | 2011 | 8
7 | 2012 | 17
7 | 2013 | 65
7 | 2014 | 504
7 | 2015 | 599
7 | 2016 | 765
7 | 2017 | 626
8 | 2009 | 2
8 | 2010 | 1
8 | 2011 | 2
8 | 2012 | 13
8 | 2013 | 43
8 | 2014 | 464
8 | 2015 | 668
8 | 2016 | 707
8 | 2017 | 625
9 | 2009 | 1
9 | 2010 | 3
9 | 2011 | 8
9 | 2012 | 25
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
tip_amount: 0.00B
pickup_borocode: 0.00B
pickup_boroct2010: 0.00B
ehail_fee: 0.00B
snow_depth: 0.00B
pickup_nyct2010_gid: 0.00B
dropoff_ntaname: 0.00B
pickup_ntaname: 0.00B
max_temperature: 0.00B
dropoff_cdeligibil: 0.00B
dropoff_ntacode: 0.00B
dropoff_boroct2010: 0.00B
dropoff_datetime: 0.00B
pickup_longitude: 0.00B
fare_amount: 0.00B
trip_distance: 0.00B
extra: 0.00B
dropoff_puma: 0.00B
total_amount: 0.00B
pickup_boroname: 0.00B
pickup_ctlabel: 0.00B
pickup_puma: 0.00B
mta_tax: 0.00B
store_and_fwd_flag: 0.00B
rate_code_id: 0.00B
dropoff_longitude: 0.00B
dropoff_borocode: 0.00B
pickup_ct2010: 0.00B
average_wind_speed: 0.00B
pickup_datetime: 0.00B
pickup: 0.00B
min_temperature: 0.00B
dropoff_nyct2010_gid: 0.00B
dropoff: 0.00B
dropoff_ctlabel: 0.00B
tolls_amount: 0.00B
pickup_latitude: 0.00B
trip_type: 0.00B
payment_type: 0.00B
pickup_ntacode: 0.00B
dropoff_ct2010: 0.00B
improvement_surcharge: 0.00B
dropoff_latitude: 0.00B
vendor_id: 0.00B
snowfall: 0.00B
pickup_cdeligibil: 0.00B
cab_type: 0.00B
precipitation: 0.00B
trip_id: 0.00B
dropoff_boroname: 0.00B
passenger_count: 0.00B
locustdb> SELECT passenger_count, to_year(pickup_datetime), count(0) FROM trips;
Scanned 1.46 billion rows in 11.2s (0.13 billion rows/s)!
passenger_count | col_0 | count_0
----------------+-------+-----------
0 | 2009 | 1126
0 | 2010 | 35487
0 | 2011 | 2547937
0 | 2012 | 1274858
0 | 2013 | 5476
0 | 2014 | 9848
0 | 2015 | 46953
0 | 2016 | 17829
0 | 2017 | 167929
1 | 2001 | 3
1 | 2002 | 1
1 | 2003 | 4
1 | 2008 | 33
1 | 2009 | 116050070
1 | 2010 | 98991938
1 | 2011 | 119969450
1 | 2012 | 123955717
1 | 2013 | 122918578
1 | 2014 | 129404151
1 | 2015 | 119150088
1 | 2016 | 106826192
1 | 2017 | 90921221
1 | 2018 | 150
1 | 2041 | 1
2 | 2009 | 27534617
2 | 2010 | 23053740
2 | 2011 | 27132349
2 | 2012 | 24900623
2 | 2013 | 23610698
2 | 2014 | 24221283
2 | 2015 | 22349143
2 | 2016 | 20290051
2 | 2017 | 17484514
2 | 2018 | 7
3 | 2008 | 1
3 | 2009 | 7860551
3 | 2010 | 6700468
3 | 2011 | 8015005
3 | 2012 | 7716364
3 | 2013 | 7343603
3 | 2014 | 7348686
3 | 2015 | 6580430
3 | 2016 | 5810400
3 | 2017 | 4969879
3 | 2018 | 1
3 | 2053 | 1
4 | 2009 | 3829136
4 | 2010 | 3282494
4 | 2011 | 3819715
4 | 2012 | 3752549
4 | 2013 | 3593459
4 | 2014 | 3512831
4 | 2015 | 3105718
4 | 2016 | 2770237
4 | 2017 | 2328848
5 | 2008 | 7
5 | 2009 | 14885624
5 | 2010 | 12311339
5 | 2011 | 13876061
5 | 2012 | 12063339
5 | 2013 | 10139454
5 | 2014 | 10056870
5 | 2015 | 8691048
5 | 2016 | 7337348
5 | 2017 | 5823513
5 | 2018 | 19
6 | 2003 | 1
6 | 2008 | 1
6 | 2009 | 734341
6 | 2010 | 595332
6 | 2011 | 1536489
6 | 2012 | 4880708
6 | 2013 | 6779135
6 | 2014 | 6396494
6 | 2015 | 5421796
6 | 2016 | 4496670
6 | 2017 | 3539630
6 | 2018 | 3
7 | 2009 | 7
7 | 2010 | 4
7 | 2011 | 8
7 | 2012 | 17
7 | 2013 | 65
7 | 2014 | 504
7 | 2015 | 599
7 | 2016 | 765
7 | 2017 | 626
8 | 2009 | 2
8 | 2010 | 1
8 | 2011 | 2
8 | 2012 | 13
8 | 2013 | 43
8 | 2014 | 464
8 | 2015 | 668
8 | 2016 | 707
8 | 2017 | 625
9 | 2009 | 1
9 | 2010 | 3
9 | 2011 | 8
9 | 2012 | 25
locustdb> :memtree
trips 5.85GiB 1464785771 rows
├─ average_wind_speed nonresident
├─ cab_type nonresident
├─ dropoff nonresident
├─ dropoff_borocode nonresident
├─ dropoff_boroct2010 nonresident
├─ dropoff_boroname nonresident
├─ dropoff_cdeligibil nonresident
├─ dropoff_ct2010 nonresident
├─ dropoff_ctlabel nonresident
├─ dropoff_datetime nonresident
├─ dropoff_latitude nonresident
├─ dropoff_longitude nonresident
├─ dropoff_ntacode nonresident
├─ dropoff_ntaname nonresident
├─ dropoff_nyct2010_gid nonresident
├─ dropoff_puma nonresident
├─ ehail_fee nonresident
├─ extra nonresident
├─ fare_amount nonresident
├─ improvement_surcharge nonresident
├─ max_temperature nonresident
├─ min_temperature nonresident
├─ mta_tax nonresident
├─ passenger_count fully resident 0.73GiB 12% 0.53B/row
├─ payment_type nonresident
├─ pickup nonresident
├─ pickup_borocode nonresident
├─ pickup_boroct2010 nonresident
├─ pickup_boroname nonresident
├─ pickup_cdeligibil nonresident
├─ pickup_ct2010 nonresident
├─ pickup_ctlabel nonresident
├─ pickup_datetime fully resident 5.1GiB 88% 3.8B/row
├─ pickup_latitude nonresident
├─ pickup_longitude nonresident
├─ pickup_ntacode nonresident
├─ pickup_ntaname nonresident
├─ pickup_nyct2010_gid nonresident
├─ pickup_puma nonresident
├─ precipitation nonresident
├─ rate_code_id nonresident
├─ snow_depth nonresident
├─ snowfall nonresident
├─ store_and_fwd_flag nonresident
├─ tip_amount nonresident
├─ tolls_amount nonresident
├─ total_amount nonresident
├─ trip_distance nonresident
├─ trip_id nonresident
├─ trip_type nonresident
└─ vendor_id nonresident
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
pickup_boroct2010: 0.00B
dropoff_ct2010: 0.00B
trip_type: 0.00B
tip_amount: 0.00B
dropoff_ntacode: 0.00B
pickup_boroname: 0.00B
pickup_borocode: 0.00B
pickup_cdeligibil: 0.00B
dropoff_puma: 0.00B
pickup_nyct2010_gid: 0.00B
trip_distance: 0.00B
dropoff_ctlabel: 0.00B
snow_depth: 0.00B
average_wind_speed: 0.00B
rate_code_id: 0.00B
pickup: 0.00B
dropoff_ntaname: 0.00B
snowfall: 0.00B
cab_type: 0.00B
dropoff_longitude: 0.00B
dropoff_cdeligibil: 0.00B
dropoff: 0.00B
pickup_latitude: 0.00B
pickup_datetime: 0.00B
pickup_ct2010: 0.00B
pickup_puma: 0.00B
fare_amount: 0.00B
vendor_id: 0.00B
dropoff_borocode: 0.00B
dropoff_nyct2010_gid: 0.00B
dropoff_boroname: 0.00B
ehail_fee: 0.00B
mta_tax: 0.00B
extra: 0.00B
trip_id: 0.00B
payment_type: 0.00B
dropoff_latitude: 0.00B
store_and_fwd_flag: 0.00B
min_temperature: 0.00B
passenger_count: 0.00B
precipitation: 0.00B
pickup_ctlabel: 0.00B
total_amount: 0.00B
improvement_surcharge: 0.00B
max_temperature: 0.00B
dropoff_boroct2010: 0.00B
dropoff_datetime: 0.00B
pickup_longitude: 0.00B
pickup_ntaname: 0.00B
tolls_amount: 0.00B
pickup_ntacode: 0.00B
locustdb> SELECT passenger_count, to_year(pickup_datetime), trip_distance / 1000, count(0) FROM trips;
Scanned 1.46 billion rows in 17.8s (82 million rows/s)!
passenger_count | col_0 | col_1 | count_0
----------------+-------+-------+---------
0 | 2009 | 0 | 570
0 | 2009 | 1 | 267
0 | 2009 | 2 | 115
0 | 2009 | 3 | 52
0 | 2009 | 4 | 28
0 | 2009 | 5 | 28
0 | 2009 | 6 | 13
0 | 2009 | 7 | 11
0 | 2009 | 8 | 8
0 | 2009 | 9 | 5
0 | 2009 | 10 | 4
0 | 2009 | 11 | 6
0 | 2009 | 12 | 2
0 | 2009 | 13 | 2
0 | 2009 | 14 | 2
0 | 2009 | 17 | 3
0 | 2009 | 18 | 2
0 | 2009 | 19 | 2
0 | 2009 | 20 | 1
0 | 2009 | 21 | 2
0 | 2009 | 22 | 2
0 | 2009 | 31 | 1
0 | 2010 | 0 | 16834
0 | 2010 | 1 | 10264
0 | 2010 | 2 | 3858
0 | 2010 | 3 | 1797
0 | 2010 | 4 | 831
0 | 2010 | 5 | 643
0 | 2010 | 6 | 418
0 | 2010 | 7 | 207
0 | 2010 | 8 | 108
0 | 2010 | 9 | 100
0 | 2010 | 10 | 109
0 | 2010 | 11 | 118
0 | 2010 | 12 | 58
0 | 2010 | 13 | 30
0 | 2010 | 14 | 20
0 | 2010 | 15 | 17
0 | 2010 | 16 | 13
0 | 2010 | 17 | 10
0 | 2010 | 18 | 3
0 | 2010 | 19 | 8
0 | 2010 | 20 | 8
0 | 2010 | 21 | 5
0 | 2010 | 22 | 2
0 | 2010 | 23 | 3
0 | 2010 | 24 | 2
0 | 2010 | 26 | 3
0 | 2010 | 27 | 1
0 | 2010 | 28 | 1
0 | 2010 | 29 | 1
0 | 2010 | 32 | 1
0 | 2010 | 33 | 1
0 | 2010 | 36 | 2
0 | 2010 | 37 | 2
0 | 2010 | 38 | 1
0 | 2010 | 43 | 1
0 | 2010 | 47 | 1
0 | 2010 | 53 | 1
0 | 2010 | 54 | 1
0 | 2010 | 70 | 1
0 | 2010 | 76 | 1
0 | 2010 | 77 | 1
0 | 2010 | 86 | 1
0 | 2011 | 0 | 679047
0 | 2011 | 1 | 884997
0 | 2011 | 2 | 430866
0 | 2011 | 3 | 211773
0 | 2011 | 4 | 112552
0 | 2011 | 5 | 69700
0 | 2011 | 6 | 43498
0 | 2011 | 7 | 26703
0 | 2011 | 8 | 16962
0 | 2011 | 9 | 13569
0 | 2011 | 10 | 10974
0 | 2011 | 11 | 8656
0 | 2011 | 12 | 6488
0 | 2011 | 13 | 4422
0 | 2011 | 14 | 3534
0 | 2011 | 15 | 3470
0 | 2011 | 16 | 4541
0 | 2011 | 17 | 4617
0 | 2011 | 18 | 3805
0 | 2011 | 19 | 2574
0 | 2011 | 20 | 1710
0 | 2011 | 21 | 965
0 | 2011 | 22 | 581
0 | 2011 | 23 | 340
0 | 2011 | 24 | 214
0 | 2011 | 25 | 189
0 | 2011 | 26 | 153
0 | 2011 | 27 | 138
0 | 2011 | 28 | 114
0 | 2011 | 29 | 63
0 | 2011 | 30 | 73
0 | 2011 | 31 | 58
0 | 2011 | 32 | 72
0 | 2011 | 33 | 39
0 | 2011 | 34 | 46
0 | 2011 | 35 | 39
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
pickup_ntaname: 0.00B
passenger_count: 0.00B
dropoff_nyct2010_gid: 0.00B
pickup: 0.00B
dropoff_ctlabel: 0.00B
pickup_boroname: 0.00B
dropoff_cdeligibil: 0.00B
trip_id: 0.00B
dropoff_boroname: 0.00B
mta_tax: 0.00B
trip_distance: 0.00B
improvement_surcharge: 0.00B
dropoff_longitude: 0.00B
pickup_datetime: 0.00B
pickup_boroct2010: 0.00B
tolls_amount: 0.00B
fare_amount: 0.00B
dropoff_ct2010: 0.00B
snow_depth: 0.00B
snowfall: 0.00B
extra: 0.00B
pickup_ct2010: 0.00B
dropoff_datetime: 0.00B
trip_type: 0.00B
store_and_fwd_flag: 0.00B
dropoff_puma: 0.00B
dropoff_borocode: 0.00B
precipitation: 0.00B
vendor_id: 0.00B
pickup_ntacode: 0.00B
ehail_fee: 0.00B
pickup_cdeligibil: 0.00B
dropoff_ntaname: 0.00B
pickup_longitude: 0.00B
dropoff_ntacode: 0.00B
tip_amount: 0.00B
pickup_latitude: 0.00B
dropoff_boroct2010: 0.00B
pickup_nyct2010_gid: 0.00B
dropoff: 0.00B
dropoff_latitude: 0.00B
pickup_ctlabel: 0.00B
average_wind_speed: 0.00B
rate_code_id: 0.00B
min_temperature: 0.00B
pickup_borocode: 0.00B
pickup_puma: 0.00B
cab_type: 0.00B
max_temperature: 0.00B
total_amount: 0.00B
payment_type: 0.00B
locustdb> SELECT passenger_count, to_year(pickup_datetime), trip_distance / 1000, count(0) FROM trips;
Scanned 1.46 billion rows in 17.8s (82 million rows/s)!
passenger_count | col_0 | col_1 | count_0
----------------+-------+-------+---------
0 | 2009 | 0 | 570
0 | 2009 | 1 | 267
0 | 2009 | 2 | 115
0 | 2009 | 3 | 52
0 | 2009 | 4 | 28
0 | 2009 | 5 | 28
0 | 2009 | 6 | 13
0 | 2009 | 7 | 11
0 | 2009 | 8 | 8
0 | 2009 | 9 | 5
0 | 2009 | 10 | 4
0 | 2009 | 11 | 6
0 | 2009 | 12 | 2
0 | 2009 | 13 | 2
0 | 2009 | 14 | 2
0 | 2009 | 17 | 3
0 | 2009 | 18 | 2
0 | 2009 | 19 | 2
0 | 2009 | 20 | 1
0 | 2009 | 21 | 2
0 | 2009 | 22 | 2
0 | 2009 | 31 | 1
0 | 2010 | 0 | 16834
0 | 2010 | 1 | 10264
0 | 2010 | 2 | 3858
0 | 2010 | 3 | 1797
0 | 2010 | 4 | 831
0 | 2010 | 5 | 643
0 | 2010 | 6 | 418
0 | 2010 | 7 | 207
0 | 2010 | 8 | 108
0 | 2010 | 9 | 100
0 | 2010 | 10 | 109
0 | 2010 | 11 | 118
0 | 2010 | 12 | 58
0 | 2010 | 13 | 30
0 | 2010 | 14 | 20
0 | 2010 | 15 | 17
0 | 2010 | 16 | 13
0 | 2010 | 17 | 10
0 | 2010 | 18 | 3
0 | 2010 | 19 | 8
0 | 2010 | 20 | 8
0 | 2010 | 21 | 5
0 | 2010 | 22 | 2
0 | 2010 | 23 | 3
0 | 2010 | 24 | 2
0 | 2010 | 26 | 3
0 | 2010 | 27 | 1
0 | 2010 | 28 | 1
0 | 2010 | 29 | 1
0 | 2010 | 32 | 1
0 | 2010 | 33 | 1
0 | 2010 | 36 | 2
0 | 2010 | 37 | 2
0 | 2010 | 38 | 1
0 | 2010 | 43 | 1
0 | 2010 | 47 | 1
0 | 2010 | 53 | 1
0 | 2010 | 54 | 1
0 | 2010 | 70 | 1
0 | 2010 | 76 | 1
0 | 2010 | 77 | 1
0 | 2010 | 86 | 1
0 | 2011 | 0 | 679047
0 | 2011 | 1 | 884997
0 | 2011 | 2 | 430866
0 | 2011 | 3 | 211773
0 | 2011 | 4 | 112552
0 | 2011 | 5 | 69700
0 | 2011 | 6 | 43498
0 | 2011 | 7 | 26703
0 | 2011 | 8 | 16962
0 | 2011 | 9 | 13569
0 | 2011 | 10 | 10974
0 | 2011 | 11 | 8656
0 | 2011 | 12 | 6488
0 | 2011 | 13 | 4422
0 | 2011 | 14 | 3534
0 | 2011 | 15 | 3470
0 | 2011 | 16 | 4541
0 | 2011 | 17 | 4617
0 | 2011 | 18 | 3805
0 | 2011 | 19 | 2574
0 | 2011 | 20 | 1710
0 | 2011 | 21 | 965
0 | 2011 | 22 | 581
0 | 2011 | 23 | 340
0 | 2011 | 24 | 214
0 | 2011 | 25 | 189
0 | 2011 | 26 | 153
0 | 2011 | 27 | 138
0 | 2011 | 28 | 114
0 | 2011 | 29 | 63
0 | 2011 | 30 | 73
0 | 2011 | 31 | 58
0 | 2011 | 32 | 72
0 | 2011 | 33 | 39
0 | 2011 | 34 | 46
0 | 2011 | 35 | 39
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.37s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
pickup_ntaname: 0.00B
dropoff_boroname: 0.00B
max_temperature: 0.00B
pickup_boroname: 0.00B
dropoff_borocode: 0.00B
pickup_ntacode: 0.00B
pickup_longitude: 0.00B
dropoff_latitude: 0.00B
store_and_fwd_flag: 0.00B
pickup_latitude: 0.00B
rate_code_id: 0.00B
passenger_count: 0.00B
dropoff_nyct2010_gid: 0.00B
dropoff_ct2010: 0.00B
dropoff_cdeligibil: 0.00B
dropoff: 0.00B
trip_distance: 0.00B
snowfall: 0.00B
fare_amount: 0.00B
pickup_datetime: 0.00B
dropoff_longitude: 0.00B
vendor_id: 0.00B
dropoff_puma: 0.00B
pickup_nyct2010_gid: 0.00B
precipitation: 0.00B
ehail_fee: 0.00B
dropoff_datetime: 0.00B
trip_type: 0.00B
pickup_boroct2010: 0.00B
pickup_puma: 0.00B
min_temperature: 0.00B
tolls_amount: 0.00B
payment_type: 0.00B
snow_depth: 0.00B
mta_tax: 0.00B
pickup_cdeligibil: 0.00B
pickup_ct2010: 0.00B
dropoff_boroct2010: 0.00B
dropoff_ntacode: 0.00B
dropoff_ctlabel: 0.00B
total_amount: 0.00B
pickup: 0.00B
pickup_borocode: 0.00B
trip_id: 0.00B
extra: 0.00B
improvement_surcharge: 0.00B
tip_amount: 0.00B
dropoff_ntaname: 0.00B
average_wind_speed: 0.00B
cab_type: 0.00B
pickup_ctlabel: 0.00B
locustdb> SELECT passenger_count, to_year(pickup_datetime), trip_distance / 1000, count(0) FROM trips;
Scanned 1.46 billion rows in 17.8s (82 million rows/s)!
passenger_count | col_0 | col_1 | count_0
----------------+-------+-------+---------
0 | 2009 | 0 | 570
0 | 2009 | 1 | 267
0 | 2009 | 2 | 115
0 | 2009 | 3 | 52
0 | 2009 | 4 | 28
0 | 2009 | 5 | 28
0 | 2009 | 6 | 13
0 | 2009 | 7 | 11
0 | 2009 | 8 | 8
0 | 2009 | 9 | 5
0 | 2009 | 10 | 4
0 | 2009 | 11 | 6
0 | 2009 | 12 | 2
0 | 2009 | 13 | 2
0 | 2009 | 14 | 2
0 | 2009 | 17 | 3
0 | 2009 | 18 | 2
0 | 2009 | 19 | 2
0 | 2009 | 20 | 1
0 | 2009 | 21 | 2
0 | 2009 | 22 | 2
0 | 2009 | 31 | 1
0 | 2010 | 0 | 16834
0 | 2010 | 1 | 10264
0 | 2010 | 2 | 3858
0 | 2010 | 3 | 1797
0 | 2010 | 4 | 831
0 | 2010 | 5 | 643
0 | 2010 | 6 | 418
0 | 2010 | 7 | 207
0 | 2010 | 8 | 108
0 | 2010 | 9 | 100
0 | 2010 | 10 | 109
0 | 2010 | 11 | 118
0 | 2010 | 12 | 58
0 | 2010 | 13 | 30
0 | 2010 | 14 | 20
0 | 2010 | 15 | 17
0 | 2010 | 16 | 13
0 | 2010 | 17 | 10
0 | 2010 | 18 | 3
0 | 2010 | 19 | 8
0 | 2010 | 20 | 8
0 | 2010 | 21 | 5
0 | 2010 | 22 | 2
0 | 2010 | 23 | 3
0 | 2010 | 24 | 2
0 | 2010 | 26 | 3
0 | 2010 | 27 | 1
0 | 2010 | 28 | 1
0 | 2010 | 29 | 1
0 | 2010 | 32 | 1
0 | 2010 | 33 | 1
0 | 2010 | 36 | 2
0 | 2010 | 37 | 2
0 | 2010 | 38 | 1
0 | 2010 | 43 | 1
0 | 2010 | 47 | 1
0 | 2010 | 53 | 1
0 | 2010 | 54 | 1
0 | 2010 | 70 | 1
0 | 2010 | 76 | 1
0 | 2010 | 77 | 1
0 | 2010 | 86 | 1
0 | 2011 | 0 | 679047
0 | 2011 | 1 | 884997
0 | 2011 | 2 | 430866
0 | 2011 | 3 | 211773
0 | 2011 | 4 | 112552
0 | 2011 | 5 | 69700
0 | 2011 | 6 | 43498
0 | 2011 | 7 | 26703
0 | 2011 | 8 | 16962
0 | 2011 | 9 | 13569
0 | 2011 | 10 | 10974
0 | 2011 | 11 | 8656
0 | 2011 | 12 | 6488
0 | 2011 | 13 | 4422
0 | 2011 | 14 | 3534
0 | 2011 | 15 | 3470
0 | 2011 | 16 | 4541
0 | 2011 | 17 | 4617
0 | 2011 | 18 | 3805
0 | 2011 | 19 | 2574
0 | 2011 | 20 | 1710
0 | 2011 | 21 | 965
0 | 2011 | 22 | 581
0 | 2011 | 23 | 340
0 | 2011 | 24 | 214
0 | 2011 | 25 | 189
0 | 2011 | 26 | 153
0 | 2011 | 27 | 138
0 | 2011 | 28 | 114
0 | 2011 | 29 | 63
0 | 2011 | 30 | 73
0 | 2011 | 31 | 58
0 | 2011 | 32 | 72
0 | 2011 | 33 | 39
0 | 2011 | 34 | 46
0 | 2011 | 35 | 39
locustdb> :memtree
trips 9.26GiB 1464785771 rows
├─ average_wind_speed nonresident
├─ cab_type nonresident
├─ dropoff nonresident
├─ dropoff_borocode nonresident
├─ dropoff_boroct2010 nonresident
├─ dropoff_boroname nonresident
├─ dropoff_cdeligibil nonresident
├─ dropoff_ct2010 nonresident
├─ dropoff_ctlabel nonresident
├─ dropoff_datetime nonresident
├─ dropoff_latitude nonresident
├─ dropoff_longitude nonresident
├─ dropoff_ntacode nonresident
├─ dropoff_ntaname nonresident
├─ dropoff_nyct2010_gid nonresident
├─ dropoff_puma nonresident
├─ ehail_fee nonresident
├─ extra nonresident
├─ fare_amount nonresident
├─ improvement_surcharge nonresident
├─ max_temperature nonresident
├─ min_temperature nonresident
├─ mta_tax nonresident
├─ passenger_count fully resident 0.73GiB 7.8% 0.53B/row
├─ payment_type nonresident
├─ pickup nonresident
├─ pickup_borocode nonresident
├─ pickup_boroct2010 nonresident
├─ pickup_boroname nonresident
├─ pickup_cdeligibil nonresident
├─ pickup_ct2010 nonresident
├─ pickup_ctlabel nonresident
├─ pickup_datetime fully resident 5.1GiB 55% 3.8B/row
├─ pickup_latitude nonresident
├─ pickup_longitude nonresident
├─ pickup_ntacode nonresident
├─ pickup_ntaname nonresident
├─ pickup_nyct2010_gid nonresident
├─ pickup_puma nonresident
├─ precipitation nonresident
├─ rate_code_id nonresident
├─ snow_depth nonresident
├─ snowfall nonresident
├─ store_and_fwd_flag nonresident
├─ tip_amount nonresident
├─ tolls_amount nonresident
├─ total_amount nonresident
├─ trip_distance fully resident 3.4GiB 37% 2.5B/row
├─ trip_id nonresident
├─ trip_type nonresident
└─ vendor_id nonresident
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.37s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
dropoff_latitude: 0.00B
dropoff: 0.00B
pickup_ntaname: 0.00B
extra: 0.00B
snowfall: 0.00B
trip_distance: 0.00B
pickup: 0.00B
pickup_datetime: 0.00B
pickup_puma: 0.00B
dropoff_ct2010: 0.00B
pickup_ct2010: 0.00B
precipitation: 0.00B
vendor_id: 0.00B
pickup_ctlabel: 0.00B
average_wind_speed: 0.00B
dropoff_boroname: 0.00B
cab_type: 0.00B
dropoff_ntacode: 0.00B
snow_depth: 0.00B
payment_type: 0.00B
fare_amount: 0.00B
pickup_boroct2010: 0.00B
total_amount: 0.00B
store_and_fwd_flag: 0.00B
tip_amount: 0.00B
max_temperature: 0.00B
pickup_longitude: 0.00B
tolls_amount: 0.00B
dropoff_cdeligibil: 0.00B
dropoff_nyct2010_gid: 0.00B
dropoff_boroct2010: 0.00B
trip_type: 0.00B
dropoff_borocode: 0.00B
dropoff_datetime: 0.00B
pickup_ntacode: 0.00B
dropoff_ctlabel: 0.00B
improvement_surcharge: 0.00B
ehail_fee: 0.00B
pickup_cdeligibil: 0.00B
pickup_boroname: 0.00B
dropoff_ntaname: 0.00B
pickup_nyct2010_gid: 0.00B
trip_id: 0.00B
passenger_count: 0.00B
dropoff_puma: 0.00B
rate_code_id: 0.00B
min_temperature: 0.00B
mta_tax: 0.00B
pickup_latitude: 0.00B
pickup_borocode: 0.00B
dropoff_longitude: 0.00B
locustdb> SELECT trip_id FROM trips WHERE (passenger_count = 5) AND (vendor_id = "CMT") AND (total_amount < 500)
AND (store_and_fwd_flag = "1") LIMIT 100;
Scanned 1.46 billion rows in 9.34s (0.16 billion rows/s)!
trip_id
----------
237961482
189096372
208684890
208926304
97545508
195132906
244996620
299069329
139768745
257680549
173799760
174182597
263677077
250557151
303171377
303600048
85701057
176394104
298515659
232337094
80845388
164837200
310082405
156105032
181868551
136472715
136644714
138080491
138089339
259106549
161072070
243127062
195778208
195827261
103186939
106048938
245281086
247696432
237368106
237523596
237547051
193071605
91804982
142031633
210730145
147380973
148254686
102569125
201367926
207059403
166444891
202610735
203089568
106309860
84895468
144660880
233908085
182124046
174687424
175280990
228237325
112520068
130612502
122800644
241317351
113478507
154865525
155088848
163106718
152363765
203844240
190937722
190953060
191366388
135130551
194235323
121789437
121970701
122012280
104895642
177367538
212714740
255415775
261961948
83385404
208072881
137556872
293214555
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
pickup_nyct2010_gid: 0.00B
dropoff_ctlabel: 0.00B
trip_type: 0.00B
dropoff_datetime: 0.00B
extra: 0.00B
snow_depth: 0.00B
pickup_cdeligibil: 0.00B
snowfall: 0.00B
pickup_boroct2010: 0.00B
tip_amount: 0.00B
dropoff_boroname: 0.00B
store_and_fwd_flag: 0.00B
dropoff_ct2010: 0.00B
dropoff_longitude: 0.00B
pickup_datetime: 0.00B
dropoff_nyct2010_gid: 0.00B
mta_tax: 0.00B
passenger_count: 0.00B
pickup_puma: 0.00B
precipitation: 0.00B
pickup_longitude: 0.00B
dropoff_cdeligibil: 0.00B
pickup_ct2010: 0.00B
cab_type: 0.00B
dropoff_ntacode: 0.00B
dropoff_ntaname: 0.00B
improvement_surcharge: 0.00B
dropoff: 0.00B
vendor_id: 0.00B
rate_code_id: 0.00B
pickup_borocode: 0.00B
pickup_boroname: 0.00B
average_wind_speed: 0.00B
dropoff_latitude: 0.00B
dropoff_borocode: 0.00B
tolls_amount: 0.00B
trip_distance: 0.00B
trip_id: 0.00B
ehail_fee: 0.00B
pickup_latitude: 0.00B
dropoff_puma: 0.00B
pickup_ctlabel: 0.00B
pickup_ntacode: 0.00B
fare_amount: 0.00B
max_temperature: 0.00B
min_temperature: 0.00B
pickup_ntaname: 0.00B
dropoff_boroct2010: 0.00B
pickup: 0.00B
total_amount: 0.00B
payment_type: 0.00B
locustdb> SELECT trip_id FROM trips WHERE (passenger_count = 5) AND (vendor_id = "CMT") AND (total_amount < 500)
AND (store_and_fwd_flag = "1") LIMIT 100;
Scanned 1.46 billion rows in 9.39s (0.16 billion rows/s)!
trip_id
----------
139768745
135130551
174687424
175280990
208072881
299069329
166444891
181868551
161072070
244996620
298515659
113478507
138080491
138089339
152363765
154865525
155088848
83385404
257680549
106048938
259106549
245281086
103186939
156105032
121789437
121970701
122012280
84895468
237961482
97545508
173799760
174182597
210730145
182124046
190937722
190953060
191366388
237368106
237523596
237547051
137556872
177367538
164837200
207059403
195778208
195827261
195132906
147380973
148254686
303171377
303600048
104895642
193071605
163106718
208684890
208926304
85701057
102569125
136472715
136644714
80845388
203844240
112520068
201367926
91804982
263677077
212714740
189096372
261961948
194235323
106309860
122800644
144660880
310082405
228237325
243127062
202610735
203089568
255415775
247696432
233908085
241317351
293214555
232337094
130612502
176394104
250557151
142031633
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
pickup_cdeligibil: 0.00B
pickup_latitude: 0.00B
pickup_ntaname: 0.00B
dropoff_puma: 0.00B
pickup_longitude: 0.00B
trip_type: 0.00B
dropoff_ctlabel: 0.00B
pickup_nyct2010_gid: 0.00B
store_and_fwd_flag: 0.00B
dropoff_ct2010: 0.00B
mta_tax: 0.00B
trip_distance: 0.00B
passenger_count: 0.00B
max_temperature: 0.00B
dropoff_borocode: 0.00B
cab_type: 0.00B
extra: 0.00B
dropoff_ntacode: 0.00B
pickup_ctlabel: 0.00B
average_wind_speed: 0.00B
pickup_datetime: 0.00B
dropoff_datetime: 0.00B
tolls_amount: 0.00B
tip_amount: 0.00B
pickup_puma: 0.00B
dropoff_boroname: 0.00B
rate_code_id: 0.00B
pickup_boroname: 0.00B
min_temperature: 0.00B
trip_id: 0.00B
pickup_boroct2010: 0.00B
payment_type: 0.00B
dropoff_nyct2010_gid: 0.00B
dropoff_longitude: 0.00B
dropoff_ntaname: 0.00B
ehail_fee: 0.00B
fare_amount: 0.00B
precipitation: 0.00B
pickup: 0.00B
pickup_ct2010: 0.00B
dropoff_cdeligibil: 0.00B
pickup_borocode: 0.00B
pickup_ntacode: 0.00B
snowfall: 0.00B
improvement_surcharge: 0.00B
total_amount: 0.00B
vendor_id: 0.00B
dropoff_boroct2010: 0.00B
dropoff: 0.00B
dropoff_latitude: 0.00B
snow_depth: 0.00B
locustdb> SELECT trip_id FROM trips WHERE (passenger_count = 5) AND (vendor_id = "CMT") AND (total_amount < 500)
AND (store_and_fwd_flag = "1") LIMIT 100;
Scanned 1.46 billion rows in 9.36s (0.16 billion rows/s)!
trip_id
----------
84895468
121789437
121970701
122012280
173799760
174182597
130612502
154865525
155088848
261961948
194235323
138080491
138089339
177367538
85701057
113478507
263677077
80845388
203844240
139768745
195132906
182124046
112520068
257680549
122800644
210730145
83385404
303171377
303600048
163106718
97545508
152363765
243127062
208072881
166444891
245281086
156105032
195778208
195827261
106048938
259106549
193071605
250557151
176394104
190937722
190953060
191366388
137556872
104895642
135130551
233908085
293214555
208684890
208926304
106309860
244996620
241317351
228237325
161072070
202610735
203089568
298515659
174687424
175280990
201367926
102569125
237368106
237523596
237547051
237961482
103186939
147380973
148254686
299069329
255415775
91804982
310082405
136472715
136644714
142031633
144660880
207059403
212714740
164837200
232337094
181868551
247696432
189096372
locustdb> :memtree
trips 5.02GiB 1464785771 rows
├─ average_wind_speed nonresident
├─ cab_type nonresident
├─ dropoff nonresident
├─ dropoff_borocode nonresident
├─ dropoff_boroct2010 nonresident
├─ dropoff_boroname nonresident
├─ dropoff_cdeligibil nonresident
├─ dropoff_ct2010 nonresident
├─ dropoff_ctlabel nonresident
├─ dropoff_datetime nonresident
├─ dropoff_latitude nonresident
├─ dropoff_longitude nonresident
├─ dropoff_ntacode nonresident
├─ dropoff_ntaname nonresident
├─ dropoff_nyct2010_gid nonresident
├─ dropoff_puma nonresident
├─ ehail_fee nonresident
├─ extra nonresident
├─ fare_amount nonresident
├─ improvement_surcharge nonresident
├─ max_temperature nonresident
├─ min_temperature nonresident
├─ mta_tax nonresident
├─ passenger_count fully resident 0.73GiB 14% 0.53B/row
├─ payment_type nonresident
├─ pickup nonresident
├─ pickup_borocode nonresident
├─ pickup_boroct2010 nonresident
├─ pickup_boroname nonresident
├─ pickup_cdeligibil nonresident
├─ pickup_ct2010 nonresident
├─ pickup_ctlabel nonresident
├─ pickup_datetime nonresident
├─ pickup_latitude nonresident
├─ pickup_longitude nonresident
├─ pickup_ntacode nonresident
├─ pickup_ntaname nonresident
├─ pickup_nyct2010_gid nonresident
├─ pickup_puma nonresident
├─ precipitation nonresident
├─ rate_code_id nonresident
├─ snow_depth nonresident
├─ snowfall nonresident
├─ store_and_fwd_flag fully resident 0.58GiB 11% 0.42B/row
├─ tip_amount nonresident
├─ tolls_amount nonresident
├─ total_amount fully resident 2.9GiB 59% 2.2B/row
├─ trip_distance nonresident
├─ trip_id fully resident 62MiB 1.2% 44mB/row
├─ trip_type nonresident
└─ vendor_id fully resident 0.71GiB 14% 0.52B/row
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
cab_type: 0.00B
max_temperature: 0.00B
dropoff_ct2010: 0.00B
pickup_borocode: 0.00B
pickup_cdeligibil: 0.00B
extra: 0.00B
snow_depth: 0.00B
pickup_boroct2010: 0.00B
precipitation: 0.00B
fare_amount: 0.00B
pickup_boroname: 0.00B
pickup: 0.00B
dropoff_borocode: 0.00B
trip_id: 0.00B
average_wind_speed: 0.00B
store_and_fwd_flag: 0.00B
min_temperature: 0.00B
mta_tax: 0.00B
pickup_ct2010: 0.00B
total_amount: 0.00B
dropoff_ntacode: 0.00B
passenger_count: 0.00B
snowfall: 0.00B
pickup_ntaname: 0.00B
pickup_datetime: 0.00B
dropoff_datetime: 0.00B
pickup_ntacode: 0.00B
dropoff_latitude: 0.00B
dropoff_cdeligibil: 0.00B
dropoff: 0.00B
dropoff_boroct2010: 0.00B
rate_code_id: 0.00B
dropoff_nyct2010_gid: 0.00B
pickup_puma: 0.00B
tip_amount: 0.00B
dropoff_longitude: 0.00B
dropoff_puma: 0.00B
dropoff_ntaname: 0.00B
payment_type: 0.00B
pickup_ctlabel: 0.00B
pickup_nyct2010_gid: 0.00B
improvement_surcharge: 0.00B
tolls_amount: 0.00B
trip_distance: 0.00B
dropoff_boroname: 0.00B
dropoff_ctlabel: 0.00B
vendor_id: 0.00B
pickup_latitude: 0.00B
ehail_fee: 0.00B
pickup_longitude: 0.00B
trip_type: 0.00B
locustdb> SELECT passenger_count, trip_distance, total_amount FROM trips ORDER BY total_amount DESC LIMIT 100;
Scanned 1.46 billion rows in 13.1s (0.11 billion rows/s)!
passenger_count | trip_distance | total_amount
----------------+---------------+--------------
1 | 5320 | 395061160
1 | 180 | 98997039
2 | 9600 | 86161105
1 | 700 | 86160324
1 | 0 | 82603997
1 | 0 | 82603997
1 | 11620 | 68590810
1 | 1160 | 65026285
2 | 0 | 64756743
1 | 2200 | 63112198
1 | 800 | 63046312
1 | 0 | 62903378
1 | 0 | 62903378
3 | 0 | 62590160
5 | 1170 | 54846335
5 | 1800 | 54143256
1 | 0 | 53858000
1 | 0 | 53848268
2 | 0 | 50530481
1 | 1300 | 50332633
1 | 0 | 42956225
1 | 4300 | 42781990
1 | 0 | 41026766
1 | 0 | 40848698
1 | 1178600 | 40411466
1 | 0 | 39847996
1 | 0 | 39846670
2 | 0 | 39846620
1 | 0 | 39322232
1 | 1000 | 39191358
1 | 0 | 37918374
3 | 2200 | 36177202
1 | 0 | 33886334
2 | 1100 | 33554521
1 | 700 | 33541449
1 | 0 | 33340289
1 | 258800 | 32549603
1 | 0 | 32274382
1 | 45600 | 30704518
2 | 0 | 26908936
1 | 1200 | 24861943
1 | 0 | 21831709
1 | 0 | 21052379
1 | 0 | 20214955
1 | 1000 | 20054751
1 | 6800 | 19864438
2 | 2700 | 18744576
1 | 2100 | 18744426
2 | 1800 | 18744226
1 | 1800 | 18613254
2 | 3700 | 17186358
1 | 600 | 17186246
1 | 2800 | 17185699
1 | 0 | 16868839
1 | 0 | 16868789
1 | 0 | 16606012
2 | 12500 | 15900116
1 | 298914200 | 15483214
1 | 151694000 | 15329622
1 | 1714056400 | 15327117
6 | 300 | 15243838
1 | 4400 | 15099774
1 | 0 | 13690599
3 | 166331600 | 13313120
2 | 166086300 | 13312500
1 | 158600 | 13306368
2 | 131091400 | 12636658
1 | 1000 | 12451927
1 | 0 | 11605382
1 | 900 | 11127165
2 | 500 | 10778818
3 | 0 | 10744509
3 | 0 | 10108073
1 | 7500 | 9397810
2 | 1500 | 9396416
1 | 198623013600 | 9395569
1 | 536862699 | 9395270
1 | 390288800 | 9150945
1 | 13100 | 8897786
2 | 1180 | 8806493
6 | 1190 | 8633055
2 | 522133000 | 8224197
1 | 3140000000 | 7579732
1 | 1530 | 6995607
1 | 288703900 | 6155303
1 | 2200 | 5119200
1 | 100 | 3736032
1 | 357894700 | 3153439
1 | 0 | 2361976
1 | 5240 | 2254411
1 | 234410600 | 2245986
1 | 400 | 2097476
1 | 71732700 | 2004407
1 | 0 | 1401030
2 | 733400 | 1289125
4 | 0 | 1275534
3 | 0 | 1248541
1 | 1524200 | 1018021
5 | 0 | 1000099
1 | 1514910 | 1000099
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.35s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
pickup_latitude: 0.00B
trip_type: 0.00B
mta_tax: 0.00B
passenger_count: 0.00B
pickup_cdeligibil: 0.00B
precipitation: 0.00B
total_amount: 0.00B
pickup_ct2010: 0.00B
rate_code_id: 0.00B
average_wind_speed: 0.00B
pickup_ntaname: 0.00B
snow_depth: 0.00B
pickup_boroname: 0.00B
dropoff_ctlabel: 0.00B
fare_amount: 0.00B
min_temperature: 0.00B
payment_type: 0.00B
pickup_ctlabel: 0.00B
trip_id: 0.00B
trip_distance: 0.00B
snowfall: 0.00B
dropoff_ct2010: 0.00B
vendor_id: 0.00B
pickup: 0.00B
store_and_fwd_flag: 0.00B
dropoff_latitude: 0.00B
dropoff_longitude: 0.00B
dropoff: 0.00B
dropoff_ntaname: 0.00B
dropoff_borocode: 0.00B
cab_type: 0.00B
dropoff_puma: 0.00B
pickup_datetime: 0.00B
max_temperature: 0.00B
improvement_surcharge: 0.00B
dropoff_ntacode: 0.00B
dropoff_nyct2010_gid: 0.00B
pickup_puma: 0.00B
pickup_ntacode: 0.00B
tip_amount: 0.00B
dropoff_boroname: 0.00B
pickup_nyct2010_gid: 0.00B
dropoff_datetime: 0.00B
dropoff_boroct2010: 0.00B
ehail_fee: 0.00B
pickup_borocode: 0.00B
dropoff_cdeligibil: 0.00B
extra: 0.00B
pickup_boroct2010: 0.00B
pickup_longitude: 0.00B
tolls_amount: 0.00B
locustdb> SELECT passenger_count, trip_distance, total_amount FROM trips ORDER BY total_amount DESC LIMIT 100;
Scanned 1.46 billion rows in 13.1s (0.11 billion rows/s)!
passenger_count | trip_distance | total_amount
----------------+---------------+--------------
1 | 5320 | 395061160
1 | 180 | 98997039
2 | 9600 | 86161105
1 | 700 | 86160324
1 | 0 | 82603997
1 | 0 | 82603997
1 | 11620 | 68590810
1 | 1160 | 65026285
2 | 0 | 64756743
1 | 2200 | 63112198
1 | 800 | 63046312
1 | 0 | 62903378
1 | 0 | 62903378
3 | 0 | 62590160
5 | 1170 | 54846335
5 | 1800 | 54143256
1 | 0 | 53858000
1 | 0 | 53848268
2 | 0 | 50530481
1 | 1300 | 50332633
1 | 0 | 42956225
1 | 4300 | 42781990
1 | 0 | 41026766
1 | 0 | 40848698
1 | 1178600 | 40411466
1 | 0 | 39847996
1 | 0 | 39846670
2 | 0 | 39846620
1 | 0 | 39322232
1 | 1000 | 39191358
1 | 0 | 37918374
3 | 2200 | 36177202
1 | 0 | 33886334
2 | 1100 | 33554521
1 | 700 | 33541449
1 | 0 | 33340289
1 | 258800 | 32549603
1 | 0 | 32274382
1 | 45600 | 30704518
2 | 0 | 26908936
1 | 1200 | 24861943
1 | 0 | 21831709
1 | 0 | 21052379
1 | 0 | 20214955
1 | 1000 | 20054751
1 | 6800 | 19864438
2 | 2700 | 18744576
1 | 2100 | 18744426
2 | 1800 | 18744226
1 | 1800 | 18613254
2 | 3700 | 17186358
1 | 600 | 17186246
1 | 2800 | 17185699
1 | 0 | 16868839
1 | 0 | 16868789
1 | 0 | 16606012
2 | 12500 | 15900116
1 | 298914200 | 15483214
1 | 151694000 | 15329622
1 | 1714056400 | 15327117
6 | 300 | 15243838
1 | 4400 | 15099774
1 | 0 | 13690599
3 | 166331600 | 13313120
2 | 166086300 | 13312500
1 | 158600 | 13306368
2 | 131091400 | 12636658
1 | 1000 | 12451927
1 | 0 | 11605382
1 | 900 | 11127165
2 | 500 | 10778818
3 | 0 | 10744509
3 | 0 | 10108073
1 | 7500 | 9397810
2 | 1500 | 9396416
1 | 198623013600 | 9395569
1 | 536862699 | 9395270
1 | 390288800 | 9150945
1 | 13100 | 8897786
2 | 1180 | 8806493
6 | 1190 | 8633055
2 | 522133000 | 8224197
1 | 3140000000 | 7579732
1 | 1530 | 6995607
1 | 288703900 | 6155303
1 | 2200 | 5119200
1 | 100 | 3736032
1 | 357894700 | 3153439
1 | 0 | 2361976
1 | 5240 | 2254411
1 | 234410600 | 2245986
1 | 400 | 2097476
1 | 71732700 | 2004407
1 | 0 | 1401030
2 | 733400 | 1289125
4 | 0 | 1275534
3 | 0 | 1248541
1 | 1524200 | 1018021
1 | 1514910 | 1000099
5 | 0 | 1000099
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.35s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
snowfall: 0.00B
pickup_ctlabel: 0.00B
pickup_ntaname: 0.00B
min_temperature: 0.00B
pickup_ct2010: 0.00B
dropoff_ntaname: 0.00B
max_temperature: 0.00B
tolls_amount: 0.00B
dropoff_boroname: 0.00B
payment_type: 0.00B
dropoff_nyct2010_gid: 0.00B
snow_depth: 0.00B
tip_amount: 0.00B
total_amount: 0.00B
passenger_count: 0.00B
vendor_id: 0.00B
pickup_boroname: 0.00B
dropoff_longitude: 0.00B
pickup_boroct2010: 0.00B
pickup_cdeligibil: 0.00B
dropoff_boroct2010: 0.00B
pickup_longitude: 0.00B
pickup_ntacode: 0.00B
improvement_surcharge: 0.00B
dropoff_cdeligibil: 0.00B
trip_id: 0.00B
pickup_datetime: 0.00B
ehail_fee: 0.00B
extra: 0.00B
dropoff_ct2010: 0.00B
cab_type: 0.00B
rate_code_id: 0.00B
dropoff_ntacode: 0.00B
pickup_borocode: 0.00B
pickup_latitude: 0.00B
trip_type: 0.00B
dropoff_borocode: 0.00B
fare_amount: 0.00B
pickup_puma: 0.00B
trip_distance: 0.00B
store_and_fwd_flag: 0.00B
mta_tax: 0.00B
dropoff_latitude: 0.00B
precipitation: 0.00B
dropoff_datetime: 0.00B
average_wind_speed: 0.00B
dropoff: 0.00B
pickup: 0.00B
pickup_nyct2010_gid: 0.00B
dropoff_ctlabel: 0.00B
dropoff_puma: 0.00B
locustdb> SELECT passenger_count, trip_distance, total_amount FROM trips ORDER BY total_amount DESC LIMIT 100;
Scanned 1.46 billion rows in 13.1s (0.11 billion rows/s)!
passenger_count | trip_distance | total_amount
----------------+---------------+--------------
1 | 5320 | 395061160
1 | 180 | 98997039
2 | 9600 | 86161105
1 | 700 | 86160324
1 | 0 | 82603997
1 | 0 | 82603997
1 | 11620 | 68590810
1 | 1160 | 65026285
2 | 0 | 64756743
1 | 2200 | 63112198
1 | 800 | 63046312
1 | 0 | 62903378
1 | 0 | 62903378
3 | 0 | 62590160
5 | 1170 | 54846335
5 | 1800 | 54143256
1 | 0 | 53858000
1 | 0 | 53848268
2 | 0 | 50530481
1 | 1300 | 50332633
1 | 0 | 42956225
1 | 4300 | 42781990
1 | 0 | 41026766
1 | 0 | 40848698
1 | 1178600 | 40411466
1 | 0 | 39847996
1 | 0 | 39846670
2 | 0 | 39846620
1 | 0 | 39322232
1 | 1000 | 39191358
1 | 0 | 37918374
3 | 2200 | 36177202
1 | 0 | 33886334
2 | 1100 | 33554521
1 | 700 | 33541449
1 | 0 | 33340289
1 | 258800 | 32549603
1 | 0 | 32274382
1 | 45600 | 30704518
2 | 0 | 26908936
1 | 1200 | 24861943
1 | 0 | 21831709
1 | 0 | 21052379
1 | 0 | 20214955
1 | 1000 | 20054751
1 | 6800 | 19864438
2 | 2700 | 18744576
1 | 2100 | 18744426
2 | 1800 | 18744226
1 | 1800 | 18613254
2 | 3700 | 17186358
1 | 600 | 17186246
1 | 2800 | 17185699
1 | 0 | 16868839
1 | 0 | 16868789
1 | 0 | 16606012
2 | 12500 | 15900116
1 | 298914200 | 15483214
1 | 151694000 | 15329622
1 | 1714056400 | 15327117
6 | 300 | 15243838
1 | 4400 | 15099774
1 | 0 | 13690599
3 | 166331600 | 13313120
2 | 166086300 | 13312500
1 | 158600 | 13306368
2 | 131091400 | 12636658
1 | 1000 | 12451927
1 | 0 | 11605382
1 | 900 | 11127165
2 | 500 | 10778818
3 | 0 | 10744509
3 | 0 | 10108073
1 | 7500 | 9397810
2 | 1500 | 9396416
1 | 198623013600 | 9395569
1 | 536862699 | 9395270
1 | 390288800 | 9150945
1 | 13100 | 8897786
2 | 1180 | 8806493
6 | 1190 | 8633055
2 | 522133000 | 8224197
1 | 3140000000 | 7579732
1 | 1530 | 6995607
1 | 288703900 | 6155303
1 | 2200 | 5119200
1 | 100 | 3736032
1 | 357894700 | 3153439
1 | 0 | 2361976
1 | 5240 | 2254411
1 | 234410600 | 2245986
1 | 400 | 2097476
1 | 71732700 | 2004407
1 | 0 | 1401030
2 | 733400 | 1289125
4 | 0 | 1275534
3 | 0 | 1248541
1 | 1524200 | 1018021
1 | 1514910 | 1000099
5 | 0 | 1000099
locustdb> :memtree
trips 7.09GiB 1464785771 rows
├─ average_wind_speed nonresident
├─ cab_type nonresident
├─ dropoff nonresident
├─ dropoff_borocode nonresident
├─ dropoff_boroct2010 nonresident
├─ dropoff_boroname nonresident
├─ dropoff_cdeligibil nonresident
├─ dropoff_ct2010 nonresident
├─ dropoff_ctlabel nonresident
├─ dropoff_datetime nonresident
├─ dropoff_latitude nonresident
├─ dropoff_longitude nonresident
├─ dropoff_ntacode nonresident
├─ dropoff_ntaname nonresident
├─ dropoff_nyct2010_gid nonresident
├─ dropoff_puma nonresident
├─ ehail_fee nonresident
├─ extra nonresident
├─ fare_amount nonresident
├─ improvement_surcharge nonresident
├─ max_temperature nonresident
├─ min_temperature nonresident
├─ mta_tax nonresident
├─ passenger_count fully resident 0.73GiB 10% 0.53B/row
├─ payment_type nonresident
├─ pickup nonresident
├─ pickup_borocode nonresident
├─ pickup_boroct2010 nonresident
├─ pickup_boroname nonresident
├─ pickup_cdeligibil nonresident
├─ pickup_ct2010 nonresident
├─ pickup_ctlabel nonresident
├─ pickup_datetime nonresident
├─ pickup_latitude nonresident
├─ pickup_longitude nonresident
├─ pickup_ntacode nonresident
├─ pickup_ntaname nonresident
├─ pickup_nyct2010_gid nonresident
├─ pickup_puma nonresident
├─ precipitation nonresident
├─ rate_code_id nonresident
├─ snow_depth nonresident
├─ snowfall nonresident
├─ store_and_fwd_flag nonresident
├─ tip_amount nonresident
├─ tolls_amount nonresident
├─ total_amount fully resident 2.9GiB 42% 2.2B/row
├─ trip_distance fully resident 3.4GiB 48% 2.5B/row
├─ trip_id nonresident
├─ trip_type nonresident
└─ vendor_id nonresident
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.36s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
dropoff_latitude: 0.00B
dropoff_puma: 0.00B
pickup_ntacode: 0.00B
pickup: 0.00B
pickup_ntaname: 0.00B
precipitation: 0.00B
average_wind_speed: 0.00B
dropoff_boroname: 0.00B
pickup_puma: 0.00B
pickup_longitude: 0.00B
pickup_boroct2010: 0.00B
total_amount: 0.00B
ehail_fee: 0.00B
dropoff_nyct2010_gid: 0.00B
tolls_amount: 0.00B
dropoff_ntaname: 0.00B
dropoff_cdeligibil: 0.00B
mta_tax: 0.00B
max_temperature: 0.00B
dropoff_borocode: 0.00B
vendor_id: 0.00B
trip_distance: 0.00B
dropoff: 0.00B
fare_amount: 0.00B
pickup_ctlabel: 0.00B
extra: 0.00B
snowfall: 0.00B
pickup_datetime: 0.00B
passenger_count: 0.00B
dropoff_ntacode: 0.00B
payment_type: 0.00B
pickup_boroname: 0.00B
pickup_latitude: 0.00B
store_and_fwd_flag: 0.00B
cab_type: 0.00B
dropoff_boroct2010: 0.00B
pickup_cdeligibil: 0.00B
dropoff_longitude: 0.00B
dropoff_ctlabel: 0.00B
min_temperature: 0.00B
trip_id: 0.00B
dropoff_ct2010: 0.00B
trip_type: 0.00B
pickup_nyct2010_gid: 0.00B
dropoff_datetime: 0.00B
improvement_surcharge: 0.00B
pickup_borocode: 0.00B
pickup_ct2010: 0.00B
tip_amount: 0.00B
snow_depth: 0.00B
rate_code_id: 0.00B
locustdb> SELECT pickup_puma, dropoff_puma, passenger_count, count(0) from trips;
Scanned 1.46 billion rows in 6.62s (0.22 billion rows/s)!
pickup_puma | dropoff_puma | passenger_count | count_0
------------+--------------+-----------------+-----------
0 | 0 | 0 | 239636
0 | 0 | 1 | 158504430
0 | 0 | 2 | 30589460
0 | 0 | 3 | 8775405
0 | 0 | 4 | 4182591
0 | 0 | 5 | 11051643
0 | 0 | 6 | 6082326
0 | 0 | 7 | 1168
0 | 0 | 8 | 1107
0 | 0 | 9 | 703
0 | 0 | 10 | 14
0 | 0 | 36 | 1
0 | 0 | 47 | 1
0 | 0 | 49 | 26
0 | 0 | 58 | 2
0 | 0 | 70 | 1
0 | 0 | 84 | 1
0 | 0 | 113 | 1
0 | 0 | 134 | 1
0 | 0 | 137 | 1
0 | 0 | 141 | 1
0 | 0 | 155 | 1
0 | 0 | 158 | 1
0 | 0 | 160 | 1
0 | 0 | 163 | 1
0 | 0 | 164 | 1
0 | 0 | 165 | 1
0 | 0 | 192 | 1
0 | 0 | 208 | 812
0 | 0 | 213 | 4
0 | 0 | 225 | 1
0 | 0 | 247 | 1
0 | 0 | 255 | 6
0 | 3701 | 0 | 13
0 | 3701 | 1 | 3593
0 | 3701 | 2 | 760
0 | 3701 | 3 | 209
0 | 3701 | 4 | 69
0 | 3701 | 5 | 417
0 | 3701 | 6 | 56
0 | 3701 | 7 | 2
0 | 3701 | 8 | 2
0 | 3701 | 9 | 1
0 | 3702 | 0 | 12
0 | 3702 | 1 | 2706
0 | 3702 | 2 | 511
0 | 3702 | 3 | 131
0 | 3702 | 4 | 56
0 | 3702 | 5 | 291
0 | 3702 | 6 | 58
0 | 3702 | 8 | 1
0 | 3703 | 0 | 5
0 | 3703 | 1 | 3673
0 | 3703 | 2 | 796
0 | 3703 | 3 | 237
0 | 3703 | 4 | 95
0 | 3703 | 5 | 614
0 | 3703 | 6 | 47
0 | 3703 | 8 | 1
0 | 3704 | 0 | 13
0 | 3704 | 1 | 1850
0 | 3704 | 2 | 390
0 | 3704 | 3 | 81
0 | 3704 | 4 | 43
0 | 3704 | 5 | 164
0 | 3704 | 6 | 23
0 | 3704 | 8 | 2
0 | 3705 | 0 | 13
0 | 3705 | 1 | 2913
0 | 3705 | 2 | 594
0 | 3705 | 3 | 186
0 | 3705 | 4 | 69
0 | 3705 | 5 | 317
0 | 3705 | 6 | 43
0 | 3705 | 8 | 1
0 | 3706 | 0 | 9
0 | 3706 | 1 | 2551
0 | 3706 | 2 | 469
0 | 3706 | 3 | 122
0 | 3706 | 4 | 39
0 | 3706 | 5 | 211
0 | 3706 | 6 | 28
0 | 3706 | 8 | 4
0 | 3706 | 9 | 1
0 | 3707 | 0 | 8
0 | 3707 | 1 | 2695
0 | 3707 | 2 | 523
0 | 3707 | 3 | 167
0 | 3707 | 4 | 67
0 | 3707 | 5 | 255
0 | 3707 | 6 | 34
0 | 3707 | 8 | 1
0 | 3708 | 0 | 16
0 | 3708 | 1 | 4899
0 | 3708 | 2 | 1045
0 | 3708 | 3 | 310
0 | 3708 | 4 | 128
0 | 3708 | 5 | 768
0 | 3708 | 6 | 50
0 | 3708 | 7 | 3
locustdb>
➜ LocustDB git:(master) sudo sh -c "sync; sudo echo 1 > /proc/sys/vm/drop_caches"
➜ LocustDB git:(master) RUSTFLAGS="-g -Ccodegen-units=1" CARGO_INCREMENTAL=0 cargo run --release --bin repl --features 'enable_lz4 enable_rocksdb' -- --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4
Finished release [optimized] target(s) in 0.29s
Running `target/release/repl --db-path /home/clemens/ldb-20 --mem-limit-tables=45 --mem-lz4`
# Table `trips` (1464785771 rows, 12.0KiB) #
fare_amount: 0.00B
average_wind_speed: 0.00B
dropoff_latitude: 0.00B
pickup_ctlabel: 0.00B
pickup_datetime: 0.00B
ehail_fee: 0.00B