Clemens Winter cswinter
cswinter
/ locustdb-lz4-vs-nolz4
Created
August 12, 2018 17:35
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ➜ ~ uname -a | |
| Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | |
| ➜ ~ lscpu | |
| Architecture: x86_64 | |
| CPU op-mode(s): 32-bit, 64-bit | |
| Byte Order: Little Endian | |
| CPU(s): 12 | |
| On-line CPU(s) list: 0-11 | |
| Thread(s) per core: 2 | |
| Core(s) per socket: 6 |
cswinter
/ locustdb-ingestion-hdd
Created
August 12, 2018 17:49
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ➜ ~ uname -a | |
| Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | |
| ➜ ~ sudo lshw -class disk -class storage | |
| [sudo] password for clemens: | |
| *-storage | |
| description: SATA controller | |
| product: ASM1062 Serial ATA Controller | |
| vendor: ASMedia Technology Inc. | |
| physical id: 0 | |
| bus info: pci@0000:06:00.0 |
cswinter
/ locustdb-ingestion-ssd
Created
August 12, 2018 17:52
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ➜ ~ uname -a | |
| Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | |
| ➜ ~ sudo lshw -class disk -class storage | |
| [sudo] password for clemens: | |
| *-storage | |
| description: SATA controller | |
| product: ASM1062 Serial ATA Controller | |
| vendor: ASMedia Technology Inc. | |
| physical id: 0 | |
| bus info: pci@0000:06:00.0 |
cswinter
/ locustdb-bulk-load-ssd-hdd
Created
August 12, 2018 17:53
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ➜ ~ uname -a | |
| Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | |
| ➜ ~ sudo lshw -class disk -class storage | |
| [sudo] password for clemens: | |
| *-storage | |
| description: SATA controller | |
| product: ASM1062 Serial ATA Controller | |
| vendor: ASMedia Technology Inc. | |
| physical id: 0 | |
| bus info: pci@0000:06:00.0 |
cswinter
/ clickhouse-cold-query-hdd-12-threads
Created
August 12, 2018 17:59
This file has been truncated, but you can view the full file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ➜ nyc-clickhouse uname -a | |
| Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | |
| ➜ nyc-clickhouse sudo lshw -class disk -class storage | |
| *-storage | |
| description: SATA controller | |
| product: ASM1062 Serial ATA Controller | |
| vendor: ASMedia Technology Inc. | |
| physical id: 0 | |
| bus info: pci@0000:06:00.0 |
cswinter
/ clickhouse-cold-query-ssd
Created
August 12, 2018 18:04
This file has been truncated, but you can view the full file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ➜ ~ uname -a | |
| Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | |
| ➜ ~ sudo lshw -class disk -class storage | |
| [sudo] password for clemens: | |
| *-storage | |
| description: SATA controller | |
| product: ASM1062 Serial ATA Controller | |
| vendor: ASMedia Technology Inc. | |
| physical id: 0 | |
| bus info: pci@0000:06:00.0 |
cswinter
/ clickhouse-cold-query-hdd-1-thread
Created
August 12, 2018 18:06
This file has been truncated, but you can view the full file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ➜ ~ uname -a | |
| Linux Archon 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | |
| ➜ ~ cd nyc-benchmarks/nyc-clickhouse | |
| ➜ nyc-clickhouse sudo lshw -class disk -class storage | |
| [sudo] password for clemens: | |
| *-storage | |
| description: SATA controller | |
| product: ASM1062 Serial ATA Controller | |
| vendor: ASMedia Technology Inc. | |
| physical id: 0 |
cswinter
/ clickhouse-log-import
Created
August 12, 2018 18:10
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| time (for filename in /home/clemens/Dropbox/nyc-taxi-data-denormalized/trips_x*.csv.gz; do | |
| gunzip -c $filename | \ | |
| python trans.py | \ | |
| clickhouse-client \ | |
| --query="INSERT INTO trips FORMAT CSV" | |
| done) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import sys | |
| for line in sys.stdin: | |
| print ','.join([item if len(item.strip()) else '\N' | |
| for item in line.strip().split(',')]) |
cswinter
/ locustdb-mem-breakdown
Created
August 12, 2018 19:46
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ➜ ~ cd src/LocustDB | |
| ➜ LocustDB git:(master) git fetch origin | |
| ^C | |
| ➜ LocustDB git:(master) git fetch && rebase | |
| Enter passphrase for key '/home/clemens/.ssh/id_rsa': | |
| ➜ LocustDB git:(master) git fetch && git rebase | |
| Enter passphrase for key '/home/clemens/.ssh/id_rsa': | |
| remote: Counting objects: 21, done. | |
| remote: Compressing objects: 100% (3/3), done. |