I've been working with Apache Kafka for over 7 years. I inevitably find myself doing the same set of activities while I'm developing or working with someone else's system. Here's a set of Kafka productivity hacks for doing a few things way faster than you're probably doing them now. 🔥
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
numnodes=2 | |
baseip="192.168.10" | |
#global script | |
$global = <<SCRIPT | |
#check for private key for vm-vm comm | |
[ -f /vagrant/id_rsa ] || { | |
ssh-keygen -t rsa -f /vagrant/id_rsa -q -N '' | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CREATE TABLE large_test (num1 bigint, num2 double precision, num3 double precision); | |
INSERT INTO large_test (num1, num2, num3) | |
SELECT round(random()*10), random(), random()*142 | |
FROM generate_series(1, 20000000) s(i); | |
EXPLAIN (analyse, buffers) | |
SELECT num1, avg(num3) as num3_avg, sum(num2) as num2_sum | |
FROM large_test | |
GROUP BY num1; |
- Create
UNLOGGED
table. This reduces the amount of data written to persistent storage by up to 2x. - Set
WITH (autovacuum_enabled=false)
on the table. This saves CPU time and IO bandwidth on useless vacuuming of the table (since we neverDELETE
orUPDATE
the table). - Insert rows with
COPY FROM STDIN
. This is the fastest possible approach to insert rows into table. - Minimize the number of indexes in the table, since they slow down inserts. Usually an index
on
time timestamp with time zone
is enough. - Add
synchronous_commit = off
topostgresql.conf
. - Use table inheritance for fast removal of old data:
create different ssh key according the article Mac Set-Up Git
$ ssh-keygen -t rsa -C "your_email@youremail.com"