I've been working with Apache Kafka for over 7 years. I inevitably find myself doing the same set of activities while I'm developing or working with someone else's system. Here's a set of Kafka productivity hacks for doing a few things way faster than you're probably doing them now. 🔥
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"_gen": "weightedOneOf", | |
"choices": [ | |
{ | |
"weight": 98, | |
"choice": { | |
"_gen": "add", | |
"args": [ | |
{ "_gen": "var", "var": "pickup" }, | |
{ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"generators": [ | |
{ | |
"topic": "customers", | |
"value": { | |
"customerId": { "_gen": "uuid" }, | |
"name": { "_gen": "string", "expr": "#{Name.full_name}" }, | |
"address": { "_gen": "string", "expr": "#{Address.full_address}" }, | |
"membership": { | |
"_gen": "oneOf", |
- Get ShadowTraffic
- Make a file called
heap-readings.json
(see below) - Run it with
docker run --env-file license.env -v ./heap-readings.json:/home/config.json shadowtraffic/shadowtraffic:latest --config /home/config.json --sample 15000 --quiet --no-pretty --stdout > data.json
To visualize the output:
- Download Visidata
- Make a file called
vgraph.vdj
(see below)
- Get ShadowTraffic
- Make a file called
flights.json
(see below) - Run it with
docker run --env-file license.env -v ./flights.json:/home/config.json shadowtraffic/shadowtraffic:latest --config /home/config.json --sample 100000 --quiet --no-pretty --stdout > data.json
To visualize the output:
- Download Visidata
- Make a file called
vgraph.vdj
(see below)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"generators": [ | |
{ | |
"topic": "flights", | |
"fork": { | |
"key": { | |
"_gen": "sequentialString", | |
"expr": "flight-~d" | |
}, | |
"stagger": {"ms": 5000}, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"generators": [ | |
{ | |
"topic": "funnelEvents", | |
"fork": { | |
"key": { | |
"_gen": "string", | |
"expr": "#{Name.username}" | |
}, | |
"stagger": { "ms": 200 } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"generators": [ | |
{ | |
"table": "sensors", | |
"vars": { | |
"id": { | |
"_gen": "oneOf", | |
"choices": ["sensor-1", "sensor-2", "sensor-3", "sensor-4", "sensor-5"] | |
} | |
}, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"generators": [ | |
{ | |
"topic": "pushEvents", | |
"value": { | |
"commits": { | |
"_gen": "repeatedly", | |
"n": { "_gen": "uniformDistribution", "bounds": [1, 10] }, | |
"target": { | |
"id": {"_gen": "gitSha"}, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"generators": [ | |
{ | |
"topic": "passwordResetFlow", | |
"vars": { | |
"username": { | |
"_gen": "oneOf", | |
"choices": [ | |
{ "_gen" : "string", "expr" : "#{Name.username}" }, | |
{ |
NewerOlder