Skip to content

Instantly share code, notes, and snippets.

Apurva Mehta apurvam

View GitHub Profile
@apurvam
apurvam / get_open_prs.py
Last active Feb 13, 2019
Get all open PRs in all repositories matching a regex.
View get_open_prs.py
#!/usr/bin/python
from urllib2 import Request, urlopen, URLError
import sys
from argparse import ArgumentParser
import json
import re
from dateutil.parser import parse
MAX_RESULTS = 50
@apurvam
apurvam / timestamp-format.sql
Last active Apr 28, 2018
Example of specify TIMESTAMP_FORMAT in KSQL
View timestamp-format.sql
CREATE STREAM foo (col1 varchar, col2 varchar, timecol varchar) \
WITH (kafka_topic=’foo_topic’, TIMESTAMP=’timecol’, \
TIMESTAMP_FORMAT=’yyyy-MM-ddTHH:mm:SSZ’, value_format=’json’);
@apurvam
apurvam / table-aggregations.sql
Last active Apr 28, 2018
Example of table aggregations in KSQL
View table-aggregations.sql
CREATE TABLE users (userid bigint, zipcode int, created_at bigint) \
WITH (kafka_topic=’users’, value_format=’json’, key=’userid’);
SELECT zipcode, count(*) from users GROUP BY zipcode;
@apurvam
apurvam / safe-drop-example.sh
Created Apr 24, 2018
Example of how KSQL protects against dropping tables and streams which are in use.
View safe-drop-example.sh
ksql> show queries;
Query ID | Kafka Topic | Query String
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CSAS_PAGEVIEWS_ENRICHED | PAGEVIEWS_ENRICHED | CREATE STREAM pageviews_enriched AS SELECT users_original.userid AS userid, pageid, regionid, gender FROM pageviews_original LEFT JOIN users_original ON pageviews_original.userid = users_original.userid;
CSAS_PAGEVIEWS_FEMALE | PAGEVIEWS_FEMALE | CREATE STREAM pageviews_female AS SELECT * FROM pageviews_enriched WHERE gender = 'FEMALE';
CSAS_PAGEVIEWS_FEMALE_LIKE_89 | pageviews_enriched_r8_r9 | CREATE STREAM pageviews_female_like_89 WITH (kaf
@apurvam
apurvam / ksql-print-metrics-example.txt
Last active Jan 31, 2018
Example output for the ksql-print-metrics-tool
View ksql-print-metrics-example.txt
$ ./bin/ksql-print-metrics
messages-consumed-avg: 96416.96196183885
messages-consumed-min: 88900.3329377909
error-rate: 0.0
num-persistent-queries: 2.0
messages-consumed-per-sec: 193024.78294586178
messages-produced-per-sec: 193025.4730374501
num-active-queries: 2.0
num-idle-queries: 0.0
messages-consumed-max: 103397.81191436431
@apurvam
apurvam / topkdistinct.sql
Last active Jan 31, 2018
KSQL TOPKDISTINCT example
View topkdistinct.sql
SELECT pageid, TOPKDISTINCT(viewtime, 5) FROM pageviews_users GROUP BY pageid;
@apurvam
apurvam / topk.sql
Created Jan 30, 2018
KSQL TopK example
View topk.sql
SELECT orderzip_code, TOPK(order_total, 5) FROM orders WINDOW TUMBLING (SIZE 1 HOUR) GROUP BY order_zipcode;
@apurvam
apurvam / print-topic.sh
Last active Jan 25, 2018
An example of how to dump the contents of a topic using 'print'.
View print-topic.sh
ksql> print 'pageviews_kafka_topic_json';
Format:JSON
{"ROWTIME":1516753301093,"ROWKEY":"1","viewtime":1516753300961,"pageid":"Page_53","userid":"User_9"}
{"ROWTIME":1516753301458,"ROWKEY":"11","viewtime":1516753301458,"pageid":"Page_38","userid":"User_5"}
{"ROWTIME":1516753301660,"ROWKEY":"21","viewtime":1516753301660,"pageid":"Page_25","userid":"User_2"}
^C
@apurvam
apurvam / show-topics.sh
Created Jan 24, 2018
An example of the updates to the 'show topics' command.
View show-topics.sh
ksql> show topics;
Kafka Topic | Registered | Partitions | Partition Replicas | Consumers | Consumer Groups
----------------------------------------------------------------------------------------------------------
__confluent.support.metrics | false | 1 | 1 | 0 | 0
_schemas | false | 1 | 1 | 0 | 0
connect-configs | false | 1 | 1 | 0 | 0
connect-offsets | false | 25 | 1 | 0 | 0
connect-statuses | false | 5 | 1 | 0 | 0
groupbydevice | true | 4 | 1 | 0 | 0
View KafkaTransactionsExample.java
KafkaProducer producer = createKafkaProducer(
“bootstrap.servers”, “localhost:9092”,
“transactional.id”, “my-transactional-id”);
producer.initTransactions();
KafkaConsumer consumer = createKafkaConsumer(
“bootstrap.servers”, “localhost:9092”,
“group.id”, “my-group-id”,
"isolation.level", "read_committed");
You can’t perform that action at this time.