Skip to content

Instantly share code, notes, and snippets.

Apurva Mehta apurvam

Block or report user

Report or block apurvam

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@apurvam
apurvam / get_open_prs.py
Last active Feb 13, 2019
Get all open PRs in all repositories matching a regex.
View get_open_prs.py
#!/usr/bin/python
from urllib2 import Request, urlopen, URLError
import sys
from argparse import ArgumentParser
import json
import re
from dateutil.parser import parse
MAX_RESULTS = 50
@apurvam
apurvam / timestamp-format.sql
Last active Apr 28, 2018
Example of specify TIMESTAMP_FORMAT in KSQL
View timestamp-format.sql
CREATE STREAM foo (col1 varchar, col2 varchar, timecol varchar) \
WITH (kafka_topic=’foo_topic’, TIMESTAMP=’timecol’, \
TIMESTAMP_FORMAT=’yyyy-MM-ddTHH:mm:SSZ’, value_format=’json’);
@apurvam
apurvam / table-aggregations.sql
Last active Apr 28, 2018
Example of table aggregations in KSQL
View table-aggregations.sql
CREATE TABLE users (userid bigint, zipcode int, created_at bigint) \
WITH (kafka_topic=’users’, value_format=’json’, key=’userid’);
SELECT zipcode, count(*) from users GROUP BY zipcode;
@apurvam
apurvam / safe-drop-example.sh
Created Apr 24, 2018
Example of how KSQL protects against dropping tables and streams which are in use.
View safe-drop-example.sh
ksql> show queries;
Query ID | Kafka Topic | Query String
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CSAS_PAGEVIEWS_ENRICHED | PAGEVIEWS_ENRICHED | CREATE STREAM pageviews_enriched AS SELECT users_original.userid AS userid, pageid, regionid, gender FROM pageviews_original LEFT JOIN users_original ON pageviews_original.userid = users_original.userid;
CSAS_PAGEVIEWS_FEMALE | PAGEVIEWS_FEMALE | CREATE STREAM pageviews_female AS SELECT * FROM pageviews_enriched WHERE gender = 'FEMALE';
CSAS_PAGEVIEWS_FEMALE_LIKE_89 | pageviews_enriched_r8_r9 | CREATE STREAM pageviews_female_like_89 WITH (kaf
@apurvam
apurvam / ksql-print-metrics-example.txt
Last active Jan 31, 2018
Example output for the ksql-print-metrics-tool
View ksql-print-metrics-example.txt
$ ./bin/ksql-print-metrics
messages-consumed-avg: 96416.96196183885
messages-consumed-min: 88900.3329377909
error-rate: 0.0
num-persistent-queries: 2.0
messages-consumed-per-sec: 193024.78294586178
messages-produced-per-sec: 193025.4730374501
num-active-queries: 2.0
num-idle-queries: 0.0
messages-consumed-max: 103397.81191436431
@apurvam
apurvam / topkdistinct.sql
Last active Jan 31, 2018
KSQL TOPKDISTINCT example
View topkdistinct.sql
SELECT pageid, TOPKDISTINCT(viewtime, 5) FROM pageviews_users GROUP BY pageid;
@apurvam
apurvam / topk.sql
Created Jan 30, 2018
KSQL TopK example
View topk.sql
SELECT orderzip_code, TOPK(order_total, 5) FROM orders WINDOW TUMBLING (SIZE 1 HOUR) GROUP BY order_zipcode;
@apurvam
apurvam / print-topic.sh
Last active Jan 25, 2018
An example of how to dump the contents of a topic using 'print'.
View print-topic.sh
ksql> print 'pageviews_kafka_topic_json';
Format:JSON
{"ROWTIME":1516753301093,"ROWKEY":"1","viewtime":1516753300961,"pageid":"Page_53","userid":"User_9"}
{"ROWTIME":1516753301458,"ROWKEY":"11","viewtime":1516753301458,"pageid":"Page_38","userid":"User_5"}
{"ROWTIME":1516753301660,"ROWKEY":"21","viewtime":1516753301660,"pageid":"Page_25","userid":"User_2"}
^C
@apurvam
apurvam / show-topics.sh
Created Jan 24, 2018
An example of the updates to the 'show topics' command.
View show-topics.sh
ksql> show topics;
Kafka Topic | Registered | Partitions | Partition Replicas | Consumers | Consumer Groups
----------------------------------------------------------------------------------------------------------
__confluent.support.metrics | false | 1 | 1 | 0 | 0
_schemas | false | 1 | 1 | 0 | 0
connect-configs | false | 1 | 1 | 0 | 0
connect-offsets | false | 25 | 1 | 0 | 0
connect-statuses | false | 5 | 1 | 0 | 0
groupbydevice | true | 4 | 1 | 0 | 0
View KafkaTransactionsExample.java
KafkaProducer producer = createKafkaProducer(
“bootstrap.servers”, “localhost:9092”,
“transactional.id”, “my-transactional-id”);
producer.initTransactions();
KafkaConsumer consumer = createKafkaConsumer(
“bootstrap.servers”, “localhost:9092”,
“group.id”, “my-group-id”,
"isolation.level", "read_committed");
You can’t perform that action at this time.