Skip to content

Instantly share code, notes, and snippets.

Avatar

Vikas Lalwani vikaslalwani

View GitHub Profile
View benchmark-commands.txt
Producer
Setup
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test-rep-one --partitions 6 --replication-factor 1
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test --partitions 6 --replication-factor 3
Single thread, no replication
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test7 50000000 100 -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 batch.size=8196
View StreamsJoinWithRepartitioning.java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
View StreamsDSLAndProcessorExample.java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
View StreamsDSLExample.java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
@vikaslalwani
vikaslalwani / aggregation.java
Created Mar 4, 2021 — forked from confluentgist/aggregation.java
Kafka Streams Example: Continuously aggregating a stream into a table
View aggregation.java
// Continuously aggregating a KStream into a KTable.
KStream<String, String> locationUpdatesStream = ...;
KTable<String, Long> locationsPerUser
= locationUpdatesStream
.groupBy((k, v) -> v.username)
.count();
View Streaming_Machine_Learning_with_Kafka_and_TensorFlow.py
import numpy as np
import tensorflow as tf
import tensorflow_io.kafka as kafka_io
# 1. Consume streaming data with Kafka and TensorFlow I/O
def func_x(x):
# Decode image to (28, 28)
x = tf.io.decode_raw(x, out_type=tf.uint8)
x = tf.reshape(x, [28, 28])
# Convert to float32 for tf.keras
View ProcessorAPIExample.java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
@vikaslalwani
vikaslalwani / self-signed-ssl-mongo.sh
Created Mar 2, 2021 — forked from exAspArk/self-signed-ssl-mongo.sh
Self-signed SSL Certificate with OpenSSL on MacOS | MongoDB
View self-signed-ssl-mongo.sh
openssl genrsa -out CAroot.key 2048
openssl req -new -key CAroot.key -out CAroot.csr # CN should be different from the certificates below
openssl req -x509 -days 1825 -key CAroot.key -in CAroot.csr -out CAroot.crt
cat CAroot.crt CAroot.key > CAroot.pem
openssl genrsa -out mongod.key 2048
openssl req -new -key mongod.key -out mongod.csr
openssl x509 -req -days 1825 -in mongod.csr -CA CAroot.pem -CAkey CAroot.key -CAcreateserial -out mongod.crt
cat mongod.crt mongod.key > mongod.pem
@vikaslalwani
vikaslalwani / Heap_Dump_Java_AWS_ECS.md
Created Feb 22, 2021 — forked from pwillis-els/Heap_Dump_Java_AWS_ECS.md
Dumping heap of a Java process running in AWS ECS
View Heap_Dump_Java_AWS_ECS.md

Heap dumping a Java process running in AWS ECS

Note: this guide is designed for AWS ECS services, but starting from Step 4 is functionally equivalent to any Docker container on a Linux host.

Step 1. Look up ECS service

  1. Log into the AWS Console using the appropriate AWS account
  2. Navigate to AWS ECS service clusters (https://console.aws.amazon.com/ecs/home)
  3. Make sure you are in the correct region, if not, switch to the correct region (second drop-down menu in top right corner)
  4. Select the correct cluster (ex: https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters//services)
  5. In the Services tab, In the 'Filter in this page' text box, type the name of the service