srishtyagrawal

# A Kubernetes implementation which helps in setting up a Kubernetes cluster on your system
minikube version
# Starts a Kubernetes cluster
minikube start
# kubectl is the command line interface for Kubernetes, check if it is installed
kubectl version 
# the server version is the Kubernetes version installed on the master
kubectl cluster-info

# Generic kubectl commands 

srishtyagrawal

We couldn’t find that file to show.

srishtyagrawal

Apache Storm:

• Print mapping between Task ID and Kafka Partitions
• Add option to configure pagination in Storm UI
• Map of Spout configurations from storm-kafka to storm-kafka-client

Apache Storm-Mesos:

• Removing taskIDtoOfferMap
• storm-mesos release

Worked on these with colleagues at Groupon:

srishtyagrawal

startOffsetTime	forceFromStart	FirstPollOffsetStrategy
EarliestTime	true	EARLIEST
EarliestTime	false	UNCOMMITTED_EARLIEST
LatestTime	true	LATEST
LatestTime	false	UNCOMMITTED_LATEST

List of all values for FirstPollOffsetStrategy

srishtyagrawal

The storm-kafka configs were taken from SpoutConfig and KafkaConfig. storm-kafka-client spout configurations were taken from KafkaSpoutConfig.

Storm-0.9.6 SpoutConfig	Storm-1.0.6 KafkaSpoutConfig name	KafkaSpoutConfig usage help
Setting: startOffsetTime Default: EarliestTime ________________________________________________ forceFromStart Default: false startOffsetTime & forceFromStart together determin

srishtyagrawal

2017-05-01 15:43:34.845 o.a.s.d.worker [INFO] Launching worker for StormKafka-1-1493678565 on 23b04ee0-b493-440f-88e2-bd1250c41f34:6702 with id ff87b54a-f9c8-43f0-aefc-06764208214f and conf {"topology.builtin.metrics.bucket.size.secs" 60, "nimbus.childopts" "-Xmx1024m", "ui.filter.params" nil, "storm.cluster.mode" "distributed", "storm.messaging.netty.client_worker_threads" 1, "logviewer.max.per.worker.logs.size.mb" 2048, "supervisor.run.worker.as.user" false, "topology.max.task.parallelism" nil, "topology.priority" 29, "zmq.threads" 1, "storm.group.mapping.service" "org.apache.storm.security.auth.ShellBasedGroupsMapping", "transactional.zookeeper.root" "/transactional", "topology.sleep.spout.wait.strategy.time.ms" 1, "scheduler.display.resource" false, "topology.max.replication.wait.time.sec" 60, "drpc.invocations.port" 3773, "supervisor.localizer.cache.target.size.mb" 10240, "topology.multilang.serializer" "org.apache.storm.multilang.JsonSerializer", "storm.messaging.netty.server_worker_threads" 1, "nimbus.

srishtyagrawal

o.a.k.c.c.i.ConsumerCoordinator Thread-12-kafkaspout-executor[3 10] [INFO] Setting newly assigned partitions [8topic-4, 8topic-6, 8topic-5, 8topic-7] for group kafkaSpoutTestGroup
o.a.s.k.s.KafkaSpout Thread-12-kafkaspout-executor[3 10] [INFO] Partitions reassignment. [taskID=10, consumer-group=kafkaSpoutTestGroup, consumer=org.apache.kafka.clients.consumer.KafkaConsumer@108e79ce, topic-partitions=[8topic-4, 8topic-6, 8topic-5, 8topic-7]]
o.a.k.c.c.i.ConsumerCoordinator Thread-8-kafkaspout-executor[11 18] [INFO] Setting newly assigned partitions [8topic-2, 8topic-1, 8topic-3, 8topic-0] for group kafkaSpoutTestGroup
o.a.s.k.s.KafkaSpout Thread-8-kafkaspout-executor[11 18] [INFO] Partitions reassignment. [taskID=15, consumer-group=kafkaSpoutTestGroup, consumer=org.apache.kafka.clients.consumer.KafkaConsumer@2dc37126, topic-partitions=[8topic-2, 8topic-1, 8topic-3, 8topic-0]]

srishtyagrawal

2017-05-03 13:50:47.631 o.a.k.c.c.i.ConsumerCoordinator Thread-12-kafkaspout-executor[3 10] [INFO] Setting newly assigned partitions [8topic-4, 8topic-6, 8topic-5, 8topic-7] for group kafkaSpoutTestGroup
2017-05-03 13:50:47.631 o.a.s.k.s.KafkaSpout Thread-12-kafkaspout-executor[3 10] [INFO] Partitions reassignment. [taskID=10, consumer-group=kafkaSpoutTestGroup, consumer=org.apache.kafka.clients.consumer.KafkaConsumer@108e79ce, topic-partitions=[8topic-4, 8topic-6, 8topic-5, 8topic-7]]
2017-05-03 13:50:47.631 o.a.k.c.c.i.ConsumerCoordinator Thread-8-kafkaspout-executor[11 18] [INFO] Setting newly assigned partitions [8topic-2, 8topic-1, 8topic-3, 8topic-0] for group kafkaSpoutTestGroup
2017-05-03 13:50:47.631 o.a.s.k.s.KafkaSpout Thread-8-kafkaspout-executor[11 18] [INFO] Partitions reassignment. [taskID=15, consumer-group=kafkaSpoutTestGroup, consumer=org.apache.kafka.clients.consumer.KafkaConsumer@2dc37126, topic-partitions=[8topic-2, 8topic-1, 8topic-3, 8topic-0]]
2017-05-03 13:50:47.646 o.a.s.d.task Threa

srishtyagrawal

o.a.k.c.c.i.ConsumerCoordinator Thread-12-kafkaspout-executor[3 10] [INFO] Setting newly assigned partitions [8topic-2, 8topic-1, 8topic-3, 8topic-0] for group kafkaSpoutTestGroup
o.a.k.c.c.i.ConsumerCoordinator Thread-8-kafkaspout-executor[11 18] [INFO] Setting newly assigned partitions [8topic-4, 8topic-6, 8topic-5, 8topic-7] for group kafkaSpoutTestGroup
o.a.s.k.s.KafkaSpout Thread-12-kafkaspout-executor[3 10] [INFO] Partitions reassignment. [consumer-group=kafkaSpoutTestGroup, consumer=org.apache.kafka.clients.consumer.KafkaConsumer@3de21778, topic-partitions=[8topic-2, 8topic-1, 8topic-3, 8topic-0]]
o.a.s.k.s.KafkaSpout Thread-8-kafkaspout-executor[11 18] [INFO] Partitions reassignment. [consumer-group=kafkaSpoutTestGroup, consumer=org.apache.kafka.clients.consumer.KafkaConsumer@374884cf, topic-partitions=[8topic-4, 8topic-6, 8topic-5, 8topic-7]]

srishtyagrawal

o.a.s.k.KafkaUtils [INFO] Task [2/16] assigned [Partition{host=localhost:9092, topic=topic8, partition=1}]
o.a.s.k.ZkCoordinator [INFO] Task [2/16] Deleted partition managers: []
o.a.s.k.ZkCoordinator [INFO] Task [2/16] New partition managers: [Partition{host=localhost:9092, topic=topic8, partition=1}]
o.a.s.k.PartitionManager [INFO] Read partition information from: /consumers/consumer-topic8/partition_1  --> null
o.a.s.k.PartitionManager [INFO] No partition information found, using o.a.s.k.PartitionManager [INFO] Starting Kafka localhost Partition{host=localhost:9092, topic=topic8, partition=1} from offset 14