Kafka Design

gistfile1.txt

Kafka:
	
	Messaging Queue with File System
	Developed in Java
	Every partition replicated 3 times [1 replica is leader]
	Replicas live on separate brokers
	If leader goes down, other partition becomes leader

	Actors:
		Producers
		Consumers
		Partitions
		Groups
			groupId
			MemberID
		Group Leader
		Brokers
		Broker Coordinators - facilitates communication with consumer groups


	Consumer Management:
		Offset Management:
			If consumer crashes, it starts with last saved offset (__consumer_offset topic)
		Automatic failover
		If excessive consumers added, unemployed consumers

		Consumer Group:
			Rebalancing workload

		Protocols

			Zookeerper Replacement for manager groups developed
			Dynamic Partitions
			Partition assignment needs to be handled by client
				Any attempt to enforce rules by the broker restricting the clients.
				Client has the broker's information, so client should handle ruling


			Protcol Design:
				Join the group
				elect group leader
				Assign resources, via group leader
				Maintain group memebership
				Leave group

			Usage:
				KafkaStreams: Assigning partitions to tasks to hosts
				SchemaRegistery: Leader Election

			Source Code Working:
				FindCoordinator
				JoinGroup
					Assignment Protocol:
						Sitcky (major priority)
						Roundrobin
				SyncGroup
					Rebalancing
						Consumers continuously poll, they send heartbeat

					Schema Registry:
						Schema Registry Leader

				Heartbeat
				LeaveGroup