talevy/README.md

## README.md

      
    Raw
  

              README.md
            
          
    Running Logstash Output Kafka With Custom Partitioner

checkout example round-robin partitioner and build jar

$ git clone git@github.com:talevy/kafka-round-robin-partitioner.git
$ cd kafka-round-robin-partitioner
$ ./gradlew fatJar

set java classpath env variable so logstash knows where to find the partitioner class

export CLASSPATH="$CLASSPATH:/path/to/kafka-round-robin-partitioner-all-1.0.0-SNAPSHOT.jar"

create kafka topic

$ bin/kafka-topics.sh  --create --topic roundrobin --partitions 4 --zookeeper localhost:2181 --replication-factor 1

run logstash with kafkaroundrobin-logstash.conf

$ bin/logstash -f kafkaroundrobin-logstash.conf

You should now see that the logs were are spread across the partitions


## kafkaroundrobin-logstash.conf
input {
  generator {
    count => 400
    message => "round robin"
  }
}

output {
  kafka {
    topic_id => "roundrobin"
    partitioner_class => "org.logstash.kafka.partitioner.RoundRobinPartitioner"
  }
}
	input {
	generator {
	count => 400
	message => "round robin"
	}
	}

	output {
	kafka {
	topic_id => "roundrobin"
	partitioner_class => "org.logstash.kafka.partitioner.RoundRobinPartitioner"
	}
	}