Skip to content

Instantly share code, notes, and snippets.

@thakursaurabh1998
Created August 9, 2020 15:29
Show Gist options
  • Save thakursaurabh1998/0aa3c419b841b9057237006cbdd92681 to your computer and use it in GitHub Desktop.
Save thakursaurabh1998/0aa3c419b841b9057237006cbdd92681 to your computer and use it in GitHub Desktop.
const async = require('async');
const { ConsumerGroup } = require('kafka-node');
const consumerConfig = {
kafkaHost: 'broker:9092',
groupId: 'order-service',
fetchMaxBytes: 100 * 1024, // 100 KB
autoCommit: false,
};
const consumer = new ConsumerGroup(consumerConfig, 'OrderTopic');
async function processData(queueData) {
// perform data processing here
}
const concurrency = 5;
const q = async.queue(processData, concurrency);
consumer.on('message', (data) => {
consumer.pause();
q.push(data);
});
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment