Skip to content

Instantly share code, notes, and snippets.

@octo47
Created August 8, 2013 19:34
Show Gist options
  • Save octo47/6187922 to your computer and use it in GitHub Desktop.
Save octo47/6187922 to your computer and use it in GitHub Desktop.
tsdbflume.sources = source1 source2
tsdbflume.sinks = sink1 sink2
tsdbflume.channels = channel1 channel2
# Describe/configure source1
tsdbflume.sources.source1.type = ru.yandex.opentsdb.flume.OpenTSDBSource
tsdbflume.sources.source1.port = 4444
tsdbflume.sources.source1.batchSize = 25000
tsdbflume.channels.channel1.type = FILE
tsdbflume.channels.channel1.checkpointDir = /local/hd1/opentsdb/flume/checkpoint
tsdbflume.channels.channel1.dataDirs = /local/hd1/opentsdb/flume/data,/local/hd2/opentsdb/flume/data
tsdbflume.channels.channel1.transactionCapacity = 40000
tsdbflume.channels.channel1.checkpointInterval = 200
tsdbflume.channels.channel1.maxFileSize = 2146435071
tsdbflume.channels.channel1.capacity = 1000000
#keep-alive 3 Amount of time (in sec) to wait for a put operation
#write-timeout 3 Amount of time (in sec) to wait for a write operation
tsdbflume.sinks.sink1.type = ru.yandex.opentsdb.flume.OpenTSDBSink
tsdbflume.sinks.sink1.batchSize = 25000
tsdbflume.sinks.sink1.parallel = 8
tsdbflume.sinks.sink1.zkquorum = zk.quorum.of.hbase.here
tsdbflume.sinks.sink1.zkpath = /hbase
tsdbflume.sources.source1.channels = channel1
tsdbflume.sinks.sink1.channel = channel1
# Describe/configure golovan
tsdbflume.sources.source2.type = ru.yandex.opentsdb.flume.LegacyHttpSource
tsdbflume.sources.source2.tsdb.url = http://localhost:8444
tsdbflume.sources.source2.port = 5444
tsdbflume.sources.source2.batchSize = 25000
tsdbflume.channels.channel2.type = FILE
tsdbflume.channels.channel2.checkpointDir = /local/hd1/opentsdb/flume/checkpoint2
tsdbflume.channels.channel2.dataDirs = /local/hd1/opentsdb/flume/data2,/local/hd2/opentsdb/flume/data2
tsdbflume.channels.channel2.transactionCapacity = 40000
tsdbflume.channels.channel2.checkpointInterval = 200
tsdbflume.channels.channel2.maxFileSize = 2146435071
tsdbflume.channels.channel2.capacity = 1000000
tsdbflume.sinks.sink2.type = ru.yandex.opentsdb.flume.OpenTSDBSink
tsdbflume.sinks.sink2.batchSize = 25000
tsdbflume.sinks.sink1.zkquorum = zk.quorum.of.hbase.here
tsdbflume.sinks.sink1.zkpath = /hbase
tsdbflume.sources.source2.channels = channel2
tsdbflume.sinks.sink2.channel = channel2
@spyroslalos
Copy link

So if i want to send those data:

batchhosts.max 137031 12 hostname=lxb1 status=closed_Full prod=1
batchhosts.run 137031 12 hostname=lxb2 status=closed_Full prod=1
batchhosts.max 137031 12 hostname=lxb3 status=closed_Full prod=1

How can i send them to the source and in which format?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment