Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save sureshsaggar/3993648 to your computer and use it in GitHub Desktop.
Save sureshsaggar/3993648 to your computer and use it in GitHub Desktop.
Query: Saving data in _raw_ format in HDFS using FlumeNG?
# Usage:
# bin/flume-ng agent --conf ./conf/ -f conf/flume-webserver-agents.conf -Dflume.root.logger=DEBUG,console -n agent_hikemon
# ------------------------------------------------------------------------------------
# Agent 01 - Using Exec Source & AVRO Sink
# ------------------------------------------------------------------------------------
agent_hikemon.sinks = avroSink-consolidator
agent_hikemon.sources = tailSource
agent_hikemon.channels = MemoryChannel-WebServer
agent_hikemon.channels.MemoryChannel-WebServer.type = memory
agent_hikemon.channels.MemoryChannel-WebServer.capacity = 1000
agent_hikemon.sources.tailSource.type = exec
agent_hikemon.sources.tailSource.command = tail -F /tmp/ping.txt
agent_hikemon.sources.tailSource.channels = MemoryChannel-WebServer
agent_hikemon.sinks.avroSink-consolidator.type = avro
agent_hikemon.sinks.avroSink-consolidator.channel = MemoryChannel-WebServer
agent_hikemon.sinks.avroSink-consolidator.hostname = <my_ip>
agent_hikemon.sinks.avroSink-consolidator.port = <my_port>
# Usage:
# bin/flume-ng agent --conf ./conf/ -f conf/flume-consolidate.conf -Dflume.root.logger=DEBUG,console -n agent-mqtt
#
# ------------------------------------------------------------------------------------
# Agent 02 - Using AVRO Source & HDFS Sink
# ------------------------------------------------------------------------------------
agent-mqtt.sinks = HDFS
agent-mqtt.sources = avrosource-mqtt
agent-mqtt.channels = MemoryChannel-MQTT
agent-mqtt.channels.MemoryChannel-MQTT.type = memory
agent_foo.channels.MemoryChannel-MQTT.capacity = 1000
agent-mqtt.sources.avrosource-mqtt.type = avro
agent-mqtt.sources.avrosource-mqtt.channels = MemoryChannel-MQTT
agent-mqtt.sources.avrosource-mqtt.bind = <my_ip>
agent-mqtt.sources.avrosource-mqtt.port = <my_port>
agent-mqtt.sinks.HDFS.channel = MemoryChannel-MQTT
agent-mqtt.sinks.HDFS.type = hdfs
agent-mqtt.sinks.HDFS.hdfs.path = hdfs://<hadoop_namenode_hostname>:<hadoop_namenode_port>/tmp/agent-mqtt/
agent-mqtt.sinks.HDFS.hdfs.fileType = DataStream
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment