Skip to content

Instantly share code, notes, and snippets.

@sureshsaggar

sureshsaggar/Apache FlumeNG and HDFS Sink

Created October 31, 2012 16:16
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save sureshsaggar/3987987 to your computer and use it in GitHub Desktop.
Save sureshsaggar/3987987 to your computer and use it in GitHub Desktop.
Download ZIP
Apache FlumeNG and HDFS Sink
Raw
Apache FlumeNG and HDFS Sink
# Directory:
# root@localhost:~/flume/apache-flume-1.4.0-SNAPSHOT
# Usage:
# bin/flume-ng agent --conf ./conf/ -f conf/flume-agents-tests.conf -Dflume.root.logger=DEBUG,console -n agent_test_exec_TO_hdfs
# ------------------------------------------------------------------------------------
# This workflow applies to some WebServer running flume agent and dumping data
# back in HDFS. Here /tmp/ping.txt could be any log file.
# ------------------------------------------------------------------------------------
agent_test_exec_TO_hdfs.sinks = HDFS
agent_test_exec_TO_hdfs.sources = tailSourceTest
agent_test_exec_TO_hdfs.channels = MemoryChannel-WebServerTest
agent_test_exec_TO_hdfs.channels.MemoryChannel-WebServerTest.type = memory
agent_test_exec_TO_hdfs.channels.MemoryChannel-WebServerTest.capacity = 1000
agent_test_exec_TO_hdfs.sources.tailSourceTest.type = exec
agent_test_exec_TO_hdfs.sources.tailSourceTest.command = tail -F /tmp/ping.txt
agent_test_exec_TO_hdfs.sources.tailSourceTest.channels = MemoryChannel-WebServerTest
agent_test_exec_TO_hdfs.sinks.HDFS.channel = MemoryChannel-WebServerTest
agent_test_exec_TO_hdfs.sinks.HDFS.type = hdfs
agent_test_exec_TO_hdfs.sinks.HDFS.hdfs.path = hdfs://<hadoop-namenode-hostname>:<port>/tmp/agent-webserver/
agent_test_exec_TO_hdfs.sinks.HDFS.hdfs.fileType = DataStream
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment