Skip to content

Instantly share code, notes, and snippets.

@joshmarcus

joshmarcus/flume.log

Created January 15, 2014 15:19
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save joshmarcus/8438107 to your computer and use it in GitHub Desktop.
Save joshmarcus/8438107 to your computer and use it in GitHub Desktop.
Download ZIP
flume config & logs (reload config issue)
Raw
flume.log
15 Jan 2014 10:01:55,382 INFO [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:61) - Configuration provider starting
15 Jan 2014 10:01:55,396 INFO [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133) - Reloading configuration file:/usr/local/flume/conf/agents/shared-agent.conf
15 Jan 2014 10:01:55,410 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:01:55,412 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:01:55,412 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:01:55,412 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:01:55,418 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:01:55,418 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:01:55,418 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:01:55,419 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:01:55,419 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:01:55,419 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:01:55,420 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:01:55,420 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:01:55,420 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:01:55,420 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930) - Added sinks: hdfs-sink poison-sink Agent: shared-agent
15 Jan 2014 10:01:55,424 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:01:55,424 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:01:55,424 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:01:55,424 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:01:55,425 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:01:55,425 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:01:55,425 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:01:55,490 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140) - Post-validation flume configuration contains configuration for agents: [shared-agent]
15 Jan 2014 10:01:55,490 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:150) - Creating channels
15 Jan 2014 10:01:55,506 INFO [conf-file-poller-0] (org.apache.flume.channel.DefaultChannelFactory.create:40) - Creating instance of channel poison-channel type memory
15 Jan 2014 10:01:55,517 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:205) - Created channel poison-channel
15 Jan 2014 10:01:55,518 INFO [conf-file-poller-0] (org.apache.flume.channel.DefaultChannelFactory.create:40) - Creating instance of channel memory-channel type memory
15 Jan 2014 10:01:55,518 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:205) - Created channel memory-channel
15 Jan 2014 10:01:55,520 INFO [conf-file-poller-0] (org.apache.flume.source.DefaultSourceFactory.create:39) - Creating instance of source syslogudp-source, type syslogudp
15 Jan 2014 10:01:55,570 INFO [conf-file-poller-0] (org.apache.flume.interceptor.StaticInterceptor$Builder.build:133) - Creating StaticInterceptor: preserveExisting=true,key=timestamp,value=UNKNOWN
15 Jan 2014 10:01:55,574 INFO [conf-file-poller-0] (org.apache.flume.sink.DefaultSinkFactory.create:40) - Creating instance of sink: hdfs-sink, type: hdfs
15 Jan 2014 10:01:55,954 INFO [conf-file-poller-0] (org.apache.flume.sink.hdfs.HDFSEventSink.authenticate:493) - Hadoop Security enabled: false
15 Jan 2014 10:01:55,957 INFO [conf-file-poller-0] (org.apache.flume.sink.DefaultSinkFactory.create:40) - Creating instance of sink: poison-sink, type: hdfs
15 Jan 2014 10:01:55,957 INFO [conf-file-poller-0] (org.apache.flume.sink.hdfs.HDFSEventSink.authenticate:493) - Hadoop Security enabled: false
15 Jan 2014 10:01:55,960 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:119) - Channel poison-channel connected to [syslogudp-source, poison-sink]
15 Jan 2014 10:01:55,960 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:119) - Channel memory-channel connected to [syslogudp-source, hdfs-sink]
15 Jan 2014 10:01:55,970 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138) - Starting new configuration:{ sourceRunners:{syslogudp-source=EventDrivenSourceRunner: { source:org.apache.flume.source.SyslogUDPSource{name:syslogudp-source,state:IDLE} }} sinkRunners:{hdfs-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@5c6ed322 counterGroup:{ name:null counters:{} } }, poison-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@6fe78c96 counterGroup:{ name:null counters:{} } }} channels:{poison-channel=org.apache.flume.channel.MemoryChannel{name: poison-channel}, memory-channel=org.apache.flume.channel.MemoryChannel{name: memory-channel}} }
15 Jan 2014 10:01:55,971 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:145) - Starting Channel poison-channel
15 Jan 2014 10:01:55,972 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:145) - Starting Channel memory-channel
15 Jan 2014 10:01:55,974 INFO [lifecycleSupervisor-1-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:110) - Monitoried counter group for type: CHANNEL, name: memory-channel, registered successfully.
15 Jan 2014 10:01:55,975 INFO [lifecycleSupervisor-1-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: CHANNEL, name: memory-channel started
15 Jan 2014 10:01:55,974 INFO [lifecycleSupervisor-1-1] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:110) - Monitoried counter group for type: CHANNEL, name: poison-channel, registered successfully.
15 Jan 2014 10:01:55,975 INFO [lifecycleSupervisor-1-1] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: CHANNEL, name: poison-channel started
15 Jan 2014 10:01:55,976 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:173) - Starting Sink hdfs-sink
15 Jan 2014 10:01:55,976 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:173) - Starting Sink poison-sink
15 Jan 2014 10:01:55,977 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:184) - Starting Source syslogudp-source
15 Jan 2014 10:01:55,980 INFO [lifecycleSupervisor-1-2] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:110) - Monitoried counter group for type: SINK, name: hdfs-sink, registered successfully.
15 Jan 2014 10:01:55,981 INFO [lifecycleSupervisor-1-2] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: SINK, name: hdfs-sink started
15 Jan 2014 10:01:55,981 INFO [lifecycleSupervisor-1-4] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:110) - Monitoried counter group for type: SINK, name: poison-sink, registered successfully.
15 Jan 2014 10:01:55,982 INFO [lifecycleSupervisor-1-4] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: SINK, name: poison-sink started
15 Jan 2014 10:06:18,741 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSDataStream.configure:56) - Serializer = TEXT, UseRawLocalFileSystem = false
15 Jan 2014 10:06:18,808 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.BucketWriter.open:219) - Creating hdfs://hadoop.int.meetup.com:9000/flume/shared/manual-test/date=2014-01-15/FlumeData.1389798378741.tmp
15 Jan 2014 10:06:55,983 INFO [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133) - Reloading configuration file:/usr/local/flume/conf/agents/shared-agent.conf
15 Jan 2014 10:06:55,985 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:06:55,986 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:06:55,986 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:06:55,986 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:06:55,986 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:06:55,991 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:06:55,991 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:06:55,991 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:06:55,991 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:06:55,992 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:06:55,992 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:06:55,992 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:06:55,993 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:06:55,993 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930) - Added sinks: hdfs-sink poison-sink Agent: shared-agent
15 Jan 2014 10:06:55,993 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:poison-sink
15 Jan 2014 10:06:55,993 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:06:55,993 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:06:55,994 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:06:55,994 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:06:55,994 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:06:55,994 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:hdfs-sink
15 Jan 2014 10:06:56,005 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140) - Post-validation flume configuration contains configuration for agents: [shared-agent]
15 Jan 2014 10:06:56,006 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:150) - Creating channels
15 Jan 2014 10:06:56,006 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:205) - Created channel poison-channel
15 Jan 2014 10:06:56,006 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:205) - Created channel memory-channel
15 Jan 2014 10:06:56,007 INFO [conf-file-poller-0] (org.apache.flume.source.DefaultSourceFactory.create:39) - Creating instance of source syslogudp-source, type syslogudp
15 Jan 2014 10:06:56,008 INFO [conf-file-poller-0] (org.apache.flume.interceptor.StaticInterceptor$Builder.build:133) - Creating StaticInterceptor: preserveExisting=true,key=timestamp,value=UNKNOWN
15 Jan 2014 10:06:56,009 INFO [conf-file-poller-0] (org.apache.flume.sink.DefaultSinkFactory.create:40) - Creating instance of sink: hdfs-sink, type: hdfs
15 Jan 2014 10:06:56,009 INFO [conf-file-poller-0] (org.apache.flume.sink.hdfs.HDFSEventSink.authenticate:493) - Hadoop Security enabled: false
15 Jan 2014 10:06:56,009 INFO [conf-file-poller-0] (org.apache.flume.sink.DefaultSinkFactory.create:40) - Creating instance of sink: poison-sink, type: hdfs
15 Jan 2014 10:06:56,009 INFO [conf-file-poller-0] (org.apache.flume.sink.hdfs.HDFSEventSink.authenticate:493) - Hadoop Security enabled: false
15 Jan 2014 10:06:56,010 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:119) - Channel poison-channel connected to [syslogudp-source, poison-sink]
15 Jan 2014 10:06:56,010 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:119) - Channel memory-channel connected to [syslogudp-source, hdfs-sink]
15 Jan 2014 10:06:56,010 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:101) - Shutting down configuration: { sourceRunners:{syslogudp-source=EventDrivenSourceRunner: { source:org.apache.flume.source.SyslogUDPSource{name:syslogudp-source,state:START} }} sinkRunners:{hdfs-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@5c6ed322 counterGroup:{ name:null counters:{runner.backoffs.consecutive=5, runner.backoffs=39} } }, poison-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@6fe78c96 counterGroup:{ name:null counters:{runner.backoffs.consecutive=39, runner.backoffs=39} } }} channels:{poison-channel=org.apache.flume.channel.MemoryChannel{name: poison-channel}, memory-channel=org.apache.flume.channel.MemoryChannel{name: memory-channel}} }
15 Jan 2014 10:06:56,011 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:105) - Stopping Source syslogudp-source
15 Jan 2014 10:06:56,011 INFO [conf-file-poller-0] (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:171) - Stopping component: EventDrivenSourceRunner: { source:org.apache.flume.source.SyslogUDPSource{name:syslogudp-source,state:START} }
15 Jan 2014 10:06:56,011 INFO [conf-file-poller-0] (org.apache.flume.source.SyslogUDPSource.stop:111) - Syslog UDP Source stopping...
15 Jan 2014 10:06:56,012 INFO [conf-file-poller-0] (org.apache.flume.source.SyslogUDPSource.stop:112) - Metrics:{ name:null counters:{events.success=2} }
15 Jan 2014 10:06:56,014 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:115) - Stopping Sink hdfs-sink
15 Jan 2014 10:06:56,014 INFO [conf-file-poller-0] (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:171) - Stopping component: SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@5c6ed322 counterGroup:{ name:null counters:{runner.backoffs.consecutive=5, runner.backoffs=39} } }
15 Jan 2014 10:06:56,015 INFO [conf-file-poller-0] (org.apache.flume.sink.hdfs.HDFSEventSink.stop:437) - Closing hdfs://hadoop.int.meetup.com:9000/flume/shared/manual-test/date=2014-01-15/FlumeData
15 Jan 2014 10:06:56,043 INFO [hdfs-hdfs-sink-call-runner-5] (org.apache.flume.sink.hdfs.BucketWriter$7.call:487) - Renaming hdfs://hadoop.int.meetup.com:9000/flume/shared/manual-test/date=2014-01-15/FlumeData.1389798378741.tmp to hdfs://hadoop.int.meetup.com:9000/flume/shared/manual-test/date=2014-01-15/FlumeData.1389798378741
Raw
shared-agent.conf
## SOURCES
# Define a syslogudp source called syslogudp-source on shared-agent and tell it
# to bind to 0.0.0.0:41416. Connect it to channel memory-channel.
shared-agent.sources = syslogudp-source
shared-agent.sources.syslogudp-source.channels = memory-channel poison-channel
shared-agent.sources.syslogudp-source.type = syslogudp
shared-agent.sources.syslogudp-source.bind = 0.0.0.0
shared-agent.sources.syslogudp-source.port = 41416
## INTERCEPTORS
# properties of regex_extractor interceptor called regex_extractor-interceptor for shared-agent
shared-agent.sources.syslogudp-source.interceptors = regex_extractor-interceptor static-interceptor
shared-agent.sources.syslogudp-source.interceptors.regex_extractor-interceptor.type = regex_extractor
shared-agent.sources.syslogudp-source.interceptors.regex_extractor-interceptor.regex = ^(\\S*)\\s.*
shared-agent.sources.syslogudp-source.interceptors.regex_extractor-interceptor.serializers = logtype-serializer
shared-agent.sources.syslogudp-source.interceptors.regex_extractor-interceptor.serializers.logtype-serializer.type = org.apache.flume.interceptor.RegexExtractorInterceptorPassThroughSerializer
shared-agent.sources.syslogudp-source.interceptors.regex_extractor-interceptor.serializers.logtype-serializer.name = logtype
# properties of static interceptor called static-interceptor for shared-agent
shared-agent.sources.syslogudp-source.interceptors.static-interceptor.type = static
shared-agent.sources.syslogudp-source.interceptors.static-interceptor.preserveExisting = true
shared-agent.sources.syslogudp-source.interceptors.static-interceptor.key = timestamp
shared-agent.sources.syslogudp-source.interceptors.static-interceptor.value = UNKNOWN
## SELECTORS
# properties of selector
shared-agent.sources.syslogudp-source.selector.type = multiplexing
shared-agent.sources.syslogudp-source.selector.header = timestamp
shared-agent.sources.syslogudp-source.selector.mapping.UNKNOWN = poison-channel
shared-agent.sources.syslogudp-source.selector.default = memory-channel
## CHANNELS
# properties of memory channel called memory-channel for shared-agent
shared-agent.channels = memory-channel poison-channel
shared-agent.channels.memory-channel.type = memory
shared-agent.channels.memory-channel.capacity = 100000
shared-agent.channels.memory-channel.transactionCapacity = 1000
# properties of memory channel called poison-channel for shared-agent
shared-agent.channels.poison-channel.type = memory
shared-agent.channels.poison-channel.capacity = 10000
shared-agent.channels.poison-channel.transactionCapacity = 1000
## SINKS
# Define a logger sink that simple logs all events it receives
# and connect it to the other end of the same channel.
shared-agent.sinks = hdfs-sink poison-sink
shared-agent.sinks.hdfs-sink.channel = memory-channel
shared-agent.sinks.hdfs-sink.type = hdfs
shared-agent.sinks.hdfs-sink.hdfs.path = hdfs://hadoop.int.meetup.com:9000/flume/shared/%{logtype}/date=%Y-%m-%d
shared-agent.sinks.hdfs-sink.hdfs.fileType = DataStream
shared-agent.sinks.hdfs-sink.hdfs.writeFormat = Text
shared-agent.sinks.hdfs-sink.hdfs.rollSize = 67108864
shared-agent.sinks.hdfs-sink.hdfs.rollCount = 0
shared-agent.sinks.hdfs-sink.hdfs.rollInterval = 0
shared-agent.sinks.hdfs-sink.hdfs.idleTimeout = 3600
shared-agent.sinks.hdfs-sink.hdfs.callTimeout = 100000
# properties for an hdfs sink named poison-sink for shared-agent
shared-agent.sinks.poison-sink.channel = poison-channel
shared-agent.sinks.poison-sink.type = hdfs
shared-agent.sinks.poison-sink.hdfs.path = hdfs://hadoop.int.meetup.com:9000/flume/shared/poison
shared-agent.sinks.poison-sink.hdfs.fileType = DataStream
shared-agent.sinks.poison-sink.hdfs.writeFormat = Text
shared-agent.sinks.poison-sink.hdfs.rollSize = 67108864
shared-agent.sinks.poison-sink.hdfs.rollCount = 0
shared-agent.sinks.poison-sink.hdfs.rollInterval = 0
shared-agent.sinks.poison-sink.hdfs.idleTimeout = 3600
shared-agent.sinks.poison-sink.hdfs.callTimeout = 100000
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment