flume demo

#Twitter drill demo


Install flume:

	yum install -y mapr-flume
	
	cd /opt/mapr/flume/flume-1.5.0/conf
	
	cp flume-env.sh.template flume-env.sh
	
	yum install -y git
	
	cd /mapr/clustername
	
	mkdir apps
	
	mkdir twitter
	
	cd apps
	
	
	git clone https://github.com/abengelbrecht/drill-twitter
	
	cp drill-twitter/flume/flume.conf /opt/mapr/flume/flume-1.5.0/conf/flume.conf
	
Modify `/opt/mapr/flume/flume-1.5.0/conf/flume.conf` as follows:

modify `TwitterAgent.sources.Twitter.keywords` , they are case sensitive, and comma separated.  you can use spaces (e.g.: MapR Technologies)

add this line: 

	TwitterAgent.sinks.HDFS.hdfs.fileSuffix = .json

save/quit	

install screen:

	yum install -y screen

initialize a screen session so that flume can run even if you logoff:

	screen -R flume
Launch flume:

	/opt/mapr/flume/flume-1.5.0/bin/flume-ng agent -f conf/flume.conf -C /opt/mapr/flume/flume-1.5.0/lib/flume-sources-twitter-json-0.1.jar -Dflume.root.logger=DEBUG,console -n TwitterAgent


it'll take some time for flume to build things up..best to let it run for several hours or overnight.