Skip to content

Instantly share code, notes, and snippets.

@PierreZ
Last active August 29, 2015 13:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save PierreZ/9531590 to your computer and use it in GitHub Desktop.
Save PierreZ/9531590 to your computer and use it in GitHub Desktop.
First attempt to push logs stored on Hadoop into ElasticSearch
/*
Made by Pierre Zemb
Part-time internship at System Team
Credit Mutuel Arkea
2014
*/
REGISTER elasticsearch-hadoop-1.3.0.M2-yarn.jar;
DEFINE EsStorage org.elasticsearch.hadoop.pig.EsStorage();
YUMLOG = load '/hdfs/data/Hadoop2ES/sys.log' using PigStorage(' ') AS (jour:chararray, day:int, time:chararray, host:chararray, app:chararray, msg:chararray);
DUMP YUMLOG;
STORE YUMLOG INTO 'hadooplogs/logsevent' USING EsStorage(
'es.http.timeout = 5m',
'es.http.retries = 10',
'es.http.timeout= 5m',
'es.nodes = ip:port'
);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment