This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// Factories for all Operations (Functions, Filters, Aggregators, and Buffers) | |
Function splitter = Fluid.function() | |
.RegexSplitter() | |
.fieldDeclaration( fields( "num", "char" ) ) | |
.patternString( " " ) | |
.end(); | |
// An assembly builder chaining Pipes into complex assemblies | |
AssemblyBuilder.Start assembly = Fluid.assembly(); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# change the yarn bucket name | |
OPTIONS="--bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop" | |
OPTIONS="${OPTIONS} --args -y,yarn.log-aggregation-enable=true,-y,yarn.log-aggregation.retain-seconds=-1,-y,yarn.log-aggregation.retain-check-interval-seconds=3000,-y,yarn.nodemanager.remote-app-log-dir=s3://your-bucket/emr/yarn-logs" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
if (!groups.isEmpty()) { | |
// lets see if we can filter based on groups | |
List<String> groupNames = reservation.getGroupNames(); | |
if (Collections.disjoint(groups, groupNames)) { | |
logger.trace("filtering out reservation {} based on groups {}, not part of {}", reservation.getReservationId(), groupNames, groups); | |
// continue to the next reservation | |
continue; | |
} | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
if (!groups.isEmpty()) { | |
// lets see if we can filter based on groups | |
boolean filter = true; | |
for (String group : reservation.getGroupNames()) { | |
if (groups.contains(group)) { | |
filter = false; | |
break; | |
} | |
} | |
if (filter) { |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
public class WritableSequenceFile extends SequenceFile | |
{ | |
protected Class<? extends Writable> writableType; | |
public WritableSequenceFile( Class<? extends Writable> writableType, Fields fields ) | |
{ | |
super( fields ); | |
this.writableType = writableType; | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
public void testComplexLogic() throws Exception | |
{ | |
if( !new File( inputFileLhs ).exists() ) | |
fail( "data file not found" ); | |
copyFromLocal( inputFileLhs ); | |
Tap source = new Hfs( new TextDelimited( new Fields( "num", "char" ), " " ), inputFileLhs ); | |
Pipe pipe = new Pipe( "test" ); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Map<Object, Object> properties = getProperties(); | |
properties = appProps() | |
.setName( appName ) | |
.setVersion( appVersion ) | |
.addTag( stamp ) | |
.addTag( appName ) | |
.addTag( "app-tag" ) | |
.build( properties ); // or buildProperties() or ? |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@Test | |
public void testSameSourceMergeThreeChainGroup() throws Exception | |
{ | |
getPlatform().copyFromLocal( inputFileLower ); | |
Tap sourceLower = getPlatform().getTextFile( inputFileLower ); | |
Map sources = new HashMap(); | |
sources.put( "split", sourceLower ); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
curl -s -XPUT localhost:9200/_template/template_tags -d '{ | |
"template" : "tags*", | |
"settings" : { | |
"index.analysis.analyzer.csv.type" : "pattern", | |
"index.analysis.analyzer.csv.pattern" : "," | |
}, | |
"mappings" : { | |
"_default_" : { |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
String statement = "select *\n" | |
+ "from \"example\".\"sales_fact_1997\" as s\n" | |
+ "join \"example\".\"employee\" as e\n" | |
+ "on e.\"EMPID\" = s.\"CUST_ID\""; | |
Tap empTap = getPlatform().getDelimitedFile( ",", "\"", new SQLTypeResolver(), DATA_EMPLOYEE, SinkMode.KEEP ); | |
Tap salesTap = getPlatform().getDelimitedFile( ",", "\"", new SQLTypeResolver(), DATA_SALESFACT, SinkMode.KEEP ); | |
Tap resultsTap = getPlatform().getDelimitedFile( ",", "\"", new SQLTypeResolver(), getOutputPath( "dynamic" ), SinkMode.REPLACE ); |
OlderNewer