Skip to content

Instantly share code, notes, and snippets.

@mattyb149
mattyb149 / TellTaleHeartGroovy.xml
Created February 10, 2016 20:29
Apache NiFi template - ExecuteScript (Groovy) Word Count
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>This template is a self-contained example showing how to use ExecuteScript (with Groovy) to perform a classic &quot;Word Count&quot; of an incoming flow file (namely The Tell-Tale Heart by Edgar Allan Poe). The output file consists of lines, each containing a word in the corpus and its frequency of occurrence.</description><name>TellTaleHeartGroovy</name><snippet><connections><id>de8c38dd-af7b-4c23-8480-4b83a11635c7</id><parentGroupId>faeccde3-60c2-47af-86c7-3492795ef534</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>faeccde3-60c2-47af-86c7-3492795ef534</groupId><id>388f347c-7e84-44ef-aaa3-ef42565a7cc6</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>faeccde3-60c2-47af-86c7
@mattyb149
mattyb149 / seed_hazelcast.groovy
Created February 11, 2016 00:10
Groovy script to seed Hazelcast instance for ExecuteScript example
@Grab(group='com.hazelcast', module='hazelcast', version='3.6')
@Grab(group='com.hazelcast', module='hazelcast-client', version='3.6')
import com.hazelcast.client.*
import com.hazelcast.client.config.*
import com.hazelcast.core.*
ClientConfig clientConfig = new ClientConfig();
clientConfig.getGroupConfig().setName("dev").setPassword("dev-pass");
clientConfig.getNetworkConfig().addAddress("192.168.99.100", "192.168.99.100:32780");
@mattyb149
mattyb149 / JeffAustralia.xml
Created February 17, 2016 03:19
An Apache NiFi template to create arbitrary URLs based on incoming file names
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>A sample template to route on filename to build a URL</description><name>JeffAustralia</name><snippet><connections><id>dec6cc8b-14ca-43f2-95ba-17463d587eb2</id><parentGroupId>f813e419-f52e-40c2-8061-79ae1d7d880f</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>f813e419-f52e-40c2-8061-79ae1d7d880f</groupId><id>a6d90e5e-dfe6-4c0a-a558-4283c3bd95f0</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>fileA</selectedRelationships><source><groupId>f813e419-f52e-40c2-8061-79ae1d7d880f</groupId><id>43c814e3-848e-4a04-979e-5f37d3bd5f4b</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>1bd7760e-424d-4c93-84da-e8e0ffb8facc</id><parentGroupId>f813e419-f52e-40c2-8061-79ae1d7d880f</parentGroupId><backPress
@mattyb149
mattyb149 / pr_fixver.groovy
Created March 10, 2016 15:14
A Groovy script to match Apache Jira cases with GitHub pull requests. Defaults to Apache NiFi and uses GITHUB_TOKEN env variable for OAuth
@Grab('org.kohsuke:github-api:1.73')
import org.kohsuke.github.*
def slurper = new groovy.json.JsonSlurper()
def PR_REGEX = /([nN][iI][fF][iI][ -][0-9]+).*/
def g = GitHub.connectUsingOAuth('github.com', System.getenv('GITHUB_TOKEN'))
def pulls = g.getRepository('apache/nifi').getPullRequests(GHIssueState.OPEN)
pulls.each { pr ->
def m = pr.title =~ PR_REGEX
@mattyb149
mattyb149 / IterateJsonArray.xml
Created March 17, 2016 20:34
A NiFi template that iterates over child array elements in a JSON file, and filters on a specific property
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>This template uses SplitJson to split a JSON file by child array elements, then routes on certain values of the child's &quot;phone&quot; value.</description><name>IterateJsonArray</name><snippet><processGroups><id>b45c83bf-615d-4a54-8b92-cc4b9ccb730d</id><parentGroupId>624b3e60-03e2-439f-8e68-9cdb2fc1e404</parentGroupId><position><x>-85.208424518031</x><y>-104.95458750260781</y></position><activeRemotePortCount>0</activeRemotePortCount><comments></comments><contents><connections><id>afffc4df-44e5-4ef9-b188-cefc622afc07</id><parentGroupId>b45c83bf-615d-4a54-8b92-cc4b9ccb730d</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>b45c83bf-615d-4a54-8b92-cc4b9ccb730d</groupId><id>4e82ff05-5ad6-43a8-a2ce-270c0fc1bc9a</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><
@mattyb149
mattyb149 / ParseJsonInAttribute.xml
Created March 21, 2016 17:52
A NiFi template that uses Groovy to parse an attribute containing JSON, and creating a new attribute from one of the JSON fields
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description></description><name>ParseJsonInAttribute</name><snippet><connections><id>906cd6cd-343f-45d6-8e08-6306ef089773</id><parentGroupId>08ae6f8e-de9b-4014-ab98-2c415b5e9d2a</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>08ae6f8e-de9b-4014-ab98-2c415b5e9d2a</groupId><id>3382f8b0-d0b0-41c6-bca7-8bd546a6be82</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>08ae6f8e-de9b-4014-ab98-2c415b5e9d2a</groupId><id>77b91aad-5389-4bdf-876b-5015faccfe9c</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>5b1376bf-e6d6-456c-9e83-444f554b25d8</id><parentGroupId>08ae6f8e-de9b-4014-ab98-2c415b5e9d2a</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataS
@mattyb149
mattyb149 / TestConvertAvroToOrc2.xml
Created May 31, 2016 02:03
A test template for the ConvertAvroToORC processor in Apache NiFi
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description></description><name>TestConvertAvroToOrc2</name><snippet><connections><id>02fe77fc-2c72-461d-ba76-cd7ccfe4c0ec</id><parentGroupId>9007185c-8178-4606-81a5-b6a4e930ec8b</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>9007185c-8178-4606-81a5-b6a4e930ec8b</groupId><id>17ad09e6-4c2e-4740-a046-f02433fbedc9</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>9007185c-8178-4606-81a5-b6a4e930ec8b</groupId><id>4a2f1555-737d-4d30-a717-75f8cbdb4d61</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>eba4991d-881b-4c1c-a4bc-5cc25b02e79d</id><parentGroupId>9007185c-8178-4606-81a5-b6a4e930ec8b</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureData
@mattyb149
mattyb149 / JsonToCQL.xml
Created July 7, 2016 15:37
NiFi template that takes JSON input, routes on a value, then converts to a Cassandra Query Language (CQL) statement
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template encoding-version="1.0"><description></description><groupId>6399b847-98bb-4646-baa1-65523b25a7f1</groupId><name>JsonToCQL</name><snippet><connections><id>945cd514-84c9-4d54-8335-315d4dfd79e7</id><parentGroupId>6399b847-98bb-4646-baa1-65523b25a7f1</parentGroupId><backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold><backPressureObjectThreshold>10000</backPressureObjectThreshold><destination><groupId>6399b847-98bb-4646-baa1-65523b25a7f1</groupId><id>28324be6-b7ca-4a88-95d4-e7eff0e62eb2</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>6399b847-98bb-4646-baa1-65523b25a7f1</groupId><id>3293abf1-1b9d-4973-b51d-4f2e6899f18d</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>a795d09a-5222-41b7-a155-d899f9254f2e</id><parentGroupId>6399b847-98bb-4646-baa1-65523b25a7f
@mattyb149
mattyb149 / ScriptedHazelcastClient.xml
Created February 11, 2016 00:31
ExecuteScript example for Hazelcast client in JS and Groovy
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>This template uses Hazelcast JARs in ExecuteScript processors (both in Groovy and Javascript) to generate a flow file from a Hazelcast map.</description><name>ScriptedHazelcastClient</name><snippet><connections><id>840610c3-ce4a-413d-8f54-40ea0a1f1f90</id><parentGroupId>d657baec-19f5-4689-ab5f-f72604055136</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>d657baec-19f5-4689-ab5f-f72604055136</groupId><id>91c52bb1-ba4d-4bce-9d41-e49dd4f03851</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>d657baec-19f5-4689-ab5f-f72604055136</groupId><id>87256841-565d-4559-bb6b-72fd9d6d014b</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>f6c2e6aa-f4f7-4a80-9
@mattyb149
mattyb149 / ExecuteScriptRemoteCommands.xml
Created August 10, 2016 19:18
An Apache NiFi template using ExecuteScript with Groovy and Sshoogr to execute commands on a remote node
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description>This template shows how to use ExecuteScript with Groovy and Sshoogr to execute commands (from an incoming flow file) on a remote system (via SSH).
To use, replace the values in ExecuteScript for your remote system (sshHostname, sshUsername, sshPassword, sshPort), and replace the GenerateFlowFile processor (and the associated ReplaceText processor) with your flow that produces files with lines of commands to run remotely.</description>
<groupId>71f9bd06-0156-1000-783e-41e29ccff3d8</groupId>
<name>ExecuteScriptRemoteCommands</name>
<snippet>
<connections>
<id>7492dda1-0156-1000-0000-000000000000</id>