Skip to content

Instantly share code, notes, and snippets.

View mattyb149's full-sized avatar

Matt Burgess mattyb149

92 followers · 4 following
View GitHub Profile
@mattyb149
mattyb149 / JsonToCQL.xml
Created July 7, 2016 15:37
NiFi template that takes JSON input, routes on a value, then converts to a Cassandra Query Language (CQL) statement
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template encoding-version="1.0"><description></description><groupId>6399b847-98bb-4646-baa1-65523b25a7f1</groupId><name>JsonToCQL</name><snippet><connections><id>945cd514-84c9-4d54-8335-315d4dfd79e7</id><parentGroupId>6399b847-98bb-4646-baa1-65523b25a7f1</parentGroupId><backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold><backPressureObjectThreshold>10000</backPressureObjectThreshold><destination><groupId>6399b847-98bb-4646-baa1-65523b25a7f1</groupId><id>28324be6-b7ca-4a88-95d4-e7eff0e62eb2</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>6399b847-98bb-4646-baa1-65523b25a7f1</groupId><id>3293abf1-1b9d-4973-b51d-4f2e6899f18d</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>a795d09a-5222-41b7-a155-d899f9254f2e</id><parentGroupId>6399b847-98bb-4646-baa1-65523b25a7f
@mattyb149
mattyb149 / TestConvertAvroToOrc2.xml
Created May 31, 2016 02:03
A test template for the ConvertAvroToORC processor in Apache NiFi
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description></description><name>TestConvertAvroToOrc2</name><snippet><connections><id>02fe77fc-2c72-461d-ba76-cd7ccfe4c0ec</id><parentGroupId>9007185c-8178-4606-81a5-b6a4e930ec8b</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>9007185c-8178-4606-81a5-b6a4e930ec8b</groupId><id>17ad09e6-4c2e-4740-a046-f02433fbedc9</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>9007185c-8178-4606-81a5-b6a4e930ec8b</groupId><id>4a2f1555-737d-4d30-a717-75f8cbdb4d61</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>eba4991d-881b-4c1c-a4bc-5cc25b02e79d</id><parentGroupId>9007185c-8178-4606-81a5-b6a4e930ec8b</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureData
@mattyb149
mattyb149 / SQL-to-CSV_ExecuteScript.xml
Created April 8, 2016 18:47
A template for Apache NiFi that uses ExecuteScript with Groovy to issue a SQL query and produce a flowfile containing a CSV representation of the results
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>A template that uses ExecuteScript with Groovy to issue a SQL query and produce a flowfile containing a CSV representation of the results</description><name>SQL-to-CSV_ExecuteScript</name><snippet><processors><id>26925087-23d2-48c9-86d2-6bdf7e350167</id><parentGroupId>3b5fc5a3-8ae5-4e68-9911-a46c8f9c59c3</parentGroupId><position><x>1398.5380462507696</x><y>72.79047858547945</y></position><config><bulletinLevel>WARN</bulletinLevel><comments></comments><concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount><defaultConcurrentTasks><entry><key>TIMER_DRIVEN</key><value>1</value></entry><entry><key>EVENT_DRIVEN</key><value>0</value></entry><entry><key>CRON_DRIVEN</key><value>1</value></entry></defaultConcurrentTasks><defaultSchedulingPeriod><entry><key>TIMER_DRIVEN</key><value>0 sec</value></entry><entry><key>CRON_DRIVEN</key><value>* * * * * ?</value></entry></defaultSchedulingPeriod><descriptors><entry><key>Script Engine<
@mattyb149
mattyb149 / prov2graphson_TP3.groovy
Created March 31, 2016 12:28
A Groovy script to take NiFi provenance events and transform them into Tinkerpop3-compliant GraphSON
import groovy.json.*
import java.util.UUID
def json = new JsonSlurper().parseText("file:///Users/mburgess/git/nifi-client/prov.json".toURL().text)
def links = json.links.collect { ['outV': it.targetId, 'inV': it.sourceId] }
println links
new File('prov_TP3.json').withWriter { out ->
def builder = new JsonBuilder()
@mattyb149
mattyb149 / ParseJsonInAttribute.xml
Created March 21, 2016 17:52
A NiFi template that uses Groovy to parse an attribute containing JSON, and creating a new attribute from one of the JSON fields
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description></description><name>ParseJsonInAttribute</name><snippet><connections><id>906cd6cd-343f-45d6-8e08-6306ef089773</id><parentGroupId>08ae6f8e-de9b-4014-ab98-2c415b5e9d2a</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>08ae6f8e-de9b-4014-ab98-2c415b5e9d2a</groupId><id>3382f8b0-d0b0-41c6-bca7-8bd546a6be82</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>08ae6f8e-de9b-4014-ab98-2c415b5e9d2a</groupId><id>77b91aad-5389-4bdf-876b-5015faccfe9c</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>5b1376bf-e6d6-456c-9e83-444f554b25d8</id><parentGroupId>08ae6f8e-de9b-4014-ab98-2c415b5e9d2a</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataS
@mattyb149
mattyb149 / IterateJsonArray.xml
Created March 17, 2016 20:34
A NiFi template that iterates over child array elements in a JSON file, and filters on a specific property
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>This template uses SplitJson to split a JSON file by child array elements, then routes on certain values of the child's &quot;phone&quot; value.</description><name>IterateJsonArray</name><snippet><processGroups><id>b45c83bf-615d-4a54-8b92-cc4b9ccb730d</id><parentGroupId>624b3e60-03e2-439f-8e68-9cdb2fc1e404</parentGroupId><position><x>-85.208424518031</x><y>-104.95458750260781</y></position><activeRemotePortCount>0</activeRemotePortCount><comments></comments><contents><connections><id>afffc4df-44e5-4ef9-b188-cefc622afc07</id><parentGroupId>b45c83bf-615d-4a54-8b92-cc4b9ccb730d</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>b45c83bf-615d-4a54-8b92-cc4b9ccb730d</groupId><id>4e82ff05-5ad6-43a8-a2ce-270c0fc1bc9a</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><
@mattyb149
mattyb149 / JythonJsonToJsonExample.xml
Created March 14, 2016 16:25
An Apache NiFi template that uses ExecuteScript (with Jython) to perform JSON-to-JSON conversion
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description></description><name>JythonJsonToJsonExample</name><snippet><connections><id>02d12142-26da-459b-938c-427d4f2e44cc</id><parentGroupId>68f8be3e-335a-457d-b83f-96ddaae929e6</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>68f8be3e-335a-457d-b83f-96ddaae929e6</groupId><id>60403007-026d-40fe-ba2e-d3f48b6d235f</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>68f8be3e-335a-457d-b83f-96ddaae929e6</groupId><id>45c28ea4-b46f-46d3-93f2-3dac1ffadcff</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>3aa46389-b79d-42b4-93f3-d494ce7338da</id><parentGroupId>68f8be3e-335a-457d-b83f-96ddaae929e6</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDa
@mattyb149
mattyb149 / pr_fixver.groovy
Created March 10, 2016 15:14
A Groovy script to match Apache Jira cases with GitHub pull requests. Defaults to Apache NiFi and uses GITHUB_TOKEN env variable for OAuth
@Grab('org.kohsuke:github-api:1.73')
import org.kohsuke.github.*
def slurper = new groovy.json.JsonSlurper()
def PR_REGEX = /([nN][iI][fF][iI][ -][0-9]+).*/
def g = GitHub.connectUsingOAuth('github.com', System.getenv('GITHUB_TOKEN'))
def pulls = g.getRepository('apache/nifi').getPullRequests(GHIssueState.OPEN)
pulls.each { pr ->
def m = pr.title =~ PR_REGEX
@mattyb149
mattyb149 / TestInvokeHttpPOST.xml
Created March 2, 2016 21:55
A NiFi template to test InvokeHttp's POST capability, sends JSON to a test endpoint and writes the response to a file
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>This template posts JSON to a test endpoint (http://httpbin.org/post) and writes the response to a file</description><name>TestInvokeHttpPOST</name><snippet><connections><id>db33e68a-c19f-4059-944e-56229ccc550b</id><parentGroupId>60dbaa71-2453-4f82-bafe-6087462ecc47</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>60dbaa71-2453-4f82-bafe-6087462ecc47</groupId><id>fe102472-7dbc-4719-9ef1-266992646565</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>60dbaa71-2453-4f82-bafe-6087462ecc47</groupId><id>46869aaa-0232-4a8b-9f53-7bd221fdf8b8</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>d1be0a90-92b0-4320-956f-decd9df513b1</id><parentGroupId>60dba
@mattyb149
mattyb149 / JavascriptJsonToJsonExample.xml
Created March 2, 2016 21:01
An Apache NiFi template that uses ExecuteScript (with Javascript) to perform JSON-to-JSON conversion
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>This template is an example of the ExecuteScript (using Javascript) that transforms an incoming JSON object to one with a new schema/values. It is based on the JOLT demo at http://jolt-demo.appspot.com/</description><name>JavascriptJsonToJsonExample</name><snippet><connections><id>a0eac3a6-591d-4978-888b-d2c489b1d740</id><parentGroupId>60dbaa71-2453-4f82-bafe-6087462ecc47</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>60dbaa71-2453-4f82-bafe-6087462ecc47</groupId><id>89e89d33-aea2-4777-a32c-473ee6ac9138</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>60dbaa71-2453-4f82-bafe-6087462ecc47</groupId><id>f358345b-ba2d-42da-ab3d-4fc76877438d</id><type>PROCESSOR</type></source><z