Skip to content

Instantly share code, notes, and snippets.

@mattyb149
mattyb149 / ExtractRecordFields.json
Created September 7, 2023 19:28
NiFi 1.23+ Flow Definition to extract field names from schema
{
"flowContents": {
"identifier": "a226434e-78d6-3846-bb1f-55680679a3b3",
"instanceIdentifier": "710e0519-018a-1000-072b-2e0e390412d3",
"name": "NiFi Flow",
"comments": "",
"position": {
"x": 0.0,
"y": 0.0
},
@mattyb149
mattyb149 / SequenceLookupExample.xml
Created August 22, 2018 15:14
A NiFi template with a ScriptedLookupService that retrieves a sequence number from an external database
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.2">
<description>This template gives an example of using a ScriptedLookupService to provide sequence numbers from a database.</description>
<groupId>5d7e55f8-0165-1000-1170-d42741815ddb</groupId>
<name>SequenceLookupExample</name>
<snippet>
<connections>
<id>0cf6aae2-8e5a-361f-0000-000000000000</id>
<parentGroupId>cedd92a0-903e-3f68-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
@mattyb149
mattyb149 / SplitRecord_w_Conversion.xml
Created July 11, 2018 16:23
NiFi SplitRecord example that converts CSV to Avro while splitting files
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.2">
<description></description>
<groupId>89f8c6b8-0164-1000-d937-82c55b6fdffd</groupId>
<name>SplitRecord_w_Conversion</name>
<snippet>
<controllerServices>
<id>350009ab-3f44-32bf-0000-000000000000</id>
<parentGroupId>fa0118f1-ff03-3030-0000-000000000000</parentGroupId>
<bundle>
@mattyb149
mattyb149 / ISP_template.py
Created November 7, 2017 17:50
An InvokeScriptedProcessor template (in Jython) for running ExecuteScript Jython scripts faster
#////////////////////////////////////////////////////////////
#// imports go here
#////////////////////////////////////////////////////////////
from org.apache.nifi.processor import Processor,Relationship
from java.lang import Throwable
class E():
def __init__(self):
pass
def executeScript(self,session, context, log, REL_SUCCESS, REL_FAILURE):
@mattyb149
mattyb149 / testEL.groovy
Created October 25, 2017 21:27
Groovy script for testing NiFi Expression Language expressions
@Grab(group='org.apache.nifi', module='nifi-expression-language', version='1.4.0')
import org.apache.nifi.attribute.expression.language.*
def cli = new CliBuilder(usage:'groovy testEL.groovy [options] [expressions]',
header:'Options:')
cli.help('print this message')
cli.D(args:2, valueSeparator:'=', argName:'attribute=value',
'set value for given attribute')
def options = cli.parse(args)
if(!options.arguments()) {
@mattyb149
mattyb149 / csv_to_avroschema.groovy
Created May 19, 2017 19:04
A Groovy script for NiFi ExecuteScript to extract the schema from the header line of a CSV file
import groovy.json.*
def flowFile = session.get()
if(!flowFile) return
def delim = ','
try {
delim = delimiter?.value ?: ','
} catch (MissingPropertyException mpe) { }
@mattyb149
mattyb149 / ScriptedReaderTest.xml
Created April 25, 2017 15:57
A NiFi 1.2.0 template with sample data and script to illustrate the use of ScriptedReader
<?xml version="1.0" ?>
<template encoding-version="1.1">
<description></description>
<groupId>a5a9dae4-015b-1000-7a1c-eaf2db85b188</groupId>
<name>ScriptedReaderTest</name>
<snippet>
<controllerServices>
<id>bb8e2ece-1712-3f89-0000-000000000000</id>
<parentGroupId>f3e4227c-c306-3c57-0000-000000000000</parentGroupId>
<bundle>
@mattyb149
mattyb149 / ConvertRecord_examples.xml
Created April 18, 2017 14:43
NiFi template with various tests / exercises of the ConvertRecord processor
<?xml version="1.0" ?>
<template encoding-version="1.1">
<description></description>
<groupId>7ebaf566-015b-1000-b416-922a420ea741</groupId>
<name>ConvertRecord_examples</name>
<snippet>
<connections>
<id>81dd9a51-07c4-39f0-0000-000000000000</id>
<parentGroupId>868f7eec-2b0e-352b-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
@mattyb149
mattyb149 / CDC_MySQL_Replication.xml
Created April 17, 2017 20:33
NiFi template with end-to-end table replication from a source MySQL database to a target MySQL database
<?xml version="1.0" ?>
<template encoding-version="1.1">
<description>This template shows an end-to-end example of how to replicate table(s)
from a source MySQL database to a target MySQL database</description>
<groupId>68961054-015b-1000-2c21-01b42e84dbe0</groupId>
<name>CDC_MySQL_Replication</name>
<snippet>
<connections>
<id>9a0f0bca-5c0c-3f1e-0000-000000000000</id>
<parentGroupId>23d80cd0-6e6e-31c9-0000-000000000000</parentGroupId>
@mattyb149
mattyb149 / EnforceOrder_Examples.xml
Created April 7, 2017 13:24
A NiFi Template with 2 example use cases for the EnforceOrder processor
<?xml version="1.0" ?>
<template encoding-version="1.1">
<description>This template contains two use cases for the EnforceOrder processor:
1) Random Delay: FlowFiles are routed to two separate paths and delayed by a random amount of time (0-100 millis)
2) MultipleSplits: FlowFiles are split twice with SplitText, the attributes must be updated to keep the original order
</description>
<groupId>48313cf9-015b-1000-cf31-bc9709ad2638</groupId>
<name>EnforceOrder_Examples</name>
<snippet>