Skip to content

Instantly share code, notes, and snippets.

View alopresto's full-sized avatar
🔏
Focusing on NiFi security...

Andy LoPresto alopresto

🔏
Focusing on NiFi security...
72 followers · 31 following
View GitHub Profile
@alopresto
alopresto / Output
Created September 28, 2016 00:11
The command line input and output of the Maven packaging and script invocation
hw12203:/Users/alopresto/Workspace/scratch/process-test (master) alopresto
🔒 142s @ 17:09:54 $ ll
total 40
drwxr-xr-x 9 alopresto staff 306B Sep 27 17:07 ./
drwxr-xr-x 48 alopresto staff 1.6K Sep 27 16:49 ../
-rw-r--r--@ 1 alopresto staff 6.0K Sep 27 17:06 .DS_Store
drwxr-xr-x 11 alopresto staff 374B Sep 27 17:09 .idea/
-rw-r--r-- 1 alopresto staff 642B Sep 27 16:56 pom.xml
-rw-r--r-- 1 alopresto staff 1.0K Sep 27 17:03 process-test-java.iml
-rwxr-xr-x 1 alopresto staff 82B Sep 27 17:07 script.sh*
@alopresto
alopresto / script.sh
Last active September 28, 2016 00:22
A bash script which invokes the main method of the Process class in a jar and passes the arguments provided to the script.
echo "Bash before Java invocation '$@'"
java -cp /Users/alopresto/Workspace/scratch/process-test/target/process-test-1.0-SNAPSHOT.jar org.apache.nifi.process.Process "$@"
echo "Bash after Java invocation '$@'"
@alopresto
alopresto / Output.log
Created September 28, 2016 00:27
Excerpt from `logs/nifi-app.log` showing the flowfile attributes and content including bash `echo` output
2016-09-27 17:23:13,970 INFO [StandardProcessScheduler Thread-1] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled ExecuteProcess[id=6e0d4685-0157-1000-1616-34e3ea888ad6] to run with 1 threads
2016-09-27 17:23:13,974 INFO [StandardProcessScheduler Thread-7] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled LogAttribute[id=6e22ae1c-0157-1000-96e3-9be4680eb151] to run with 1 threads
2016-09-27 17:23:14,340 INFO [Timer-Driven Process Thread-1] o.a.n.processors.standard.LogAttribute LogAttribute[id=6e22ae1c-0157-1000-96e3-9be4680eb151] logging for flow file StandardFlowFileRecord[uuid=9442ca93-3f3c-40c9-a23f-6b2e6707eba6,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1475019971713-1, container=default, section=1], offset=1751, length=149],offset=0,name=70522280527566,size=149]
--------------------------------------------------
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Tue Sep 27 17:23:14 PDT 2016'
Key: 'lineageStartDate'
Value: 'Tue Sep 27 17:23:14 PDT 2016'
Key: 'fileSize'
@alopresto
alopresto / execute_and_print.xml
Created September 28, 2016 00:30
NiFi template containing a `ExecuteProcess` processor connected to a `LogAttribute` processor.
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description></description>
<groupId>459549da-0157-1000-e381-c7ba87f8ff61</groupId>
<name>ExecuteProcess and print output</name>
<snippet>
<connections>
<id>6e22e0ec-0157-1000-0000-000000000000</id>
<parentGroupId>459549da-0157-1000-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
@alopresto
alopresto / Output.log
Created September 28, 2016 00:31
Excerpt of `logs/nifi-app.log` showing the output of a Java program invoked via bash script.
2016-09-27 17:17:59,973 INFO [Timer-Driven Process Thread-3] o.a.n.processors.standard.LogAttribute LogAttribute[id=6e22ae1c-0157-1000-96e3-9be4680eb151] logging for flow file StandardFlowFileRecord[uuid=9748ab80-3c9a-4598-be9d-4dc297975ab3,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1475019971713-1, container=default, section=1], offset=1595, length=78],offset=0,name=70207909674263,size=78]
--------------------------------------------------
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Tue Sep 27 17:17:59 PDT 2016'
Key: 'lineageStartDate'
Value: 'Tue Sep 27 17:17:59 PDT 2016'
Key: 'fileSize'
Value: '78'
FlowFile Attribute Map Content
@alopresto
alopresto / expression_language_boolean.xml
Created October 7, 2016 18:39
NiFi template: Generates an empty flowfile, uses a Groovy script to randomly assign either "test" or "prod" to the "db.table.name" attribute, updates the "expression" attribute with the (String) result of a boolean comparison of "db.table.name" to "test", and routes to "test" or "prod" based on the boolean evaluation of "expression".
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description>Generates an empty flowfile, uses a Groovy script to randomly assign either "test" or "prod" to the "db.table.name" attribute, updates the "expression" attribute with the (String) result of a boolean comparison of "db.table.name" to "test", and routes to "test" or "prod" based on the boolean evaluation of "expression". </description>
<groupId>0157100e-ae1c-1e22-9d02-c3092b285655</groupId>
<name>Expression Language Boolean Evaluation</name>
<snippet>
<connections>
<id>01571004-ae1c-1e22-0000-000000000000</id>
<parentGroupId>0157100e-ae1c-1e22-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
@alopresto
alopresto / extract_text_regex.xml
Created October 10, 2016 19:21
Apache NiFi template which generates an empty flowfile, populates the contents with an example log HTTP request line, and extracts the HTTP response code.
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description>Generates an empty flowfile, populates the contents with an example log HTTP request line, and extracts the HTTP response code. </description>
<groupId>b0049ce3-0157-1000-671f-21b7e01c3bcf</groupId>
<name>ExtractText Regex</name>
<snippet>
<connections>
<id>b0057ecb-0157-1000-0000-000000000000</id>
<parentGroupId>b0049ce3-0157-1000-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
@alopresto
alopresto / header_and_footer.xml
Last active October 25, 2016 17:51
Apache NiFi template to demonstrate `RouteOnContent` capabilities.
@alopresto
alopresto / Encrypt-attribute-via-execute-script.xml
Last active October 27, 2016 23:37
Apache NiFi template that generates an empty flowfile, populates the content with plaintext, adds two attributes, uses an ExecuteScript processor to perform AES/GCM encryption with a default key, and updates and adds attributes with the cipher text results, then logs the attributes and content of the flowfile.
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description>This template generates an empty flowfile, populates the content with plaintext, adds two attributes, uses an ExecuteScript processor to perform AES/GCM encryption with a default key, and updates and adds attributes with the cipher text results, then logs the attributes and content of the flowfile. </description>
<groupId>08499ab1-0158-1000-eb96-b892bf593753</groupId>
<name>EncryptAttribute (via ExecuteScript)</name>
<snippet>
<connections>
<id>084be0c5-0158-1000-0000-000000000000</id>
<parentGroupId>08499ab1-0158-1000-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
@alopresto
alopresto / Encrypt-and-decrypt-attribute-processor-test-template.xml
Created December 6, 2016 04:39
Apache NiFi template to test the `EncryptAttributes` processor submitted in [PR 1294](https://github.com/apache/nifi/pull/1294).
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description>Must set the password and key for various encrypt/decrypt processors. </description>
<groupId>d07d0681-0158-1000-1581-15fb569a9418</groupId>
<name>Encrypt and Decrypt Attributes</name>
<snippet>
<connections>
<id>d0813dba-0158-1000-0000-000000000000</id>
<parentGroupId>d07d0681-0158-1000-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>