Skip to content

Instantly share code, notes, and snippets.

View alopresto's full-sized avatar
🔏
Focusing on NiFi security...

Andy LoPresto alopresto

🔏
Focusing on NiFi security...
View GitHub Profile
@alopresto
alopresto / Log_output_from_InvokeHTTP.txt
Created June 19, 2019 02:58
Excerpted and annotated log output from NiFi InvokeHTTP.
# This is the LogAttribute in the "HandleHTTPRequest/UpdateAttribute/HandleHTTPResponse" flow (on the right). Please note that the flowfile content is present (the JSON snippet with id and message keys), but the attribute1 and authorization values are not present. They were not sent via HTTP request.
2019-06-18 19:46:50,750 INFO [Timer-Driven Process Thread-5] o.a.n.processors.standard.LogAttribute LogAttribute[id=6d9bac97-016b-1000-1aab-863142f46b52] logging for flow file StandardFlowFileRecord[uuid=b2c07e72-6f23-41b2-9910-78a71d590b54,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1560912333372-1, container=default, section=1], offset=251, length=56],offset=0,name=b2c07e72-6f23-41b2-9910-78a71d590b54,size=56]
--------------------------------------------------
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Tue Jun 18 19:46:50 PDT 2019'
Key: 'lineageStartDate'
Value: 'Tue Jun 18 19:46:50 PDT 2019'
Key: 'fileSize'
@alopresto
alopresto / InvokeHTTP_flowfile_attributes.xml
Created June 19, 2019 02:57
This template has a flow which generates a flowfile with attributes and sends the content via InvokeHTTP. The endpoint is a temporary web server handled by the complementary flow snippet on the right which receives a flowfile, logs it, and returns 200.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.2">
<description>This template has a flow which generates a flowfile with attributes and sends the content via InvokeHTTP. The endpoint is a temporary web server handled by the complementary flow snippet on the right which receives a flowfile, logs it, and returns 200. </description>
<groupId>ea4b02a5-0169-1000-8686-6aa24c08df76</groupId>
<name>Template to demonstrate InvokeHTTP handling of flowfile attributes</name>
<snippet>
<connections>
<id>1464960d-70e0-338f-0000-000000000000</id>
<parentGroupId>faa3f835-56e6-39a0-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
@alopresto
alopresto / la_os_meetup.txt
Created April 6, 2019 21:31
Welcome to LA Open Source meetup
Hi folks,
I’m hoping to revitalize this group in the coming months. I’ll be organizing events which focus on open source technologies, and bringing contributors to speak and teach.
I also want to encourage people who have an interest and have never presented or want to be more comfortable to get involved. The only way to be good at it is to start somewhere. My expectation is to have some events with 2 40-minute “expert” sessions and then 3-4 “lightning talks” of 5-10 minutes each to allow for new people or people exploring a new topic to share with the group.
This is our group, so please let me know what topics you’re interested in, or if you are interested in contributing or presenting, and I’ll try to organize them. I’m also gathering sponsors to provide food/drinks/locations, so if you are able to offer those as well, they are very welcome.
Looking forward to meeting you all.
@alopresto
alopresto / custom.py
Created March 25, 2019 20:21
Python script to display two lines of different-colored text on Inky pHAT.
from inky import InkyPHAT
inky_display = InkyPHAT("red")
inky_display.set_border(inky_display.WHITE)
from PIL import Image, ImageFont, ImageDraw
img = Image.new("P", (inky_display.WIDTH, inky_display.HEIGHT))
draw = ImageDraw.Draw(img)
@alopresto
alopresto / display_sense_hat.py
Created March 25, 2019 20:20
Python script to display Sense HAT readings on Inky pHAT.
from inky import InkyPHAT
inky_display = InkyPHAT("red")
inky_display.set_border(inky_display.WHITE)
#inky_display.set_rotation(180)
inky_display.v_flip=True
inky_display.h_flip=True
#inky_display.rotation=2
from PIL import Image, ImageFont, ImageDraw
@alopresto
alopresto / evaluate_json.xml
Created February 20, 2019 04:10
Simple Apache NiFi template which generates JSON and extracts one value.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.2">
<description>Simple template which generates JSON and extracts one value. </description>
<groupId>0909b1cf-0169-1000-df6a-0cf3ed63a667</groupId>
<name>EvaluateJsonPath</name>
<snippet>
<processors>
<id>6a40aed0-a311-362c-0000-000000000000</id>
<parentGroupId>21d7c043-e2c1-3a78-0000-000000000000</parentGroupId>
<position>
@alopresto
alopresto / DIM-Security-Eng-2019.02.16
Created February 16, 2019 22:33
Cloudera Data In Motion Security Engineer posting
Job Description: 
A software engineer working on the Cloudera Data in Motion security engineering team. This person will join a small subgroup focusing on information security research and development in the Apache NiFi (https://nifi.apache.org) ecosystem. We value a variety of backgrounds and experiences. 
Roles and Responsibilities:
* Research, design, implement, and test new security features
* Perform proactive threat modeling, risk assessment, and vulnerability detection and analysis on NiFi application and related libraries
* Work with cross-functional teams for application architecture and development
* Provide subject matter expertise and advice to other application developers on the Data In Motion engineering team
* Provide expert support to customers and field teams regarding security issues
@alopresto
alopresto / simple_msr.groovy
Created February 11, 2019 19:56
Very simplistic magnetic stripe track 1 & 2 parsing logic in Groovy. Extracts example fields and writes to JSON output.
import groovy.json.JsonBuilder
import groovy.json.JsonOutput
import groovy.json.JsonSlurper
import org.apache.commons.io.IOUtils
import org.apache.nifi.processor.io.StreamCallback
import java.nio.charset.StandardCharsets
def flowFile = session.get()
if (flowFile == null) {
@alopresto
alopresto / strip_blank_csv_line.xml
Created January 8, 2019 20:27
This template generates static CSV content with a leading blank line, removes it using a ReplaceText processor, and then prints the output in the log.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.2">
<description>This template generates static CSV content with a leading blank line, removes it using a ReplaceText processor, and then prints the output in the log. </description>
<groupId>2f2075af-0168-1000-a3df-beb67d2f9779</groupId>
<name>Remove blank leading line from CSV file</name>
<snippet>
<connections>
<id>28abf166-88e7-34db-0000-000000000000</id>
<parentGroupId>b22b93ea-4ad1-3221-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
@alopresto
alopresto / openssl.txt
Created December 18, 2018 19:54
Example of openssl being unable and able to decrypt NiFi-encrypted data depending on the cipher.
~W/scratch (master) 😉
🔒 0s @ 11:26:31 $ cd openssl
...sers/alopresto/Workspace/scratch/openssl (master) 😉
🔒 0s @ 11:26:34 $ ll # Showing files being output from NiFi (using SHA256_256AES)
total 16
drwxr-xr-x 4 alopresto staff 128B Dec 18 11:26 ./
drwxr-xr-x 178 alopresto staff 5.6K Dec 18 11:26 ../
-rw-r--r-- 1 alopresto staff 80B Dec 18 11:26 encrypted_2018_12_352_11_26_29_354.enc
-rw-r--r-- 1 alopresto staff 80B Dec 18 11:26 encrypted_2018_12_352_11_26_32_357.enc
...sers/alopresto/Workspace/scratch/openssl (master) 😉