Skip to content

Instantly share code, notes, and snippets.

View kkrbalam's full-sized avatar

Krishna Kishore Reddy Balam kkrbalam

5 followers · 15 following
View GitHub Profile
@kkrbalam
kkrbalam / index.md
Created February 14, 2020 03:13 — forked from alexcasalboni/index.md
Bridge Function between Kinesis Streams and Step Functions

Bridge Function between Kinesis Streams and Step Functions

For each record read from the Kinesis Stream, a StepFunction state machine will be executed asynchronously.

Required Environment Variables

  • region: the AWS region where your StepFunction state machine is defined.
  • stateMachineArn: the ARN of the StepFunction state machine you want to execute.

Notes

@kkrbalam
kkrbalam / CDC_MySQL_Replication.xml
Created April 26, 2019 09:37 — forked from mattyb149/CDC_MySQL_Replication.xml
NiFi template with end-to-end table replication from a source MySQL database to a target MySQL database
<?xml version="1.0" ?>
<template encoding-version="1.1">
<description>This template shows an end-to-end example of how to replicate table(s)
from a source MySQL database to a target MySQL database</description>
<groupId>68961054-015b-1000-2c21-01b42e84dbe0</groupId>
<name>CDC_MySQL_Replication</name>
<snippet>
<connections>
<id>9a0f0bca-5c0c-3f1e-0000-000000000000</id>
<parentGroupId>23d80cd0-6e6e-31c9-0000-000000000000</parentGroupId>
@kkrbalam
kkrbalam / pycrypto_nifi.xml
Created April 26, 2019 09:17 — forked from alopresto/pycrypto_nifi.xml
Generates a key file and an encrypted data file (both placeholders) and puts them in a directory, then watches the directory for those files and crafts a terminal command to execute a Python script with those arguments (currently mocked by 'echo').
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description>Generates a key file and an encrypted data file (both placeholders) and puts them in a directory, then watches the directory for those files and crafts a terminal command to execute a Python script with those arguments (currently mocked by 'echo'). </description>
<groupId>f6f232be-0159-1000-9698-ab4d847269d1</groupId>
<name>Pycrypto Demo</name>
<snippet>
<connections>
<id>f6f62cff-0159-1000-0000-000000000000</id>
<parentGroupId>f6f232be-0159-1000-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
@kkrbalam
kkrbalam / S3_Event_Notifications_Template.xml
Created April 26, 2019 09:14 — forked from jvwing/S3_Event_Notifications_Template.xml
Apache NiFi flow template for processing Amazon S3 event notifications. See https://www.batchiq.com/nifi-process-s3-event-notifications.html for details.
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description>Apache NiFi flow template for processing Amazon S3 event notifications.
See https://www.batchiq.com/nifi-process-s3-event-notifications.html for details. </description>
<groupId>caf31a0c-be81-46a6-1672-8cfe77634cbe</groupId>
<name>S3_Event_Notifications_Template</name>
<snippet>
<processGroups>
<id>f8805001-d079-1b7a-0000-000000000000</id>
<parentGroupId>caf31a0c-be81-46a6-0000-000000000000</parentGroupId>
@kkrbalam
kkrbalam / template-nifi-oauth2.xml
Created April 26, 2019 09:13 — forked from pvillard31/template-nifi-oauth2.xml
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description></description>
<groupId>e5a5f628-0159-1000-afdb-0edd88ca8397</groupId>
<name>template-nifi-oauth2</name>
<snippet>
<connections>
<id>f66d8688-0159-1000-0000-000000000000</id>
<parentGroupId>e5a5f628-0159-1000-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
@kkrbalam
kkrbalam / CDC_Replication.xml
Created April 26, 2019 09:13 — forked from mattyb149/CDC_Replication.xml
A test script for the GetChangeDataCaptureMySQL NiFi processor, taking CDC (binlog) events and transforming them to target SQL
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description></description>
<groupId>faf788c5-015a-1000-f344-de24ceb9d7e7</groupId>
<name>CDC_Replication</name>
<snippet>
<connections>
<id>d21bc8ee-015a-1000-0000-000000000000</id>
<parentGroupId>faf788c5-015a-1000-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
@kkrbalam
kkrbalam / DatabaseExtract-Incremental.xml
Created April 26, 2019 09:10 — forked from chrishuan9/DatabaseExtract-Incremental.xml
ETL Database Extraction with Apache NiFi Process Workflow
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>Simple flow for extracting records from a relational database and into JSON text files.</description><name>DatabaseExtract-Incremental</name><snippet><connections><id>5f3f82ab-fc02-4322-938c-ad2fb628f094</id><parentGroupId>0ef63979-973e-43fc-82b2-a5cd52db0858</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>0ef63979-973e-43fc-82b2-a5cd52db0858</groupId><id>bb5fadae-29d0-4983-b76d-4185a5003685</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>merged</selectedRelationships><source><groupId>0ef63979-973e-43fc-82b2-a5cd52db0858</groupId><id>08cf9faa-f1cd-4e94-b3bd-ff6366973fbb</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>40e2bddf-ac38-4261-b64c-07571f77b8fd</id><parentGroupId>0ef63979-973e
@kkrbalam
kkrbalam / pgp_encryption_template.xml
Created April 26, 2019 09:10 — forked from alopresto/pgp_encryption_template.xml
Generates empty flowfiles, replaces the contents with a static plaintext message, and writes them as a file to a directory. Reads files from a directory, encrypts using PGP, logs, decrypts, and writes the plaintext files to a new directory.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>Generates empty flowfiles, replaces the contents with a static plaintext message, and writes them as a file to a directory.
Reads files from a directory, encrypts using PGP, logs, decrypts, and writes the plaintext files to a new directory. </description><name>PGP Encryption Test</name><snippet><connections><id>69f574fd-134b-4946-b23c-6e63927c8141</id><parentGroupId>ea8b935a-765e-42d4-aeb8-6b22a423af81</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>ea8b935a-765e-42d4-aeb8-6b22a423af81</groupId><id>13567097-57a3-47bc-925e-c2b525b0e6d3</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>ea8b935a-765e-42d4-aeb8-6b22a423af81</groupId><id>4a8b0595-4233-4dda-ba15-64cb41db7b63</id>
@kkrbalam
kkrbalam / DatabaseExtract-Incremental.xml
Created April 26, 2019 09:09 — forked from jvwing/DatabaseExtract-Incremental.xml
Apache NiFi flow template for extracting changed records out of a relational databases and into your NiFi flow. See https://www.batchiq.com/database-extract-with-nifi.html for details.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>Simple flow for extracting records from a relational database and into JSON text files.</description><name>DatabaseExtract-Incremental</name><snippet><connections><id>5f3f82ab-fc02-4322-938c-ad2fb628f094</id><parentGroupId>0ef63979-973e-43fc-82b2-a5cd52db0858</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>0ef63979-973e-43fc-82b2-a5cd52db0858</groupId><id>bb5fadae-29d0-4983-b76d-4185a5003685</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>merged</selectedRelationships><source><groupId>0ef63979-973e-43fc-82b2-a5cd52db0858</groupId><id>08cf9faa-f1cd-4e94-b3bd-ff6366973fbb</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>40e2bddf-ac38-4261-b64c-07571f77b8fd</id><parentGroupId>0ef63979-973e
@kkrbalam
kkrbalam / database_insert.xml
Created April 26, 2019 09:09
Apache NiFi ETL Database Insert Process Flow
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>Demonstrates a simple pattern for inserting FlowFiles into a relational database using the PutSQL processor.</description><name>Database Insert</name><snippet><processGroups><id>f478ace0-9295-4ee9-9a25-96e3b1c3e94f</id><parentGroupId>23b56e53-a310-48e7-8a15-99d81109bdd9</parentGroupId><position><x>387.0</x><y>24.0</y></position><activeRemotePortCount>0</activeRemotePortCount><comments></comments><contents><connections><id>5e7de26e-13db-4b1b-8010-8f19f6611d49</id><parentGroupId>f478ace0-9295-4ee9-9a25-96e3b1c3e94f</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><bends><x>1030.955965197994</x><y>203.67437167941813</y></bends><bends><x>1067.0</x><y>253.0</y></bends><destination><groupId>f478ace0-9295-4ee9-9a25-96e3b1c3e94f</groupId><id>a0b3418c-096e-4138-9400-043dcc7da5ba</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowF