Skip to content

Instantly share code, notes, and snippets.

@jvwing
jvwing / S3_Event_Notifications_Template.xml
Created Feb 15, 2017
Apache NiFi flow template for processing Amazon S3 event notifications. See https://www.batchiq.com/nifi-process-s3-event-notifications.html for details.
View S3_Event_Notifications_Template.xml
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description>Apache NiFi flow template for processing Amazon S3 event notifications.
See https://www.batchiq.com/nifi-process-s3-event-notifications.html for details. </description>
<groupId>caf31a0c-be81-46a6-1672-8cfe77634cbe</groupId>
<name>S3_Event_Notifications_Template</name>
<snippet>
<processGroups>
<id>f8805001-d079-1b7a-0000-000000000000</id>
<parentGroupId>caf31a0c-be81-46a6-0000-000000000000</parentGroupId>
@jvwing
jvwing / CloudTrail_Event_Processing_Starter_Kit.xml
Created Jan 17, 2017
Apache NiFi flow template for processing events from Amazon CloudTrail. See https://www.batchiq.com/process-cloudtrail-events-with-nifi.html for details.
View CloudTrail_Event_Processing_Starter_Kit.xml
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description>Template for processing events from Amazon CloudTrail.
See https://www.batchiq.com/process-cloudtrail-events-with-nifi.html</description>
<groupId>f46d6392-0158-1000-73ba-16e80594f898</groupId>
<name>CloudTrail_Event_Processing_Starter_Kit</name>
<snippet>
<processGroups>
<id>363e44cf-ed77-17f2-0000-000000000000</id>
@jvwing
jvwing / twitter_to_s3_sample_v1.xml
Last active Aug 11, 2019
Apache NiFi flow template for ingesting data to Amazon S3. See https://www.batchiq.com/s3-ingest-with-nifi.html for details.
View twitter_to_s3_sample_v1.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description></description><name>twitter_to_s3_sample_v1</name><snippet><connections><id>91f2ec79-3351-44eb-aa8a-b39623e71259</id><parentGroupId>8f183367-57f8-4340-8482-d246e37bb904</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>98a2973a-191b-4a6e-90f1-1b7ad12efe7e</groupId><id>dbd8bd4b-733f-43b3-b67b-bd2212598b10</id><type>INPUT_PORT</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><source><groupId>9c9d76ac-0ae2-4dee-92a3-a78ad2613342</groupId><id>cd683e79-2b95-4437-b7c8-4fce4ca8beeb</id><type>OUTPUT_PORT</type></source><zIndex>0</zIndex></connections><connections><id>21cc3ca7-8a0a-4a38-b113-0bdc9e0abe87</id><parentGroupId>8f183367-57f8-4340-8482-d246e37bb904</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</bac
@jvwing
jvwing / tweetsummary.avsc
Created Oct 31, 2016
Avro schema for a lightweight summary of a Tweet
View tweetsummary.avsc
{
"namespace": "com.batchiq.samples.tweet",
"type": "record",
"name": "tweetsummary",
"fields": [
{"name": "id", "type": "long"},
{"name": "id_str", "type": "string"},
{"name": "created_at", "type": "string"},
{"name": "timestamp_ms", "type": "string"},
{"name": "lang", "type": "string"},
@jvwing
jvwing / FlowFileFieldsTest.xml
Created Sep 19, 2016
NiFi template demonstrating use of built-in fields in Expression Language.
View FlowFileFieldsTest.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>Tests various built-in FlowFile fields using Expression Language.</description><name>FlowFileFieldsTest</name><snippet><connections><id>ffa84c5c-a236-45f6-a139-51d826e73d3c</id><parentGroupId>8524b3ae-950f-49c5-a936-44dd56b5a4d1</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>8524b3ae-950f-49c5-a936-44dd56b5a4d1</groupId><id>6f7d661c-e476-48b7-a6fe-e06ea4dc4fd3</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>8524b3ae-950f-49c5-a936-44dd56b5a4d1</groupId><id>53074840-f32c-459f-8739-7ed866bf7c3f</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>2c2150f3-b67f-4bbe-ad88-97e4adfc7dd0</id><parentGroupId>8524b3ae-950f-49c5-a936-44dd56b5a4d1</paren
@jvwing
jvwing / DatabaseExtract-Incremental.xml
Last active Aug 11, 2019
Apache NiFi flow template for extracting changed records out of a relational databases and into your NiFi flow. See https://www.batchiq.com/database-extract-with-nifi.html for details.
View DatabaseExtract-Incremental.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>Simple flow for extracting records from a relational database and into JSON text files.</description><name>DatabaseExtract-Incremental</name><snippet><connections><id>5f3f82ab-fc02-4322-938c-ad2fb628f094</id><parentGroupId>0ef63979-973e-43fc-82b2-a5cd52db0858</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>0ef63979-973e-43fc-82b2-a5cd52db0858</groupId><id>bb5fadae-29d0-4983-b76d-4185a5003685</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>merged</selectedRelationships><source><groupId>0ef63979-973e-43fc-82b2-a5cd52db0858</groupId><id>08cf9faa-f1cd-4e94-b3bd-ff6366973fbb</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>40e2bddf-ac38-4261-b64c-07571f77b8fd</id><parentGroupId>0ef63979-973e
@jvwing
jvwing / streamsetsdc.yaml
Created Apr 28, 2016
Cloud-Init Template for running StreamSets DC on EC2 (or other service)
View streamsetsdc.yaml
#cloud-config
runcmd:
- echo "$(date) - Starting StreamSets Data Collector Install"
- yum install -y https://archives.streamsets.com/datacollector/1.3.0.1/rpm/streamsets-datacollector-1.3.0.1-1.noarch.rpm
- service sdc start
- chkconfig --add sdc
- echo "$(date) - Finished StreamSets Data Collector Install"
# StreamSets UI will be at http://<public dns>:18630/, u:admin, p:admin
# Be sure to open port 18630 on your security group, or use SSH port forwarding
# See https://streamsets.com/opensource/
@jvwing
jvwing / Database_Insert.xml
Last active Aug 11, 2019
Apache NiFi flow template for inserting data into relational databases. See https://www.batchiq.com/database-ingest-with-nifi.html for details.
View Database_Insert.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>Demonstrates a simple pattern for inserting FlowFiles into a relational database using the PutSQL processor.</description><name>Database Insert</name><snippet><processGroups><id>f478ace0-9295-4ee9-9a25-96e3b1c3e94f</id><parentGroupId>23b56e53-a310-48e7-8a15-99d81109bdd9</parentGroupId><position><x>387.0</x><y>24.0</y></position><activeRemotePortCount>0</activeRemotePortCount><comments></comments><contents><connections><id>5e7de26e-13db-4b1b-8010-8f19f6611d49</id><parentGroupId>f478ace0-9295-4ee9-9a25-96e3b1c3e94f</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><bends><x>1030.955965197994</x><y>203.67437167941813</y></bends><bends><x>1067.0</x><y>253.0</y></bends><destination><groupId>f478ace0-9295-4ee9-9a25-96e3b1c3e94f</groupId><id>a0b3418c-096e-4138-9400-043dcc7da5ba</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowF
@jvwing
jvwing / TestEvaluateXPath.xml
Last active Aug 11, 2019
NiFi Template for Testing EvaluateXPath in NiFi 0.6.0
View TestEvaluateXPath.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>Test template for validation issues with EvaluateXPath in NiFi 0.6.0 (April 2016).</description><name>TestEvaluateXPath</name><snippet><processGroups><id>099c3c68-1720-4ddf-9173-25352325feb3</id><parentGroupId>6952f8cb-8b86-4446-a6ee-a7a98d7779dd</parentGroupId><position><x>780.0</x><y>72.0</y></position><activeRemotePortCount>0</activeRemotePortCount><comments></comments><contents><connections><id>9818db0d-fa04-4201-9597-9d1cee51e0e0</id><parentGroupId>099c3c68-1720-4ddf-9173-25352325feb3</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><bends><x>503.0</x><y>224.0</y></bends><bends><x>503.0</x><y>274.0</y></bends><destination><groupId>099c3c68-1720-4ddf-9173-25352325feb3</groupId><id>1105d248-9554-4f0c-960b-807dccff774b</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></na
@jvwing
jvwing / DynamoEnumConverter
Created May 30, 2014
For use with Amazon DynamoDB DataModel API for .NET. This class can be used to automagically serialize enum properties to DynamoDB, by decorating the property with the attribute [DynamoDBProperty(typeof(DynamoEnumConverter<AccountStatus>))], where "AccountStatus" is the name of the enumerated type..
View DynamoEnumConverter
public class DynamoEnumConverter<TEnum> : IPropertyConverter
{
public object FromEntry(DynamoDBEntry entry)
{
string valueAsString = entry.AsString();
TEnum valueAsEnum = (TEnum)Enum.Parse(typeof(TEnum), valueAsString);
return valueAsEnum;
}
public DynamoDBEntry ToEntry(object value)
You can’t perform that action at this time.