Skip to content

Instantly share code, notes, and snippets.

Avatar
🎯
Focusing

Koji Kawamura ijokarumawak

🎯
Focusing
  • Yokohama Japan
View GitHub Profile
@ijokarumawak
ijokarumawak / forum_237985.ndjson
Created Jul 23, 2020
Sample Kibana visualization for the forum question 237985
View forum_237985.ndjson
{"attributes":{"fields":"[{\"name\":\"_id\",\"type\":\"string\",\"esTypes\":[\"_id\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_index\",\"type\":\"string\",\"esTypes\":[\"_index\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_score\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_source\",\"type\":\"_source\",\"esTypes\":[\"_source\"],\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_type\",\"type\":\"string\",\"esTypes\":[\"_type\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"id\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"id.keywor
@ijokarumawak
ijokarumawak / 0.README.md
Last active Aug 4, 2020
The simplest NiFi Wait Notify example
View 0.README.md

This flow using Wait processor to wait for the all split FlowFiles to be processed. When it runs, following log messages can be seen:

2019-04-05 10:02:58,752 INFO [Timer-Driven Process Thread-7] o.a.nifi.processors.standard.LogMessage LogMessage[id=eafe4890-0169-1000-9927-a81fbbafafc6] Processing 1/5 message for 875bd765-209d-4136-a75d-5d14241384fe
2019-04-05 10:02:58,769 INFO [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.LogMessage LogMessage[id=eafe4890-0169-1000-9927-a81fbbafafc6] Processing 2/5 message for 875bd765-209d-4136-a75d-5d14241384fe
2019-04-05 10:02:58,770 INFO [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.LogMessage LogMessage[id=eafe4890-0169-1000-9927-a81fbbafafc6] Processing 3/5 message for 875bd765-209d-4136-a75d-5d14241384fe
2019-04-05 10:02:58,770 INFO [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.LogMessage LogMessage[id=eafe4890-0169-1000-9927-a81fbbafafc6] Processing 4/5 message for 875bd765-209d-4136-a75d-5d14241384fe
2019-04-05 10:02:5
@ijokarumawak
ijokarumawak / Wait_Notify_multiple_counter_names.xml
Created Jan 7, 2019
NiFi template: Wait Notify multiple counter names
View Wait_Notify_multiple_counter_names.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.2">
<description></description>
<groupId>2615750e-0168-1000-cd37-5162c89dd01d</groupId>
<name>Wait Notify multiple counter names</name>
<snippet>
<controllerServices>
<id>a557c12c-476b-391d-0000-000000000000</id>
<parentGroupId>cd5035a3-f241-317a-0000-000000000000</parentGroupId>
<bundle>
@ijokarumawak
ijokarumawak / NIFI-5826_Fix_backslash_escaping.xml
Created Dec 5, 2018
NiFi flow template to test NIFI-5826 changes
View NIFI-5826_Fix_backslash_escaping.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.2">
<description></description>
<groupId>7bf539f4-0167-1000-97f1-3a91b5de75c4</groupId>
<name>NIFI-5826 Fix backslash escaping</name>
<snippet>
<processGroups>
<id>cf87c2e1-c49b-303b-0000-000000000000</id>
<parentGroupId>1e08db49-ce5e-30c7-0000-000000000000</parentGroupId>
<position>
@ijokarumawak
ijokarumawak / ListFile.java
Created Nov 30, 2018
Simple Java code to list files using NIO while waiting user input to proceed. This can be useful to see how it makes OS/native API calls with strace command.
View ListFile.java
import java.io.File;
import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Files;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;
@ijokarumawak
ijokarumawak / Merge_XML_Records.xml
Created Nov 29, 2018
A NiFi example template to illustrate how to merge multiple XML files.
View Merge_XML_Records.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.2">
<description></description>
<groupId>39379f66-0167-1000-9951-3cf7c004e310</groupId>
<name>Merge XML Records</name>
<snippet>
<controllerServices>
<id>36c4d83a-ff47-38e2-0000-000000000000</id>
<parentGroupId>376efa9a-48fc-3e3d-0000-000000000000</parentGroupId>
<bundle>
@ijokarumawak
ijokarumawak / AddTimestamp.xml
Created Nov 19, 2018
Example NiFi template to add new CSV timestamp column by UpdateRecord
View AddTimestamp.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.2">
<description></description>
<groupId>29cd7683-0167-1000-0886-c9dc91c022a5</groupId>
<name>AddTimestamp</name>
<snippet>
<connections>
<id>09785868-682a-3058-0000-000000000000</id>
<parentGroupId>818236f9-e91f-324b-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
@ijokarumawak
ijokarumawak / 0.InvokeHTTP_Attributes.md
Last active Feb 13, 2019
This NiFi flow template illustrates how incoming FlowFile attributes are carried to the InvokeHTTP output FlowFile.
View 0.InvokeHTTP_Attributes.md

This NiFi flow template illustrates how incoming FlowFile attributes are carried to the InvokeHTTP output FlowFile.

@ijokarumawak
ijokarumawak / 00_README.md
Last active Feb 13, 2019
NiFi example to ingest a set of files only when a complete set of files is ready.
View 00_README.md

This example flow can be used to process files with following requirements:

  • A group of files can only be processed when every files for a specific group is ready

  • Each filename has groupId (e.g. 123_456) and a type name (e.g. ab/cd/ef)

  • Example set of files for group '123_456'

    • file_123_456_ab.ex1
    • file_123_456_cd.ex1
    • file_123_456_ef.ex1
    • file_123_456.ex2
@ijokarumawak
ijokarumawak / 0.README.md
Created May 17, 2018
Docker example to run Dante SOCKS proxy server.
View 0.README.md

How to use Dante SOCKS proxy server

Create sockd.conf file locally on the docker host machine and map it with -v option of docker run.

Run socks server

docker run -d -p 1080:1080 -v /etc/sockd.conf:/etc/sockd.conf --name socks-server vimagick/dante

Add user

You can’t perform that action at this time.