Skip to content

Instantly share code, notes, and snippets.

View ijokarumawak's full-sized avatar
🎯
Focusing

Koji Kawamura ijokarumawak

🎯
Focusing
83 followers · 16 following
  • Yokohama Japan
View GitHub Profile
@ijokarumawak
ijokarumawak / 0.NIFI-3545: Let M FlowFilews pass through once N signals arrive.md
Last active September 9, 2023 02:58

This Gist contains three (3) sample NiFi data flow templates to illustrate how NIFI-3545 enhancement works.

Wait N signals and let M FlowFiles go

Template file

Gate open/close

@ijokarumawak
ijokarumawak / 0.JoinCSVRecords.md
Last active August 31, 2023 06:13
A NiFi example flow which joins two CSV files using LookupRecord processor.

Join, Enrich multiple columns by looking up an external CSV file

This example illustrates how to use LookupRecord processor to join multiple columns from another CSV file. The key configuration is to define a result schema at 'RecordWriter' having the original columns AND the columns those are enriched by the lookuped values. Also, use 'Insert Record Fields' as LookupRecord 'Record Result Content', and '/' as 'Result Record Path' to join the lookuped values into the original record.

Given file-1.csv to lookup file-2.csv, this example outputs folloing content as a result:

Emp_Id,Name,Address,Mobile_No,Salary,Department
@ijokarumawak
ijokarumawak / 0.README.md
Last active August 31, 2023 04:56
AdVEGAnt Calendar - A Vega visualization example to create Kibana filter

AdVEGAnt Calendar

image

This example dashboard contains a Vega visualization, which renders a Christmas tree! When each node in the tree is clicked, a corresponding Kibana filter is created, so that the advent calendar blog post can be seen on the TSVB Markdown on the right hand side.

I was planning to write a post for Japanese Elastic Stack (Elasticsearch) Advent Calendar 2022, but every slot is filled. Nice! So I'm sharing it here... :)

@ijokarumawak
ijokarumawak / 0.NIFI_Calcurate_MIN_MAX_AVG_in_CSV.md
Created June 7, 2017 02:09
NiFi example: Calcurate Min, Max and Avg in a CSV file

NiFi example: Calcurate Min, Max and Avg in a CSV file

Let's calcurate Min, Max and Average value from this CSV file:

id, value
a, 345
b, 234
c, 1256
@ijokarumawak
ijokarumawak / 0.NIFI-Split_Filter_Wait-example.md
Last active August 24, 2023 04:47

NiFi: Split Filter and Wait Example

Detailed explanation is available here.

@ijokarumawak
ijokarumawak / read-flowfile-contents.py
Last active April 13, 2023 11:20
Example Python script to use from NiFi ExecuteScript processor which reads the first line from an incoming flow file.
from org.apache.nifi.processors.script import ExecuteScript
from org.apache.nifi.processor.io import InputStreamCallback
from java.io import BufferedReader, InputStreamReader
class ReadFirstLine(InputStreamCallback) :
__line = None;
def __init__(self) :
pass
@ijokarumawak
ijokarumawak / vega-sample.json
Created December 13, 2022 11:04
{
"$schema": "https://vega.github.io/schema/vega/v5.json",
"description": "A basic bar chart example, with value labels shown upon mouse hover.",
"data": [
{
"name": "table",
"values": [
{"name": "A", "current": 28, "upper": 80, "lower": 20},
{"name": "B", "current": 55, "upper": 70, "lower": 30},
{"name": "C", "current": 43, "upper": 60, "lower": 20},
@ijokarumawak
ijokarumawak / 0_README.md
Last active November 8, 2022 06:34
A simple Logstash example.

Simple Logstash example

  • Load CSV data
  • Apply some filters
  • Changing index name besed on value

How to run

Clone this Gist on your host.

@ijokarumawak
ijokarumawak / 0_README.md
Last active September 14, 2022 08:20
NiFi Example Flow template interacting with MS SQL Server.

NiFi Example: Copy rows from a MS SQL Server table to another

NOTE: You need to specify the right 'Catalog Name', 'Schema Name' and 'Table Name' at ConvertJSONToSQL processor to get table schema correctly.

-- Source database and table
create database nifi_a;
CREATE TABLE nifi_a.dbo.Test1
   (DATE_INSERT DATE,  
 G_INSTITUTION_ID varchar(25)
@ijokarumawak
ijokarumawak / 0_README.md
Last active June 15, 2022 09:34
Use Logstash ruby filter to rename field name based on value type to avoid Elasticsearch mapping conflict

This is Logstash ruby filter example to rename field names within a specified hash, based on each value type. The original question was asked at this Elastic Discuss.

How to test the filter

With -t, the filter can be tested:

logstash -e "filter { ruby { path => '/{path_of_the_ruby_script}/rename_by_data_type.rb' script_params => { 'field' => 'the_name_of_target_field' } } }" -t

Output: