Skip to content

Instantly share code, notes, and snippets.

Avatar

Pierre Villard pvillard31

View GitHub Profile
View nifi-oauth.groovy
import java.security.SignatureException
import javax.crypto.Mac
import javax.crypto.spec.SecretKeySpec
def flowFile = session.get()
if (!flowFile) return
def static hmac(String data, String key) throws java.security.SignatureException
{
String result
View NiFi-Spark.scala
import java.nio.charset.StandardCharsets
import scala.collection.mutable.ListBuffer
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SQLContext
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.{Seconds, StreamingContext, Time}
View NiFi-TestHttpIssue.java
package test;
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.zip.GZIPOutputStream;
View PutIgnite.log
2016-06-14 23:08:09,027 WARN [StandardProcessScheduler Thread-2] o.a.n.controller.StandardProcessorNode Timed out while waiting for OnScheduled of 'PutIgniteCache' processor to finish. An attempt is made to cancel the task via Thread.interrupt(). However it does not guarantee that the task will be canceled since the code inside current OnScheduled operation may have been written to ignore interrupts which may result in runaway thread which could lead to more issues eventually requiring NiFi to be restarted. This is usually a bug in the target Processor 'PutIgniteCache[id=7e30ad66-d142-4501-b806-04ee2906f794]' that needs to be documented, reported and eventually fixed.
2016-06-14 23:08:09,027 ERROR [StandardProcessScheduler Thread-2] o.a.n.p.ignite.cache.PutIgniteCache PutIgniteCache[id=7e30ad66-d142-4501-b806-04ee2906f794] PutIgniteCache[id=7e30ad66-d142-4501-b806-04ee2906f794] failed to invoke @OnScheduled method due to java.lang.RuntimeException: Timed out while executing one of processor's OnScheduled task
@pvillard31
pvillard31 / config.yml
Created Jul 9, 2016
MiNiFi testing configuration
View config.yml
Flow Controller:
name: MiNiFi-test
comment: ''
Core Properties:
flow controller graceful shutdown period: 10 sec
flow service write delay interval: 500 ms
administrative yield duration: 30 sec
View gist:0d8d0d3122ecdd179cc39fbd83d4b9c4
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description></description>
<groupId>4bfc34ce-0157-1000-4886-447ec76bac19</groupId>
<name>ValidateJSON</name>
<snippet>
<connections>
<id>4c005bd2-0157-1000-0000-000000000000</id>
<parentGroupId>4bfc34ce-0157-1000-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
@pvillard31
pvillard31 / TestIgnite
Created Oct 3, 2016
NiFi workflow to test PutIgnite and GetIgnite processors
View TestIgnite
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description></description>
<groupId>72102b36-ec80-4ab1-a6c0-345cf0c58126</groupId>
<name>TestIgnite</name>
<snippet>
<connections>
<id>8c6f014a-0157-1000-0000-000000000000</id>
<parentGroupId>72102b36-ec80-4ab1-a6c0-345cf0c58126</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
View template-nifi-oauth2.xml
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description></description>
<groupId>e5a5f628-0159-1000-afdb-0edd88ca8397</groupId>
<name>template-nifi-oauth2</name>
<snippet>
<connections>
<id>f66d8688-0159-1000-0000-000000000000</id>
<parentGroupId>e5a5f628-0159-1000-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
View hive.errors.odbc.log
2017-02-02 15:17:39,342 INFO [HiveServer2-Handler-Pool: Thread-14784]: session.HiveSessionImpl (HiveSessionImpl.java:acquire(304)) - We are setting the hadoop caller context to f4f2c170-dafc-487b-8e72-2f0da7cf6188 for thread HiveServer2-Handler-Pool: Thread-14784
2017-02-02 15:17:39,343 INFO [HiveServer2-Handler-Pool: Thread-14784]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
2017-02-02 15:17:39,343 INFO [HiveServer2-Handler-Pool: Thread-14784]: ql.Driver (Driver.java:compile(417)) - We are setting the hadoop caller context from HIVE_SSN_ID:f4f2c170-dafc-487b-8e72-2f0da7cf6188 to hive_20170202151739_dd052a5e-0de3-44a0-b8fc-bf9a7b134c6e
2017-02-02 15:17:39,344 INFO [HiveServer2-Handler-Pool: Thread-14784]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
2017-02-02 15:17:39,344 INFO [HiveServer2-Handler-Pool: Thread-14784]: parse.ParseDriver (ParseDriver.java:parse(185))
View commands_beeline.txt
[root@pvillard ~]# su test
[test@pvillard root]$ beeline
Beeline version 1.2.1000.2.5.3.0-37 by Apache Hive
beeline> !connect jdbc:hive2://pvillard:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Connecting to jdbc:hive2://pvillard:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Enter username for jdbc:hive2://pvillard:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: test
Enter password for jdbc:hive2://pvillard:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: ********
Connected to: Apache Hive (version 1.2.1000.2.5.3.0-37)
Driver: Hive JDBC (version 1.2.1000.2.5.3.0-37)
Transaction isolation: TRANSACTION_REPEATABLE_READ