pvillard31

import java.nio.charset.StandardCharsets
import scala.collection.mutable.ListBuffer

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SQLContext
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.{Seconds, StreamingContext, Time}

pvillard31

package test;

import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.zip.GZIPOutputStream;

pvillard31

2016-06-14 23:08:09,027 WARN [StandardProcessScheduler Thread-2] o.a.n.controller.StandardProcessorNode Timed out while waiting for OnScheduled of 'PutIgniteCache' processor to finish. An attempt is made to cancel the task via Thread.interrupt(). However it does not guarantee that the task will be canceled since the code inside current OnScheduled operation may have been written to ignore interrupts which may result in runaway thread which could lead to more issues eventually requiring NiFi to be restarted. This is usually a bug in the target Processor 'PutIgniteCache[id=7e30ad66-d142-4501-b806-04ee2906f794]' that needs to be documented, reported and eventually fixed.
2016-06-14 23:08:09,027 ERROR [StandardProcessScheduler Thread-2] o.a.n.p.ignite.cache.PutIgniteCache PutIgniteCache[id=7e30ad66-d142-4501-b806-04ee2906f794] PutIgniteCache[id=7e30ad66-d142-4501-b806-04ee2906f794] failed to invoke @OnScheduled method due to java.lang.RuntimeException: Timed out while executing one of processor's OnScheduled task

pvillard31

Flow Controller:

  name: MiNiFi-test
  comment: ''
  
Core Properties:

  flow controller graceful shutdown period: 10 sec
  flow service write delay interval: 500 ms
  administrative yield duration: 30 sec

pvillard31

<?xml version="1.0" ?>
<template encoding-version="1.0">
  <description></description>
  <groupId>4bfc34ce-0157-1000-4886-447ec76bac19</groupId>
  <name>ValidateJSON</name>
  <snippet>
    <connections>
      <id>4c005bd2-0157-1000-0000-000000000000</id>
      <parentGroupId>4bfc34ce-0157-1000-0000-000000000000</parentGroupId>
      <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>

pvillard31

<?xml version="1.0" ?>
<template encoding-version="1.0">
  <description></description>
  <groupId>72102b36-ec80-4ab1-a6c0-345cf0c58126</groupId>
  <name>TestIgnite</name>
  <snippet>
    <connections>
      <id>8c6f014a-0157-1000-0000-000000000000</id>
      <parentGroupId>72102b36-ec80-4ab1-a6c0-345cf0c58126</parentGroupId>
      <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>

pvillard31

2017-02-02 15:17:39,342 INFO  [HiveServer2-Handler-Pool: Thread-14784]: session.HiveSessionImpl (HiveSessionImpl.java:acquire(304)) - We are setting the hadoop caller context to f4f2c170-dafc-487b-8e72-2f0da7cf6188 for thread HiveServer2-Handler-Pool: Thread-14784
2017-02-02 15:17:39,343 INFO  [HiveServer2-Handler-Pool: Thread-14784]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
2017-02-02 15:17:39,343 INFO  [HiveServer2-Handler-Pool: Thread-14784]: ql.Driver (Driver.java:compile(417)) - We are setting the hadoop caller context from HIVE_SSN_ID:f4f2c170-dafc-487b-8e72-2f0da7cf6188 to hive_20170202151739_dd052a5e-0de3-44a0-b8fc-bf9a7b134c6e
2017-02-02 15:17:39,344 INFO  [HiveServer2-Handler-Pool: Thread-14784]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
2017-02-02 15:17:39,344 INFO  [HiveServer2-Handler-Pool: Thread-14784]: parse.ParseDriver (ParseDriver.java:parse(185))

pvillard31

[root@pvillard ~]# su test
[test@pvillard root]$ beeline
Beeline version 1.2.1000.2.5.3.0-37 by Apache Hive
beeline> !connect jdbc:hive2://pvillard:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Connecting to jdbc:hive2://pvillard:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Enter username for jdbc:hive2://pvillard:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: test
Enter password for jdbc:hive2://pvillard:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: ********
Connected to: Apache Hive (version 1.2.1000.2.5.3.0-37)
Driver: Hive JDBC (version 1.2.1000.2.5.3.0-37)
Transaction isolation: TRANSACTION_REPEATABLE_READ

pvillard31

<?xml version="1.0" ?>
<template encoding-version="1.0">
  <description></description>
  <groupId>23e5393e-015a-1000-9be8-df55d535750c</groupId>
  <name>alluxioV2</name>
  <snippet>
    <connections>
      <id>015a1001-51f2-13a1-0000-000000000000</id>
      <parentGroupId>23e5393e-015a-1000-0000-000000000000</parentGroupId>
      <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>

pvillard31

<?xml version="1.0" ?>
<template encoding-version="1.0">
  <description></description>
  <groupId>d9025594-015a-1000-5d0d-c845d3f5a438</groupId>
  <name>routeOnFirstLine</name>
  <snippet>
    <connections>
      <id>d902a3f9-015a-1000-0000-000000000000</id>
      <parentGroupId>d9025594-015a-1000-0000-000000000000</parentGroupId>
      <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>