Skip to content

Instantly share code, notes, and snippets.

@gyfora

gyfora/gist:2bb5569fb703bbd7e47ba60352f90086 Secret

Created May 15, 2017 12:01
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save gyfora/2bb5569fb703bbd7e47ba60352f90086 to your computer and use it in GitHub Desktop.
Save gyfora/2bb5569fb703bbd7e47ba60352f90086 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
Logged in as: dr.who
Application
About
Jobs
Tools
Log Type: jobmanager.log
Log Upload Time: Mon May 15 14:00:48 +0200 2017
Log Length: 438303
2017-05-15 13:29:57,186 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - --------------------------------------------------------------------------------
2017-05-15 13:29:57,187 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Starting YARN ApplicationMaster / ResourceManager / JobManager (Version: 1.3-SNAPSHOT, Rev:29a221b, Date:15.05.2017 @ 11:22:12 CEST)
2017-05-15 13:29:57,198 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Current user: splat
2017-05-15 13:29:57,198 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.131-b11
2017-05-15 13:29:57,198 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Maximum heap size: 406 MiBytes
2017-05-15 13:29:57,198 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - JAVA_HOME: /fjord/java/
2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Hadoop version: 2.6.0
2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - JVM Options:
2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -Xmx424m
2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -Dlog.file=/fjord/hadoop/data/3/yarn/container-logs/application_1494426363399_0021/container_1494426363399_0021_01_000001/jobmanager.log
2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -Dlogback.configurationFile=file:logback.xml
2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -Dlog4j.configuration=file:log4j.properties
2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Program Arguments: (none)
2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Classpath: lib/flink-connector-kafka-0.8_2.10-1.3-SNAPSHOT.jar:lib/flink-connector-kafka-base_2.10-1.3-SNAPSHOT.jar:lib/flink-dist_2.10-1.3-SNAPSHOT.jar:lib/flink-python_2.10-1.3-SNAPSHOT.jar:lib/flink-shaded-hadoop2-1.3-SNAPSHOT.jar:lib/kafka-clients-0.8.2.2.jar:lib/kafka_2.10-0.8.2.2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:rbea-on-flink-2.0-SNAPSHOT.jar:flink.jar:flink-conf.yaml::/etc/hadoop/conf.cloudera.yarn2:/run/cloudera-scm-agent/process/948-yarn-NODEMANAGER:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-protobuf.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-generator.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-cascading.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-auth.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-auth-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-aws-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-thrift.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-azure-datalake.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-nfs.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-avro.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-annotations-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-azure-datalake-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-test-hadoop2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-encoding.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-nfs-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-jackson.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-pig-bundle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-column.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-format.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-scrooge_2.10.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-format-sources.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-annotations.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-hadoop-bundle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-tools.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common-2.6.0-cdh5.11.0-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-hadoop.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-pig.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-aws.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-scala_2.10.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-format-javadoc.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-codec-1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jersey-core-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/slf4j-log4j12.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/servlet-api-2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-digester-1.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/curator-client-2.7.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/xz-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jetty-util-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/activation-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/curator-recipes-2.7.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/asm-3.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/stax-api-1.0-2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-dynamodb-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/guava-11.0.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jsp-api-2.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/logredactor-1.0.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/httpcore-4.2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-kms-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/hamcrest-core-1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jets3t-0.9.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-s3-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/avro.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/slf4j-api-1.7.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-el-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/netty-3.10.5.Final.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-math3-3.1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/log4j-1.2.17.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/gson-2.2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/azure-data-lake-store-sdk-2.1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-io-2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-core-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-compress-1.4.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-logging-1.1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/curator-framework-2.7.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jsch-0.1.42.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-beanutils-1.9.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jaxb-api-2.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-collections-3.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/protobuf-java-2.5.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-sts-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jetty-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/httpclient-4.2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-lang-2.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jersey-server-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-configuration-1.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jsr305-3.0.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jackson-xc-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jettison-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/hue-plugins-3.9.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/zookeeper.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/junit-4.11.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-net-3.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/htrace-core4-4.0.1-incubating.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-cli-1.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/api-util-1.0.0-M20.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/paranamer-2.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jersey-json-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/xmlenc-0.52.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/mockito-all-1.8.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-httpclient-3.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-2.6.0-cdh5.11.0-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/asm-3.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/netty-3.10.5.Final.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/htrace-core4-4.0.1-incubating.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-client.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-api-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-registry-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-registry.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-client-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-api.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-common-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-tests-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-common-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/javax.inject-1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/xz-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jetty-util-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/activation-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/asm-3.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/guava-11.0.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/spark-yarn-shuffle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-core-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jline-2.11.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-io-2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/guice-3.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-mapper-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/spark-1.6.0-cdh5.11.0-yarn-shuffle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jetty-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-xc-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-jaxrs-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jettison-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/zookeeper.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-json-1.9.jar
2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - --------------------------------------------------------------------------------
2017-05-15 13:29:57,201 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Registered UNIX signal handlers for [TERM, HUP, INT]
2017-05-15 13:29:57,206 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - remoteKeytabPrincipal obtained null
2017-05-15 13:29:57,206 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - YARN daemon is running as: splat Yarn client user obtainer: splat
2017-05-15 13:29:57,210 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Loading config from directory /fjord/hadoop/data/nvme/splat/yarn-local-dir/usercache/splat/appcache/application_1494426363399_0021/container_1494426363399_0021_01_000001
2017-05-15 13:29:57,212 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.ask.timeout, 120 s
2017-05-15 13:29:57,212 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.client.timeout, 1200 s
2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.lookup.timeout, 120 s
2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.log.dir, /home/splat/flink/log/
2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.hdfs.hadoopconf, /home/splat/rbea-on-flink/deployment/environments/test/yarn-conf
2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.mode, zookeeper
2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.zookeeper.path.root, /flink-splat
2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.zookeeper.quorum, zk04.sto.midasplayer.com:2181,zk05.sto.midasplayer.com:2181,zk06.sto.midasplayer.com:2181
2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.zookeeper.storageDir, hdfs:///flink/ha
2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.checkpoints.dir, hdfs:///flink/external-checkpoints/bifrost
2017-05-15 13:29:57,214 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.savepoints.dir, hdfs:///flink/external-checkpoints/bifrost
2017-05-15 13:29:57,214 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.application-attempts, 10
2017-05-15 13:29:57,214 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.containers.vcores, 4
2017-05-15 13:29:57,214 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.heap-cutoff-ratio, 0.1
2017-05-15 13:29:57,214 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.maximum-failed-containers, 100
2017-05-15 13:29:57,244 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to splat (auth:SIMPLE)
2017-05-15 13:29:57,265 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - YARN assigned hostname for application master: splat34.sto.midasplayer.com
2017-05-15 13:29:57,270 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TaskManagers will be created with 1 task slots
2017-05-15 13:29:57,270 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TaskManagers will be started with container size 5000 MB, JVM heap size 4400 MB, JVM direct memory limit -1 MB
2017-05-15 13:29:57,274 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Trying to start actor system at splat34.sto.midasplayer.com:33785
2017-05-15 13:29:57,618 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2017-05-15 13:29:57,651 INFO Remoting - Starting remoting
2017-05-15 13:29:57,846 INFO Remoting - Remoting started; listening on addresses :[akka.tcp://flink@splat34.sto.midasplayer.com:33785]
2017-05-15 13:29:57,850 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Actor system started at akka.tcp://flink@splat34.sto.midasplayer.com:33785
2017-05-15 13:29:57,850 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Actor system bound to hostname splat34.sto.midasplayer.com.
2017-05-15 13:29:57,853 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TM:remote keytab path obtained null
2017-05-15 13:29:57,853 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TM:remote keytab principal obtained null
2017-05-15 13:29:57,853 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TM:remote yarn conf path obtained null
2017-05-15 13:29:57,853 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TM:remote krb5 path obtained null
2017-05-15 13:29:58,321 INFO org.apache.flink.yarn.Utils - Copying from file:/fjord/hadoop/data/nvme/splat/yarn-local-dir/usercache/splat/appcache/application_1494426363399_0021/container_1494426363399_0021_01_000001/b87cbc44-9ced-4a61-a7df-a790571dbf9e-taskmanager-conf.yaml to hdfs://splat34.sto.midasplayer.com:8020/user/splat/.flink/application_1494426363399_0021/b87cbc44-9ced-4a61-a7df-a790571dbf9e-taskmanager-conf.yaml
2017-05-15 13:29:58,612 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Prepared local resource for modified yaml: resource { scheme: "hdfs" host: "splat34.sto.midasplayer.com" port: 8020 file: "/user/splat/.flink/application_1494426363399_0021/b87cbc44-9ced-4a61-a7df-a790571dbf9e-taskmanager-conf.yaml" } size: 987 timestamp: 1494847798535 type: FILE visibility: APPLICATION
2017-05-15 13:29:58,616 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Creating container launch context for TaskManagers
2017-05-15 13:29:58,617 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Starting TaskManagers with command: $JAVA_HOME/bin/java -Xms4400m -Xmx4400m -Dlog.file=<LOG_DIR>/taskmanager.log -Dlogback.configurationFile=file:./logback.xml -Dlog4j.configuration=file:./log4j.properties org.apache.flink.yarn.YarnTaskManager --configDir . 1> <LOG_DIR>/taskmanager.out 2> <LOG_DIR>/taskmanager.err
2017-05-15 13:29:58,628 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.mode' instead of proper key 'high-availability'
2017-05-15 13:29:58,631 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.zookeeper.quorum' instead of proper key 'high-availability.zookeeper.quorum'
2017-05-15 13:29:58,631 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.zookeeper.path.root' instead of proper key 'high-availability.zookeeper.path.root'
2017-05-15 13:29:58,631 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections
2017-05-15 13:29:58,632 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink-splat/application_1494426363399_0021' as Zookeeper namespace.
2017-05-15 13:29:58,676 INFO org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=splat34.sto.midasplayer.com
2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_131
2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre
2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=lib/flink-connector-kafka-0.8_2.10-1.3-SNAPSHOT.jar:lib/flink-connector-kafka-base_2.10-1.3-SNAPSHOT.jar:lib/flink-dist_2.10-1.3-SNAPSHOT.jar:lib/flink-python_2.10-1.3-SNAPSHOT.jar:lib/flink-shaded-hadoop2-1.3-SNAPSHOT.jar:lib/kafka-clients-0.8.2.2.jar:lib/kafka_2.10-0.8.2.2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:rbea-on-flink-2.0-SNAPSHOT.jar:flink.jar:flink-conf.yaml::/etc/hadoop/conf.cloudera.yarn2:/run/cloudera-scm-agent/process/948-yarn-NODEMANAGER:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-protobuf.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-generator.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-cascading.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-auth.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-auth-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-aws-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-thrift.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-azure-datalake.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-nfs.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-avro.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-annotations-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-azure-datalake-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-test-hadoop2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-encoding.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-nfs-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-jackson.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-pig-bundle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-column.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-format.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-scrooge_2.10.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-format-sources.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-annotations.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-hadoop-bundle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-tools.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common-2.6.0-cdh5.11.0-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-hadoop.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-pig.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-aws.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-scala_2.10.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-format-javadoc.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-codec-1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jersey-core-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/slf4j-log4j12.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/servlet-api-2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-digester-1.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/curator-client-2.7.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/xz-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jetty-util-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/activation-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/curator-recipes-2.7.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/asm-3.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/stax-api-1.0-2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-dynamodb-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/guava-11.0.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jsp-api-2.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/logredactor-1.0.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/httpcore-4.2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-kms-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/hamcrest-core-1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jets3t-0.9.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-s3-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/avro.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/slf4j-api-1.7.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-el-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/netty-3.10.5.Final.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-math3-3.1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/log4j-1.2.17.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/gson-2.2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/azure-data-lake-store-sdk-2.1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-io-2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-core-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-compress-1.4.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-logging-1.1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/curator-framework-2.7.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jsch-0.1.42.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-beanutils-1.9.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jaxb-api-2.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-collections-3.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/protobuf-java-2.5.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-sts-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jetty-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/httpclient-4.2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-lang-2.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jersey-server-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-configuration-1.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jsr305-3.0.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jackson-xc-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jettison-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/hue-plugins-3.9.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/zookeeper.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/junit-4.11.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-net-3.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/htrace-core4-4.0.1-incubating.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-cli-1.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/api-util-1.0.0-M20.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/paranamer-2.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jersey-json-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/xmlenc-0.52.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/mockito-all-1.8.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-httpclient-3.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-2.6.0-cdh5.11.0-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/asm-3.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/netty-3.10.5.Final.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/htrace-core4-4.0.1-incubating.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-client.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-api-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-registry-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-registry.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-client-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-api.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-common-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-tests-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-common-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/javax.inject-1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/xz-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jetty-util-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/activation-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/asm-3.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/guava-11.0.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/spark-yarn-shuffle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-core-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jline-2.11.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-io-2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/guice-3.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-mapper-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/spark-1.6.0-cdh5.11.0-yarn-shuffle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jetty-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-xc-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-jaxrs-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jettison-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/zookeeper.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-json-1.9.jar
2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.16.0-4-amd64
2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=yarn
2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/var/lib/hadoop-yarn
2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/fjord/hadoop/data/nvme/splat/yarn-local-dir/usercache/splat/appcache/application_1494426363399_0021/container_1494426363399_0021_01_000001
2017-05-15 13:29:58,683 INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zk04.sto.midasplayer.com:2181,zk05.sto.midasplayer.com:2181,zk06.sto.midasplayer.com:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@781a9412
2017-05-15 13:29:58,702 WARN org.apache.zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-5646645451050700926.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-05-15 13:29:58,703 INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zk05.sto.midasplayer.com/172.26.82.243:2181
2017-05-15 13:29:58,703 ERROR org.apache.flink.shaded.org.apache.curator.ConnectionState - Authentication failed
2017-05-15 13:29:58,704 INFO org.apache.zookeeper.ClientCnxn - Socket connection established to zk05.sto.midasplayer.com/172.26.82.243:2181, initiating session
2017-05-15 13:29:58,709 INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zk05.sto.midasplayer.com/172.26.82.243:2181, sessionid = 0x25aad587316ddb2, negotiated timeout = 40000
2017-05-15 13:29:58,710 INFO org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2017-05-15 13:29:58,721 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.mode' instead of proper key 'high-availability'
2017-05-15 13:29:58,721 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.zookeeper.storageDir' instead of proper key 'high-availability.storageDir'
2017-05-15 13:29:58,754 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at hdfs:///flink/ha/application_1494426363399_0021/blob
2017-05-15 13:29:58,769 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-50fb85c1-d862-4d21-bfc7-eb1ac920f2f1
2017-05-15 13:29:58,771 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:41349 - max concurrent requests: 50 - max backlog: 1000
2017-05-15 13:29:58,790 INFO org.apache.flink.runtime.metrics.MetricRegistry - No metrics reporter configured, no metrics will be exposed/reported.
2017-05-15 13:29:58,801 INFO org.apache.flink.runtime.jobmanager.MemoryArchivist - Started memory archivist akka://flink/user/$a
2017-05-15 13:29:58,801 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.zookeeper.storageDir' instead of proper key 'high-availability.storageDir'
2017-05-15 13:29:58,825 WARN org.apache.flink.shaded.org.apache.curator.utils.ZKPaths - The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead.
2017-05-15 13:29:58,846 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.mode' instead of proper key 'high-availability'
2017-05-15 13:29:58,847 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Starting JobManager Web Frontend
2017-05-15 13:29:58,850 INFO org.apache.flink.yarn.YarnJobManager - Starting JobManager at akka.tcp://flink@splat34.sto.midasplayer.com:33785/user/jobmanager.
2017-05-15 13:29:58,850 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService@5ebb8b4b.
2017-05-15 13:29:58,854 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of JobManager log file: /fjord/hadoop/data/3/yarn/container-logs/application_1494426363399_0021/container_1494426363399_0021_01_000001/jobmanager.log
2017-05-15 13:29:58,854 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of JobManager stdout file: /fjord/hadoop/data/3/yarn/container-logs/application_1494426363399_0021/container_1494426363399_0021_01_000001/jobmanager.out
2017-05-15 13:29:58,854 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-cb0ef96f-0a47-412c-8e82-43b4bdb6dc61 for the web interface files
2017-05-15 13:29:58,854 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-d05454bc-860d-4b6e-a694-2306a5b962ef for web frontend JAR file uploads
2017-05-15 13:29:58,956 INFO org.apache.flink.yarn.YarnJobManager - JobManager akka.tcp://flink@splat34.sto.midasplayer.com:33785/user/jobmanager was granted leadership with leader session ID Some(c33ca0bb-8d46-47ab-a450-a39471007f79).
2017-05-15 13:29:58,965 INFO org.apache.flink.yarn.YarnJobManager - Delaying recovery of all jobs by 120000 milliseconds.
2017-05-15 13:29:59,110 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web frontend listening at 0.0.0.0:34993
2017-05-15 13:29:59,110 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Starting with JobManager akka.tcp://flink@splat34.sto.midasplayer.com:33785/user/jobmanager on port 34993
2017-05-15 13:29:59,110 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService.
2017-05-15 13:29:59,113 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - YARN application tolerates 100 failed TaskManager containers before giving up
2017-05-15 13:29:59,116 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - YARN Application Master started
2017-05-15 13:29:59,117 INFO org.apache.flink.runtime.webmonitor.JobManagerRetriever - New leader reachable under akka.tcp://flink@splat34.sto.midasplayer.com:33785/user/jobmanager:c33ca0bb-8d46-47ab-a450-a39471007f79.
2017-05-15 13:29:59,123 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService.
2017-05-15 13:29:59,124 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Initializing YARN resource master
2017-05-15 13:29:59,131 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at splat34.sto.midasplayer.com/172.26.83.103:8030
2017-05-15 13:29:59,148 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0
2017-05-15 13:29:59,148 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Registering Application Master with tracking url http://splat34.sto.midasplayer.com:34993
2017-05-15 13:29:59,245 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Trying to associate with JobManager leader akka.tcp://flink@splat34.sto.midasplayer.com:33785/user/jobmanager
2017-05-15 13:29:59,252 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Resource Manager associating with leading JobManager Actor[akka://flink/user/jobmanager#-193256653] - leader session c33ca0bb-8d46-47ab-a450-a39471007f79
2017-05-15 13:29:59,252 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Requesting new TaskManager container with 5000 megabytes memory. Pending requests: 1
2017-05-15 13:30:00,297 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : splat35.sto.midasplayer.com:8041
2017-05-15 13:30:00,309 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Received new container: container_1494426363399_0021_01_000002 - Remaining pending container requests: 0
2017-05-15 13:30:00,310 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Launching TaskManager in container ContainerInLaunch @ 1494847800310: Container: [ContainerId: container_1494426363399_0021_01_000002, NodeId: splat35.sto.midasplayer.com:8041, NodeHttpAddress: splat35.sto.midasplayer.com:8042, Resource: <memory:5120, vCores:4>, Priority: 0, Token: Token { kind: ContainerToken, service: 172.26.83.104:8041 }, ] on host splat35.sto.midasplayer.com
2017-05-15 13:30:00,312 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : splat35.sto.midasplayer.com:8041
2017-05-15 13:30:01,645 INFO org.apache.flink.yarn.YarnJobManager - ApplicationMaster will shut down session when job 1f387e6136f6faf1824808898750878e has finished.
2017-05-15 13:30:03,672 INFO org.apache.flink.yarn.YarnFlinkResourceManager - TaskManager container_1494426363399_0021_01_000002 has started.
2017-05-15 13:30:03,674 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at splat35 (akka.tcp://flink@splat35.sto.midasplayer.com:37378/user/taskmanager) as 983ddd2d6460628883d44134215e7987. Current number of registered hosts is 1. Current number of alive task slots is 1.
2017-05-15 13:30:03,901 INFO org.apache.flink.yarn.YarnJobManager - Submitting job 1f387e6136f6faf1824808898750878e (event.bifrost.log).
2017-05-15 13:30:03,915 INFO org.apache.flink.yarn.YarnJobManager - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, delayBetweenRestartAttempts=10000) for 1f387e6136f6faf1824808898750878e.
2017-05-15 13:30:03,929 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart
2017-05-15 13:30:03,944 INFO org.apache.flink.yarn.YarnJobManager - Running initialization on master for job event.bifrost.log (1f387e6136f6faf1824808898750878e).
2017-05-15 13:30:03,944 INFO org.apache.flink.yarn.YarnJobManager - Successfully ran initialization on master in 0 ms.
2017-05-15 13:30:03,960 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.zookeeper.storageDir' instead of proper key 'high-availability.storageDir'
2017-05-15 13:30:03,968 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Initialized in '/checkpoints/1f387e6136f6faf1824808898750878e'.
2017-05-15 13:30:03,977 INFO org.apache.flink.yarn.YarnJobManager - Using application-defined state backend for checkpoint/savepoint metadata: RocksDB State Backend {isInitialized=false, configuredDbBasePaths=null, initializedDbBasePaths=null, checkpointStreamBackend=File State Backend @ hdfs://splat34.sto.midasplayer.com:8020/flink/checkpoints/event.bifrost.log}.
2017-05-15 13:30:03,979 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Persisting periodic checkpoints externally at hdfs:///flink/external-checkpoints/bifrost.
2017-05-15 13:30:04,118 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Added SubmittedJobGraph(1f387e6136f6faf1824808898750878e, JobInfo(clients: Set((Actor[akka.tcp://flink@deploy.sto.midasplayer.com:38106/temp/$v],DETACHED)), start: 1494847803889)) to ZooKeeper.
2017-05-15 13:30:04,119 INFO org.apache.flink.yarn.YarnJobManager - Scheduling job 1f387e6136f6faf1824808898750878e (event.bifrost.log).
2017-05-15 13:30:04,119 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING.
2017-05-15 13:30:04,122 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (83626b435e5ba85b71abff97fba20cff) switched from CREATED to SCHEDULED.
2017-05-15 13:30:04,128 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2a3b9cd8726849c2c652a4e6d8fb59d5) switched from CREATED to SCHEDULED.
2017-05-15 13:30:04,129 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7b3e5fc47473abba62c4353c10c6a4b3) switched from CREATED to SCHEDULED.
2017-05-15 13:30:04,129 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bee6085558210aa1501d42432bdf7246) switched from CREATED to SCHEDULED.
2017-05-15 13:30:04,129 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (2c79d2f52e79f021b284dacf341adc9e) switched from CREATED to SCHEDULED.
2017-05-15 13:30:04,129 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (59574c2c8369b21866db827d30ea80be) switched from CREATED to SCHEDULED.
2017-05-15 13:30:04,130 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (65a61cfe9e7b4cbba965b6140bbf4a86) switched from CREATED to SCHEDULED.
2017-05-15 13:30:04,130 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (b44566133816ba074a4dfdea4b6c326b) switched from CREATED to SCHEDULED.
2017-05-15 13:30:04,130 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (c4fe2994919c03d68f629c611c70f832) switched from CREATED to SCHEDULED.
2017-05-15 13:30:04,130 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (1539ced5d6fc47263117e1fa3b68fbea) switched from CREATED to SCHEDULED.
2017-05-15 13:30:04,130 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (53a9ec178aa90a8e156a800ceab190c0) switched from CREATED to SCHEDULED.
2017-05-15 13:30:04,130 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (d4f53929056c373d92cd21814812afe1) switched from CREATED to SCHEDULED.
2017-05-15 13:30:04,130 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (a9b57a05d728fbb5e377f1c2442ac13e) switched from CREATED to SCHEDULED.
2017-05-15 13:30:04,134 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (83626b435e5ba85b71abff97fba20cff) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:04,135 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #0) to splat35
2017-05-15 13:30:04,141 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2a3b9cd8726849c2c652a4e6d8fb59d5) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:04,141 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #0) to splat35
2017-05-15 13:30:04,142 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7b3e5fc47473abba62c4353c10c6a4b3) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:04,142 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #0) to splat35
2017-05-15 13:30:04,144 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bee6085558210aa1501d42432bdf7246) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:04,144 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #0) to splat35
2017-05-15 13:30:04,145 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (2c79d2f52e79f021b284dacf341adc9e) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:04,145 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #0) to splat35
2017-05-15 13:30:04,146 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (59574c2c8369b21866db827d30ea80be) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:04,146 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #0) to splat35
2017-05-15 13:30:04,151 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (65a61cfe9e7b4cbba965b6140bbf4a86) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:04,151 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #0) to splat35
2017-05-15 13:30:04,152 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (b44566133816ba074a4dfdea4b6c326b) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:04,153 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #0) to splat35
2017-05-15 13:30:04,156 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (c4fe2994919c03d68f629c611c70f832) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:04,156 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #0) to splat35
2017-05-15 13:30:04,157 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (1539ced5d6fc47263117e1fa3b68fbea) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:04,157 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #0) to splat35
2017-05-15 13:30:04,158 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (53a9ec178aa90a8e156a800ceab190c0) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:04,158 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #0) to splat35
2017-05-15 13:30:04,159 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (d4f53929056c373d92cd21814812afe1) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:04,160 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #0) to splat35
2017-05-15 13:30:04,160 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (a9b57a05d728fbb5e377f1c2442ac13e) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:04,160 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #0) to splat35
2017-05-15 13:30:04,231 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (65a61cfe9e7b4cbba965b6140bbf4a86) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:04,231 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7b3e5fc47473abba62c4353c10c6a4b3) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:04,232 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (59574c2c8369b21866db827d30ea80be) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:04,233 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (53a9ec178aa90a8e156a800ceab190c0) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:04,234 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (1539ced5d6fc47263117e1fa3b68fbea) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:04,234 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2a3b9cd8726849c2c652a4e6d8fb59d5) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:04,235 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (c4fe2994919c03d68f629c611c70f832) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:04,236 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (83626b435e5ba85b71abff97fba20cff) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:04,236 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (b44566133816ba074a4dfdea4b6c326b) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:04,237 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (d4f53929056c373d92cd21814812afe1) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:04,238 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (2c79d2f52e79f021b284dacf341adc9e) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:04,239 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bee6085558210aa1501d42432bdf7246) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:04,239 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (a9b57a05d728fbb5e377f1c2442ac13e) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:44,550 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 1 @ 1494847844481
2017-05-15 13:30:46,364 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7b3e5fc47473abba62c4353c10c6a4b3) switched from RUNNING to FAILED.
AsynchronousException{java.lang.Exception: Could not materialize checkpoint 1 for operator Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1).}
at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:966)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Could not materialize checkpoint 1 for operator Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1).
... 6 more
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Could not open output stream for state backend
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43)
at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:902)
... 5 more
Suppressed: java.lang.Exception: Could not properly cancel managed operator state future.
at org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:98)
at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.cleanup(StreamTask.java:1018)
at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:957)
... 5 more
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Could not open output stream for state backend
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43)
at org.apache.flink.runtime.state.StateUtil.discardStateFuture(StateUtil.java:85)
at org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:96)
... 7 more
Caused by: java.io.IOException: Could not open output stream for state backend
at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:371)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.flush(FsCheckpointStreamFactory.java:228)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.write(FsCheckpointStreamFactory.java:203)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at org.apache.flink.api.java.typeutils.runtime.DataOutputViewStream.write(DataOutputViewStream.java:41)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1286)
at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1427)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1577)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:351)
at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:323)
at org.apache.flink.runtime.state.JavaSerializer.serialize(JavaSerializer.java:69)
at org.apache.flink.runtime.state.JavaSerializer.serialize(JavaSerializer.java:33)
at org.apache.flink.runtime.state.DefaultOperatorStateBackend$PartitionableListState.write(DefaultOperatorStateBackend.java:432)
at org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:233)
at org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:203)
at org.apache.flink.runtime.io.async.AbstractAsyncIOCallable.call(AbstractAsyncIOCallable.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:40)
at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:902)
... 5 more
Caused by: java.io.IOException: Cannot register Closeable, registry is already closed. Closing argument.
at org.apache.flink.util.AbstractCloseableRegistry.registerClosable(AbstractCloseableRegistry.java:66)
at org.apache.flink.core.fs.ClosingFSDataOutputStream.wrapSafe(ClosingFSDataOutputStream.java:101)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:125)
at org.apache.flink.core.fs.FileSystem.create(FileSystem.java:621)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:362)
... 27 more
[CIRCULAR REFERENCE:java.io.IOException: Could not open output stream for state backend]
2017-05-15 13:30:46,366 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING.
AsynchronousException{java.lang.Exception: Could not materialize checkpoint 1 for operator Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1).}
at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:966)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Could not materialize checkpoint 1 for operator Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1).
... 6 more
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Could not open output stream for state backend
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43)
at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:902)
... 5 more
Suppressed: java.lang.Exception: Could not properly cancel managed operator state future.
at org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:98)
at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.cleanup(StreamTask.java:1018)
at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:957)
... 5 more
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Could not open output stream for state backend
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43)
at org.apache.flink.runtime.state.StateUtil.discardStateFuture(StateUtil.java:85)
at org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:96)
... 7 more
Caused by: java.io.IOException: Could not open output stream for state backend
at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:371)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.flush(FsCheckpointStreamFactory.java:228)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.write(FsCheckpointStreamFactory.java:203)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at org.apache.flink.api.java.typeutils.runtime.DataOutputViewStream.write(DataOutputViewStream.java:41)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1286)
at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1427)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1577)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:351)
at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:323)
at org.apache.flink.runtime.state.JavaSerializer.serialize(JavaSerializer.java:69)
at org.apache.flink.runtime.state.JavaSerializer.serialize(JavaSerializer.java:33)
at org.apache.flink.runtime.state.DefaultOperatorStateBackend$PartitionableListState.write(DefaultOperatorStateBackend.java:432)
at org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:233)
at org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:203)
at org.apache.flink.runtime.io.async.AbstractAsyncIOCallable.call(AbstractAsyncIOCallable.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:40)
at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:902)
... 5 more
Caused by: java.io.IOException: Cannot register Closeable, registry is already closed. Closing argument.
at org.apache.flink.util.AbstractCloseableRegistry.registerClosable(AbstractCloseableRegistry.java:66)
at org.apache.flink.core.fs.ClosingFSDataOutputStream.wrapSafe(ClosingFSDataOutputStream.java:101)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:125)
at org.apache.flink.core.fs.FileSystem.create(FileSystem.java:621)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:362)
... 27 more
[CIRCULAR REFERENCE:java.io.IOException: Could not open output stream for state backend]
2017-05-15 13:30:46,373 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (83626b435e5ba85b71abff97fba20cff) switched from RUNNING to CANCELING.
2017-05-15 13:30:46,376 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2a3b9cd8726849c2c652a4e6d8fb59d5) switched from RUNNING to CANCELING.
2017-05-15 13:30:46,377 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bee6085558210aa1501d42432bdf7246) switched from RUNNING to CANCELING.
2017-05-15 13:30:46,377 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (2c79d2f52e79f021b284dacf341adc9e) switched from RUNNING to CANCELING.
2017-05-15 13:30:46,377 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (59574c2c8369b21866db827d30ea80be) switched from RUNNING to CANCELING.
2017-05-15 13:30:46,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (65a61cfe9e7b4cbba965b6140bbf4a86) switched from RUNNING to CANCELING.
2017-05-15 13:30:46,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (b44566133816ba074a4dfdea4b6c326b) switched from RUNNING to CANCELING.
2017-05-15 13:30:46,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (c4fe2994919c03d68f629c611c70f832) switched from RUNNING to CANCELING.
2017-05-15 13:30:46,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (1539ced5d6fc47263117e1fa3b68fbea) switched from RUNNING to CANCELING.
2017-05-15 13:30:46,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (53a9ec178aa90a8e156a800ceab190c0) switched from RUNNING to CANCELING.
2017-05-15 13:30:46,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (d4f53929056c373d92cd21814812afe1) switched from RUNNING to CANCELING.
2017-05-15 13:30:46,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (a9b57a05d728fbb5e377f1c2442ac13e) switched from RUNNING to CANCELING.
2017-05-15 13:30:46,396 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (83626b435e5ba85b71abff97fba20cff) switched from CANCELING to CANCELED.
2017-05-15 13:30:46,406 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bee6085558210aa1501d42432bdf7246) switched from CANCELING to CANCELED.
2017-05-15 13:30:46,407 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (59574c2c8369b21866db827d30ea80be) switched from CANCELING to CANCELED.
2017-05-15 13:30:46,408 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (1539ced5d6fc47263117e1fa3b68fbea) switched from CANCELING to CANCELED.
2017-05-15 13:30:46,409 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (65a61cfe9e7b4cbba965b6140bbf4a86) switched from CANCELING to CANCELED.
2017-05-15 13:30:46,410 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (53a9ec178aa90a8e156a800ceab190c0) switched from CANCELING to CANCELED.
2017-05-15 13:30:46,410 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (b44566133816ba074a4dfdea4b6c326b) switched from CANCELING to CANCELED.
2017-05-15 13:30:46,411 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (c4fe2994919c03d68f629c611c70f832) switched from CANCELING to CANCELED.
2017-05-15 13:30:46,412 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (d4f53929056c373d92cd21814812afe1) switched from CANCELING to CANCELED.
2017-05-15 13:30:46,412 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (a9b57a05d728fbb5e377f1c2442ac13e) switched from CANCELING to CANCELED.
2017-05-15 13:30:46,470 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2a3b9cd8726849c2c652a4e6d8fb59d5) switched from CANCELING to CANCELED.
2017-05-15 13:30:46,614 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (2c79d2f52e79f021b284dacf341adc9e) switched from CANCELING to CANCELED.
2017-05-15 13:30:46,614 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible.
2017-05-15 13:30:46,614 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING.
2017-05-15 13:30:46,614 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e).
2017-05-15 13:30:46,615 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ...
2017-05-15 13:30:56,616 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED.
2017-05-15 13:30:56,616 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper.
2017-05-15 13:30:56,621 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 0 checkpoints in ZooKeeper.
2017-05-15 13:30:56,621 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING.
2017-05-15 13:30:56,621 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (a805efa3b73f0379008da6d15fafc4d7) switched from CREATED to SCHEDULED.
2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2856c0d36955d3e90435ce1b1a2a9bb0) switched from CREATED to SCHEDULED.
2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d876b4b2d0f567a09562ce0e8720abd6) switched from CREATED to SCHEDULED.
2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (5359a06611829c1d9bfdecf0b66540a4) switched from CREATED to SCHEDULED.
2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (5eeb7c066ee255a446cc70edfbea2b42) switched from CREATED to SCHEDULED.
2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (61cb52d8e9d07d6645fe2d6f12db676e) switched from CREATED to SCHEDULED.
2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (08c882c354263ddfeb8711b12249c87e) switched from CREATED to SCHEDULED.
2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (92affefa971d217fda6a8c94fb64f2a5) switched from CREATED to SCHEDULED.
2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (279e8a00c652f78c5938d9b432852475) switched from CREATED to SCHEDULED.
2017-05-15 13:30:56,623 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (8ae81acf282ca346d76b8bb6694994a4) switched from CREATED to SCHEDULED.
2017-05-15 13:30:56,623 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (5af3e14dc574394bf24fe7e3b9186812) switched from CREATED to SCHEDULED.
2017-05-15 13:30:56,623 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (a46f9e0b28f10f3d4a02e6fa6699840a) switched from CREATED to SCHEDULED.
2017-05-15 13:30:56,623 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7534412a727ea1a1daebef412095c0c7) switched from CREATED to SCHEDULED.
2017-05-15 13:30:56,623 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (a805efa3b73f0379008da6d15fafc4d7) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:56,624 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #1) to splat35
2017-05-15 13:30:56,624 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2856c0d36955d3e90435ce1b1a2a9bb0) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:56,624 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #1) to splat35
2017-05-15 13:30:56,625 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d876b4b2d0f567a09562ce0e8720abd6) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:56,625 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #1) to splat35
2017-05-15 13:30:56,625 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (5359a06611829c1d9bfdecf0b66540a4) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:56,625 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #1) to splat35
2017-05-15 13:30:56,625 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (5eeb7c066ee255a446cc70edfbea2b42) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:56,625 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #1) to splat35
2017-05-15 13:30:56,626 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (61cb52d8e9d07d6645fe2d6f12db676e) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:56,626 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #1) to splat35
2017-05-15 13:30:56,626 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (08c882c354263ddfeb8711b12249c87e) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:56,626 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #1) to splat35
2017-05-15 13:30:56,626 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (92affefa971d217fda6a8c94fb64f2a5) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:56,626 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #1) to splat35
2017-05-15 13:30:56,626 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (279e8a00c652f78c5938d9b432852475) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:56,627 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #1) to splat35
2017-05-15 13:30:56,627 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (8ae81acf282ca346d76b8bb6694994a4) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:56,627 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #1) to splat35
2017-05-15 13:30:56,627 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (5af3e14dc574394bf24fe7e3b9186812) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:56,627 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #1) to splat35
2017-05-15 13:30:56,627 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (a46f9e0b28f10f3d4a02e6fa6699840a) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:56,627 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #1) to splat35
2017-05-15 13:30:56,628 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7534412a727ea1a1daebef412095c0c7) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:30:56,628 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #1) to splat35
2017-05-15 13:30:56,640 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (a805efa3b73f0379008da6d15fafc4d7) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:56,642 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2856c0d36955d3e90435ce1b1a2a9bb0) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:56,643 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d876b4b2d0f567a09562ce0e8720abd6) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:56,644 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (5359a06611829c1d9bfdecf0b66540a4) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:56,645 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (5eeb7c066ee255a446cc70edfbea2b42) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:56,649 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (61cb52d8e9d07d6645fe2d6f12db676e) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:56,649 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (08c882c354263ddfeb8711b12249c87e) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:56,668 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (92affefa971d217fda6a8c94fb64f2a5) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:56,670 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (279e8a00c652f78c5938d9b432852475) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:56,671 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (8ae81acf282ca346d76b8bb6694994a4) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:56,671 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (5af3e14dc574394bf24fe7e3b9186812) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:56,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (a46f9e0b28f10f3d4a02e6fa6699840a) switched from DEPLOYING to RUNNING.
2017-05-15 13:30:56,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7534412a727ea1a1daebef412095c0c7) switched from DEPLOYING to RUNNING.
2017-05-15 13:31:56,624 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 2 @ 1494847916621
2017-05-15 13:31:57,135 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 2 (1108645 bytes in 462 ms).
2017-05-15 13:31:58,980 INFO org.apache.flink.yarn.YarnJobManager - Attempting to recover all jobs.
2017-05-15 13:31:58,984 INFO org.apache.flink.yarn.YarnJobManager - There are 1 jobs to recover. Starting the job recovery.
2017-05-15 13:31:58,987 INFO org.apache.flink.yarn.YarnJobManager - Attempting to recover job 1f387e6136f6faf1824808898750878e.
2017-05-15 13:31:59,042 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Recovered SubmittedJobGraph(1f387e6136f6faf1824808898750878e, JobInfo(clients: Set((Actor[akka.tcp://flink@deploy.sto.midasplayer.com:38106/temp/$v],DETACHED)), start: 1494847803889)).
2017-05-15 13:31:59,043 INFO org.apache.flink.yarn.YarnJobManager - Ignoring job recovery for 1f387e6136f6faf1824808898750878e, because it is already submitted.
2017-05-15 13:32:57,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 3 @ 1494847977134
2017-05-15 13:32:57,816 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 3 (1120696 bytes in 624 ms).
2017-05-15 13:32:57,819 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 2.
2017-05-15 13:32:57,878 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 2.
java.lang.Exception: Could not execute callable action for checkpoint 2.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:33:57,818 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 4 @ 1494848037815
2017-05-15 13:33:58,319 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 4 (1132754 bytes in 460 ms).
2017-05-15 13:33:58,321 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 3.
2017-05-15 13:33:58,351 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 3.
java.lang.Exception: Could not execute callable action for checkpoint 3.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:34:58,321 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 5 @ 1494848098319
2017-05-15 13:34:58,894 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 5 (1145382 bytes in 530 ms).
2017-05-15 13:34:58,896 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 4.
2017-05-15 13:34:58,927 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 4.
java.lang.Exception: Could not execute callable action for checkpoint 4.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:35:58,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 6 @ 1494848158893
2017-05-15 13:35:59,401 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 6 (1124229 bytes in 464 ms).
2017-05-15 13:35:59,403 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 5.
2017-05-15 13:35:59,426 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 5.
java.lang.Exception: Could not execute callable action for checkpoint 5.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:36:59,403 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 7 @ 1494848219401
2017-05-15 13:36:59,893 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 7 (1139094 bytes in 448 ms).
2017-05-15 13:36:59,895 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 6.
2017-05-15 13:36:59,926 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 6.
java.lang.Exception: Could not execute callable action for checkpoint 6.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:37:59,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 8 @ 1494848279893
2017-05-15 13:38:00,779 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 8 (1154702 bytes in 733 ms).
2017-05-15 13:38:00,780 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 7.
2017-05-15 13:38:00,878 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 7.
java.lang.Exception: Could not execute callable action for checkpoint 7.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:39:00,780 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 9 @ 1494848340778
2017-05-15 13:39:01,353 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 9 (1170306 bytes in 531 ms).
2017-05-15 13:39:01,355 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 8.
2017-05-15 13:39:01,385 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 8.
java.lang.Exception: Could not execute callable action for checkpoint 8.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:40:01,356 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 10 @ 1494848401352
2017-05-15 13:40:01,861 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 10 (1131407 bytes in 465 ms).
2017-05-15 13:40:01,862 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 9.
2017-05-15 13:40:01,885 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 9.
java.lang.Exception: Could not execute callable action for checkpoint 9.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:41:01,863 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 11 @ 1494848461860
2017-05-15 13:41:02,411 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 11 (1149375 bytes in 507 ms).
2017-05-15 13:41:02,413 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 10.
2017-05-15 13:41:02,443 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 10.
java.lang.Exception: Could not execute callable action for checkpoint 10.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:42:02,412 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 12 @ 1494848522410
2017-05-15 13:42:02,986 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 12 (1167347 bytes in 532 ms).
2017-05-15 13:42:02,988 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 11.
2017-05-15 13:42:03,017 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 11.
java.lang.Exception: Could not execute callable action for checkpoint 11.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:43:02,988 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 13 @ 1494848582986
2017-05-15 13:43:03,442 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 13 (1185638 bytes in 411 ms).
2017-05-15 13:43:03,444 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 12.
2017-05-15 13:43:03,473 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 12.
java.lang.Exception: Could not execute callable action for checkpoint 12.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:44:03,444 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 14 @ 1494848643442
2017-05-15 13:44:03,824 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 14 (1135623 bytes in 355 ms).
2017-05-15 13:44:03,826 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 13.
2017-05-15 13:44:03,847 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 13.
java.lang.Exception: Could not execute callable action for checkpoint 13.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:45:03,826 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 15 @ 1494848703824
2017-05-15 13:45:04,266 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 15 (1154541 bytes in 414 ms).
2017-05-15 13:45:04,268 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 14.
2017-05-15 13:45:04,298 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 14.
java.lang.Exception: Could not execute callable action for checkpoint 14.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:46:04,268 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 16 @ 1494848764266
2017-05-15 13:46:04,741 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 16 (1174659 bytes in 447 ms).
2017-05-15 13:46:04,743 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 15.
2017-05-15 13:46:04,772 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 15.
java.lang.Exception: Could not execute callable action for checkpoint 15.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:47:04,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 17 @ 1494848824740
2017-05-15 13:47:05,157 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 17 (1194777 bytes in 389 ms).
2017-05-15 13:47:05,158 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 16.
2017-05-15 13:47:05,189 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 16.
java.lang.Exception: Could not execute callable action for checkpoint 16.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:48:05,158 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 18 @ 1494848885156
2017-05-15 13:48:05,600 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 18 (1140489 bytes in 416 ms).
2017-05-15 13:48:05,601 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 17.
2017-05-15 13:48:05,624 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 17.
java.lang.Exception: Could not execute callable action for checkpoint 17.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:49:05,601 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 19 @ 1494848945599
2017-05-15 13:49:05,999 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 19 (1161882 bytes in 373 ms).
2017-05-15 13:49:06,001 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 18.
2017-05-15 13:49:06,023 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 18.
java.lang.Exception: Could not execute callable action for checkpoint 18.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:50:06,001 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 20 @ 1494849005998
2017-05-15 13:50:06,417 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 20 (1183918 bytes in 391 ms).
2017-05-15 13:50:06,419 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 19.
2017-05-15 13:50:06,448 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 19.
java.lang.Exception: Could not execute callable action for checkpoint 19.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:51:06,419 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 21 @ 1494849066417
2017-05-15 13:51:06,858 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 21 (1206982 bytes in 405 ms).
2017-05-15 13:51:06,859 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 20.
2017-05-15 13:51:06,889 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 20.
java.lang.Exception: Could not execute callable action for checkpoint 20.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:52:06,859 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 22 @ 1494849126857
2017-05-15 13:52:07,274 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 22 (1146260 bytes in 390 ms).
2017-05-15 13:52:07,275 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 21.
2017-05-15 13:52:07,298 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 21.
java.lang.Exception: Could not execute callable action for checkpoint 21.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:53:07,276 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 23 @ 1494849187274
2017-05-15 13:53:07,757 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 23 (1170386 bytes in 447 ms).
2017-05-15 13:53:07,759 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 22.
2017-05-15 13:53:07,780 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 22.
java.lang.Exception: Could not execute callable action for checkpoint 22.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:54:07,758 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 24 @ 1494849247756
2017-05-15 13:54:08,290 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 24 (1194506 bytes in 498 ms).
2017-05-15 13:54:08,291 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 23.
2017-05-15 13:54:08,313 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 23.
java.lang.Exception: Could not execute callable action for checkpoint 23.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:55:08,291 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 25 @ 1494849308289
2017-05-15 13:55:08,864 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 25 (1218646 bytes in 540 ms).
2017-05-15 13:55:08,866 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 24.
2017-05-15 13:55:08,896 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 24.
java.lang.Exception: Could not execute callable action for checkpoint 24.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:56:08,866 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 26 @ 1494849368864
2017-05-15 13:56:09,422 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 26 (1148346 bytes in 523 ms).
2017-05-15 13:56:09,424 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 25.
2017-05-15 13:56:09,446 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 25.
java.lang.Exception: Could not execute callable action for checkpoint 25.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:57:09,425 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 27 @ 1494849429422
2017-05-15 13:57:09,881 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 27 (1173336 bytes in 423 ms).
2017-05-15 13:57:09,883 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 26.
2017-05-15 13:57:09,904 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 26.
java.lang.Exception: Could not execute callable action for checkpoint 26.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:58:09,883 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 28 @ 1494849489880
2017-05-15 13:58:10,374 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 28 (1198326 bytes in 459 ms).
2017-05-15 13:58:10,376 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 27.
2017-05-15 13:58:10,406 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 27.
java.lang.Exception: Could not execute callable action for checkpoint 27.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:59:10,376 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 29 @ 1494849550373
2017-05-15 13:59:10,982 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 29 (1223316 bytes in 574 ms).
2017-05-15 13:59:10,984 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 28.
2017-05-15 13:59:11,014 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 28.
java.lang.Exception: Could not execute callable action for checkpoint 28.
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380)
at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251)
at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163)
at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131)
at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288)
at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318)
at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378)
... 4 more
2017-05-15 13:59:20,335 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Disassociated]
2017-05-15 13:59:21,286 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Container container_1494426363399_0021_01_000002 failed. Exit status: 143
2017-05-15 13:59:21,287 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Diagnostics for container container_1494426363399_0021_01_000002 in state COMPLETE : exitStatus=143 diagnostics=Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal
2017-05-15 13:59:21,287 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Total number of failed containers so far: 1
2017-05-15 13:59:21,287 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Requesting new TaskManager container with 5000 megabytes memory. Pending requests: 1
2017-05-15 13:59:21,287 INFO org.apache.flink.yarn.YarnJobManager - Task manager akka.tcp://flink@splat35.sto.midasplayer.com:37378/user/taskmanager terminated.
2017-05-15 13:59:21,288 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (5359a06611829c1d9bfdecf0b66540a4) switched from RUNNING to FAILED.
java.lang.Exception: TaskManager was lost/killed: container_1494426363399_0021_01_000002 @ splat35.sto.midasplayer.com (dataPort=41610)
at org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:217)
at org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:533)
at org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:192)
at org.apache.flink.runtime.instance.Instance.markDead(Instance.java:167)
at org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:212)
at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$handleTaskManagerTerminated(JobManager.scala:1228)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:474)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at org.apache.flink.runtime.clusterframework.ContaineredJobManager$$anonfun$handleContainerMessage$1.applyOrElse(ContaineredJobManager.scala:100)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162)
at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:38)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:125)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2017-05-15 13:59:21,288 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING.
java.lang.Exception: TaskManager was lost/killed: container_1494426363399_0021_01_000002 @ splat35.sto.midasplayer.com (dataPort=41610)
at org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:217)
at org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:533)
at org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:192)
at org.apache.flink.runtime.instance.Instance.markDead(Instance.java:167)
at org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:212)
at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$handleTaskManagerTerminated(JobManager.scala:1228)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:474)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at org.apache.flink.runtime.clusterframework.ContaineredJobManager$$anonfun$handleContainerMessage$1.applyOrElse(ContaineredJobManager.scala:100)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162)
at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:38)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:125)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2017-05-15 13:59:21,292 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (a805efa3b73f0379008da6d15fafc4d7) switched from RUNNING to CANCELING.
2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2856c0d36955d3e90435ce1b1a2a9bb0) switched from RUNNING to CANCELING.
2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d876b4b2d0f567a09562ce0e8720abd6) switched from RUNNING to CANCELING.
2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (5eeb7c066ee255a446cc70edfbea2b42) switched from RUNNING to CANCELING.
2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (61cb52d8e9d07d6645fe2d6f12db676e) switched from RUNNING to CANCELING.
2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (08c882c354263ddfeb8711b12249c87e) switched from RUNNING to CANCELING.
2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (92affefa971d217fda6a8c94fb64f2a5) switched from RUNNING to CANCELING.
2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (279e8a00c652f78c5938d9b432852475) switched from RUNNING to CANCELING.
2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (8ae81acf282ca346d76b8bb6694994a4) switched from RUNNING to CANCELING.
2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (5af3e14dc574394bf24fe7e3b9186812) switched from RUNNING to CANCELING.
2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (a46f9e0b28f10f3d4a02e6fa6699840a) switched from RUNNING to CANCELING.
2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7534412a727ea1a1daebef412095c0c7) switched from RUNNING to CANCELING.
2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (61cb52d8e9d07d6645fe2d6f12db676e) switched from CANCELING to CANCELED.
2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d876b4b2d0f567a09562ce0e8720abd6) switched from CANCELING to CANCELED.
2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (5af3e14dc574394bf24fe7e3b9186812) switched from CANCELING to CANCELED.
2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (08c882c354263ddfeb8711b12249c87e) switched from CANCELING to CANCELED.
2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2856c0d36955d3e90435ce1b1a2a9bb0) switched from CANCELING to CANCELED.
2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (279e8a00c652f78c5938d9b432852475) switched from CANCELING to CANCELED.
2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (a805efa3b73f0379008da6d15fafc4d7) switched from CANCELING to CANCELED.
2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (a46f9e0b28f10f3d4a02e6fa6699840a) switched from CANCELING to CANCELED.
2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (5eeb7c066ee255a446cc70edfbea2b42) switched from CANCELING to CANCELED.
2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (92affefa971d217fda6a8c94fb64f2a5) switched from CANCELING to CANCELED.
2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (8ae81acf282ca346d76b8bb6694994a4) switched from CANCELING to CANCELED.
2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7534412a727ea1a1daebef412095c0c7) switched from CANCELING to CANCELED.
2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible.
2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING.
2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e).
2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ...
2017-05-15 13:59:21,296 INFO org.apache.flink.runtime.instance.InstanceManager - Unregistered task manager splat35.sto.midasplayer.com/172.26.83.104. Number of registered task managers 0. Number of available slots 0.
2017-05-15 13:59:25,371 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 13:59:27,291 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : splat34.sto.midasplayer.com:8041
2017-05-15 13:59:27,293 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Received new container: container_1494426363399_0021_01_000003 - Remaining pending container requests: 0
2017-05-15 13:59:27,293 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Launching TaskManager in container ContainerInLaunch @ 1494849567293: Container: [ContainerId: container_1494426363399_0021_01_000003, NodeId: splat34.sto.midasplayer.com:8041, NodeHttpAddress: splat34.sto.midasplayer.com:8042, Resource: <memory:5120, vCores:4>, Priority: 0, Token: Token { kind: ContainerToken, service: 172.26.83.103:8041 }, ] on host splat34.sto.midasplayer.com
2017-05-15 13:59:27,293 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : splat34.sto.midasplayer.com:8041
2017-05-15 13:59:27,305 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Received new container: container_1494426363399_0021_01_000004 - Remaining pending container requests: 0
2017-05-15 13:59:27,305 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Returning excess container container_1494426363399_0021_01_000004
2017-05-15 13:59:27,799 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Container container_1494426363399_0021_01_000004 completed successfully with diagnostics: Container released by application
2017-05-15 13:59:29,962 INFO org.apache.flink.yarn.YarnFlinkResourceManager - TaskManager container_1494426363399_0021_01_000003 has started.
2017-05-15 13:59:29,962 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at splat34 (akka.tcp://flink@splat34.sto.midasplayer.com:53480/user/taskmanager) as 326104999a117409c1609bdd0b38403a. Current number of registered hosts is 1. Current number of alive task slots is 1.
2017-05-15 13:59:30,394 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 13:59:31,300 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED.
2017-05-15 13:59:31,300 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper.
2017-05-15 13:59:31,303 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 1 checkpoints in ZooKeeper.
2017-05-15 13:59:31,304 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29.
2017-05-15 13:59:31,309 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29.
2017-05-15 13:59:31,311 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 29 @ 1494849550373 for 1f387e6136f6faf1824808898750878e.
2017-05-15 13:59:31,314 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore
2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING.
2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (0bd17588f4674f754ced2562c447ed0a) switched from CREATED to SCHEDULED.
2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (006621199f1282f7ec623fadb1cb65dc) switched from CREATED to SCHEDULED.
2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (79ded85532660a95ac8280f21ae9d8d4) switched from CREATED to SCHEDULED.
2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bf80bf5edfc2c072fe36a61156ff935a) switched from CREATED to SCHEDULED.
2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (fbb19bea4823fc2f0fd1482d61054a20) switched from CREATED to SCHEDULED.
2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a55516edcbcc448edc5fd77e31d55d64) switched from CREATED to SCHEDULED.
2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (114fe598aed8f758e58fef63caca9d72) switched from CREATED to SCHEDULED.
2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3e600d83298e0b81658d57bef7e8536c) switched from CREATED to SCHEDULED.
2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (84b8b824b499518947ae477dad73924d) switched from CREATED to SCHEDULED.
2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (56c3498a9b9ed624d608bc75ece2142b) switched from CREATED to SCHEDULED.
2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (484eae4801564b1c5064fc74408327bc) switched from CREATED to SCHEDULED.
2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (e89114ab129b9a766bae9394d08b721a) switched from CREATED to SCHEDULED.
2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (e0ef852e975307d3794663d38f7488ca) switched from CREATED to SCHEDULED.
2017-05-15 13:59:31,316 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (0bd17588f4674f754ced2562c447ed0a) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:31,316 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #2) to splat34
2017-05-15 13:59:31,316 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (006621199f1282f7ec623fadb1cb65dc) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:31,316 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #2) to splat34
2017-05-15 13:59:31,316 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (79ded85532660a95ac8280f21ae9d8d4) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:31,316 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #2) to splat34
2017-05-15 13:59:31,317 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bf80bf5edfc2c072fe36a61156ff935a) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:31,317 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #2) to splat34
2017-05-15 13:59:31,317 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (fbb19bea4823fc2f0fd1482d61054a20) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:31,317 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #2) to splat34
2017-05-15 13:59:31,317 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a55516edcbcc448edc5fd77e31d55d64) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:31,317 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #2) to splat34
2017-05-15 13:59:31,317 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (114fe598aed8f758e58fef63caca9d72) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:31,318 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #2) to splat34
2017-05-15 13:59:31,318 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3e600d83298e0b81658d57bef7e8536c) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:31,318 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #2) to splat34
2017-05-15 13:59:31,318 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (84b8b824b499518947ae477dad73924d) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:31,318 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #2) to splat34
2017-05-15 13:59:31,318 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (56c3498a9b9ed624d608bc75ece2142b) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:31,318 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #2) to splat34
2017-05-15 13:59:31,319 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (484eae4801564b1c5064fc74408327bc) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:31,319 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #2) to splat34
2017-05-15 13:59:31,319 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (e89114ab129b9a766bae9394d08b721a) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:31,319 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #2) to splat34
2017-05-15 13:59:31,319 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (e0ef852e975307d3794663d38f7488ca) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:31,319 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #2) to splat34
2017-05-15 13:59:31,399 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (114fe598aed8f758e58fef63caca9d72) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:31,399 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (79ded85532660a95ac8280f21ae9d8d4) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:31,400 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (fbb19bea4823fc2f0fd1482d61054a20) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:31,401 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (0bd17588f4674f754ced2562c447ed0a) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:31,401 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (006621199f1282f7ec623fadb1cb65dc) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:31,402 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bf80bf5edfc2c072fe36a61156ff935a) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:31,403 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a55516edcbcc448edc5fd77e31d55d64) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:31,404 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3e600d83298e0b81658d57bef7e8536c) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:31,405 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (84b8b824b499518947ae477dad73924d) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:31,406 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (56c3498a9b9ed624d608bc75ece2142b) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:31,407 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (484eae4801564b1c5064fc74408327bc) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:31,407 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (e89114ab129b9a766bae9394d08b721a) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:31,408 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (e0ef852e975307d3794663d38f7488ca) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:31,887 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (114fe598aed8f758e58fef63caca9d72) switched from RUNNING to FAILED.
java.lang.IllegalStateException: Could not initialize keyed state backend.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85)
at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)
... 6 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)
... 28 more
2017-05-15 13:59:31,888 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING.
java.lang.IllegalStateException: Could not initialize keyed state backend.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85)
at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)
... 6 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)
... 28 more
2017-05-15 13:59:31,890 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (0bd17588f4674f754ced2562c447ed0a) switched from RUNNING to CANCELING.
2017-05-15 13:59:31,890 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (006621199f1282f7ec623fadb1cb65dc) switched from RUNNING to CANCELING.
2017-05-15 13:59:31,890 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (79ded85532660a95ac8280f21ae9d8d4) switched from RUNNING to CANCELING.
2017-05-15 13:59:31,890 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bf80bf5edfc2c072fe36a61156ff935a) switched from RUNNING to CANCELING.
2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (fbb19bea4823fc2f0fd1482d61054a20) switched from RUNNING to CANCELING.
2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a55516edcbcc448edc5fd77e31d55d64) switched from RUNNING to CANCELING.
2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3e600d83298e0b81658d57bef7e8536c) switched from RUNNING to CANCELING.
2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (84b8b824b499518947ae477dad73924d) switched from RUNNING to CANCELING.
2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (56c3498a9b9ed624d608bc75ece2142b) switched from RUNNING to CANCELING.
2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (484eae4801564b1c5064fc74408327bc) switched from RUNNING to CANCELING.
2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (e89114ab129b9a766bae9394d08b721a) switched from RUNNING to CANCELING.
2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (e0ef852e975307d3794663d38f7488ca) switched from RUNNING to CANCELING.
2017-05-15 13:59:31,932 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (0bd17588f4674f754ced2562c447ed0a) switched from CANCELING to CANCELED.
2017-05-15 13:59:31,933 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bf80bf5edfc2c072fe36a61156ff935a) switched from CANCELING to CANCELED.
2017-05-15 13:59:31,933 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a55516edcbcc448edc5fd77e31d55d64) switched from CANCELING to CANCELED.
2017-05-15 13:59:31,934 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3e600d83298e0b81658d57bef7e8536c) switched from CANCELING to CANCELED.
2017-05-15 13:59:31,934 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (84b8b824b499518947ae477dad73924d) switched from CANCELING to CANCELED.
2017-05-15 13:59:31,935 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (56c3498a9b9ed624d608bc75ece2142b) switched from CANCELING to CANCELED.
2017-05-15 13:59:31,935 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (484eae4801564b1c5064fc74408327bc) switched from CANCELING to CANCELED.
2017-05-15 13:59:31,935 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (e89114ab129b9a766bae9394d08b721a) switched from CANCELING to CANCELED.
2017-05-15 13:59:31,935 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (e0ef852e975307d3794663d38f7488ca) switched from CANCELING to CANCELED.
2017-05-15 13:59:34,078 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (006621199f1282f7ec623fadb1cb65dc) switched from CANCELING to CANCELED.
2017-05-15 13:59:34,290 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (79ded85532660a95ac8280f21ae9d8d4) switched from CANCELING to CANCELED.
2017-05-15 13:59:35,199 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (fbb19bea4823fc2f0fd1482d61054a20) switched from CANCELING to CANCELED.
2017-05-15 13:59:35,199 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible.
2017-05-15 13:59:35,199 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING.
2017-05-15 13:59:35,199 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e).
2017-05-15 13:59:35,199 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ...
2017-05-15 13:59:35,414 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 13:59:40,434 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 13:59:45,200 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED.
2017-05-15 13:59:45,200 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper.
2017-05-15 13:59:45,204 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 1 checkpoints in ZooKeeper.
2017-05-15 13:59:45,204 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29.
2017-05-15 13:59:45,211 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29.
2017-05-15 13:59:45,214 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 29 @ 1494849550373 for 1f387e6136f6faf1824808898750878e.
2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore
2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING.
2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (31c20e3773238419f0ad99e2fb2da7fa) switched from CREATED to SCHEDULED.
2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (368276f9f3101752a52a9b3ef4ab090a) switched from CREATED to SCHEDULED.
2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (6f3eadbb122384d3c186417e7c359d4d) switched from CREATED to SCHEDULED.
2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (7507653dacef8da0f710f6eafeb5310f) switched from CREATED to SCHEDULED.
2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (3da695760bd3750f38048853db4b3ada) switched from CREATED to SCHEDULED.
2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (1f70d3824513abaa0150c020f134daf5) switched from CREATED to SCHEDULED.
2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (4bbfb4d9e406c459222e1ea8304ecf2d) switched from CREATED to SCHEDULED.
2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (94d2b1c4903a17c029b343c02afe5ed7) switched from CREATED to SCHEDULED.
2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (ddb73b0e50b2b9e6115edd93a2ee8314) switched from CREATED to SCHEDULED.
2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (0faff6f59295a616585130b79b40cf56) switched from CREATED to SCHEDULED.
2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (9b9e596b89cc450adb0d8d8106c5de23) switched from CREATED to SCHEDULED.
2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (96710b7b73e8b145284d469017c94619) switched from CREATED to SCHEDULED.
2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (48ccd602fbdf5dbf99f2fcb035395b31) switched from CREATED to SCHEDULED.
2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (31c20e3773238419f0ad99e2fb2da7fa) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #3) to splat34
2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (368276f9f3101752a52a9b3ef4ab090a) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #3) to splat34
2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (6f3eadbb122384d3c186417e7c359d4d) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #3) to splat34
2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (7507653dacef8da0f710f6eafeb5310f) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #3) to splat34
2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (3da695760bd3750f38048853db4b3ada) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #3) to splat34
2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (1f70d3824513abaa0150c020f134daf5) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #3) to splat34
2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (4bbfb4d9e406c459222e1ea8304ecf2d) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #3) to splat34
2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (94d2b1c4903a17c029b343c02afe5ed7) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #3) to splat34
2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (ddb73b0e50b2b9e6115edd93a2ee8314) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #3) to splat34
2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (0faff6f59295a616585130b79b40cf56) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #3) to splat34
2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (9b9e596b89cc450adb0d8d8106c5de23) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #3) to splat34
2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (96710b7b73e8b145284d469017c94619) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #3) to splat34
2017-05-15 13:59:45,219 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (48ccd602fbdf5dbf99f2fcb035395b31) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:45,219 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #3) to splat34
2017-05-15 13:59:45,233 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (368276f9f3101752a52a9b3ef4ab090a) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:45,233 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (31c20e3773238419f0ad99e2fb2da7fa) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:45,235 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (6f3eadbb122384d3c186417e7c359d4d) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:45,236 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (7507653dacef8da0f710f6eafeb5310f) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:45,237 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (3da695760bd3750f38048853db4b3ada) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:45,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (1f70d3824513abaa0150c020f134daf5) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:45,246 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (4bbfb4d9e406c459222e1ea8304ecf2d) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:45,253 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (94d2b1c4903a17c029b343c02afe5ed7) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:45,255 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (ddb73b0e50b2b9e6115edd93a2ee8314) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:45,261 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (0faff6f59295a616585130b79b40cf56) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:45,263 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (9b9e596b89cc450adb0d8d8106c5de23) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:45,264 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (96710b7b73e8b145284d469017c94619) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:45,264 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (48ccd602fbdf5dbf99f2fcb035395b31) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:45,329 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (4bbfb4d9e406c459222e1ea8304ecf2d) switched from RUNNING to FAILED.
java.lang.IllegalStateException: Could not initialize keyed state backend.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85)
at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)
... 6 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)
... 28 more
2017-05-15 13:59:45,329 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING.
java.lang.IllegalStateException: Could not initialize keyed state backend.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85)
at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)
... 6 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)
... 28 more
2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (31c20e3773238419f0ad99e2fb2da7fa) switched from RUNNING to CANCELING.
2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (368276f9f3101752a52a9b3ef4ab090a) switched from RUNNING to CANCELING.
2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (6f3eadbb122384d3c186417e7c359d4d) switched from RUNNING to CANCELING.
2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (7507653dacef8da0f710f6eafeb5310f) switched from RUNNING to CANCELING.
2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (3da695760bd3750f38048853db4b3ada) switched from RUNNING to CANCELING.
2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (1f70d3824513abaa0150c020f134daf5) switched from RUNNING to CANCELING.
2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (94d2b1c4903a17c029b343c02afe5ed7) switched from RUNNING to CANCELING.
2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (ddb73b0e50b2b9e6115edd93a2ee8314) switched from RUNNING to CANCELING.
2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (0faff6f59295a616585130b79b40cf56) switched from RUNNING to CANCELING.
2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (9b9e596b89cc450adb0d8d8106c5de23) switched from RUNNING to CANCELING.
2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (96710b7b73e8b145284d469017c94619) switched from RUNNING to CANCELING.
2017-05-15 13:59:45,332 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (48ccd602fbdf5dbf99f2fcb035395b31) switched from RUNNING to CANCELING.
2017-05-15 13:59:45,360 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (31c20e3773238419f0ad99e2fb2da7fa) switched from CANCELING to CANCELED.
2017-05-15 13:59:45,361 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (7507653dacef8da0f710f6eafeb5310f) switched from CANCELING to CANCELED.
2017-05-15 13:59:45,362 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (1f70d3824513abaa0150c020f134daf5) switched from CANCELING to CANCELED.
2017-05-15 13:59:45,362 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (94d2b1c4903a17c029b343c02afe5ed7) switched from CANCELING to CANCELED.
2017-05-15 13:59:45,363 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (ddb73b0e50b2b9e6115edd93a2ee8314) switched from CANCELING to CANCELED.
2017-05-15 13:59:45,363 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (0faff6f59295a616585130b79b40cf56) switched from CANCELING to CANCELED.
2017-05-15 13:59:45,364 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (9b9e596b89cc450adb0d8d8106c5de23) switched from CANCELING to CANCELED.
2017-05-15 13:59:45,365 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (96710b7b73e8b145284d469017c94619) switched from CANCELING to CANCELED.
2017-05-15 13:59:45,365 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (48ccd602fbdf5dbf99f2fcb035395b31) switched from CANCELING to CANCELED.
2017-05-15 13:59:45,453 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 13:59:46,029 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (6f3eadbb122384d3c186417e7c359d4d) switched from CANCELING to CANCELED.
2017-05-15 13:59:46,253 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (368276f9f3101752a52a9b3ef4ab090a) switched from CANCELING to CANCELED.
2017-05-15 13:59:46,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (3da695760bd3750f38048853db4b3ada) switched from CANCELING to CANCELED.
2017-05-15 13:59:46,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible.
2017-05-15 13:59:46,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING.
2017-05-15 13:59:46,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e).
2017-05-15 13:59:46,332 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ...
2017-05-15 13:59:50,474 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 13:59:55,494 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 13:59:56,332 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED.
2017-05-15 13:59:56,332 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper.
2017-05-15 13:59:56,334 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 1 checkpoints in ZooKeeper.
2017-05-15 13:59:56,334 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29.
2017-05-15 13:59:56,341 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29.
2017-05-15 13:59:56,345 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 29 @ 1494849550373 for 1f387e6136f6faf1824808898750878e.
2017-05-15 13:59:56,345 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore
2017-05-15 13:59:56,345 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING.
2017-05-15 13:59:56,345 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (6b4d03aa9e6202c7dac653e1d6b34047) switched from CREATED to SCHEDULED.
2017-05-15 13:59:56,345 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (4d136e14c501f525db8d65b3fa800cd9) switched from CREATED to SCHEDULED.
2017-05-15 13:59:56,345 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (492faa5407fe4b7e3d9a84d886180b0e) switched from CREATED to SCHEDULED.
2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (6be80899f60475fc6b3a25f5e78d51fa) switched from CREATED to SCHEDULED.
2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (22665b351c702c2d90d601b97c8f4020) switched from CREATED to SCHEDULED.
2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a0f5b8bc2c01d52b595fc4301ecc9a38) switched from CREATED to SCHEDULED.
2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (251dad84bbd0cdfd36e6dd5845925231) switched from CREATED to SCHEDULED.
2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (dcba9058b8b1323d726246f33de13a78) switched from CREATED to SCHEDULED.
2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (80f9ee6b489b0afaa4f0539d5d15297a) switched from CREATED to SCHEDULED.
2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (471d42853a1b07df565bdb20dadc5168) switched from CREATED to SCHEDULED.
2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (c8546b82cdb3269499d7580845e5d039) switched from CREATED to SCHEDULED.
2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (c12496053493ec64c7d3ca72a79abbf4) switched from CREATED to SCHEDULED.
2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (cee78d5edb7d161f1be2742a46a52f2f) switched from CREATED to SCHEDULED.
2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (6b4d03aa9e6202c7dac653e1d6b34047) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #4) to splat34
2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (4d136e14c501f525db8d65b3fa800cd9) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #4) to splat34
2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (492faa5407fe4b7e3d9a84d886180b0e) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #4) to splat34
2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (6be80899f60475fc6b3a25f5e78d51fa) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #4) to splat34
2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (22665b351c702c2d90d601b97c8f4020) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #4) to splat34
2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a0f5b8bc2c01d52b595fc4301ecc9a38) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #4) to splat34
2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (251dad84bbd0cdfd36e6dd5845925231) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #4) to splat34
2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (dcba9058b8b1323d726246f33de13a78) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #4) to splat34
2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (80f9ee6b489b0afaa4f0539d5d15297a) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #4) to splat34
2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (471d42853a1b07df565bdb20dadc5168) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #4) to splat34
2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (c8546b82cdb3269499d7580845e5d039) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #4) to splat34
2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (c12496053493ec64c7d3ca72a79abbf4) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #4) to splat34
2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (cee78d5edb7d161f1be2742a46a52f2f) switched from SCHEDULED to DEPLOYING.
2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #4) to splat34
2017-05-15 13:59:56,360 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (6b4d03aa9e6202c7dac653e1d6b34047) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:56,360 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (4d136e14c501f525db8d65b3fa800cd9) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:56,366 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (492faa5407fe4b7e3d9a84d886180b0e) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:56,369 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (6be80899f60475fc6b3a25f5e78d51fa) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:56,370 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (22665b351c702c2d90d601b97c8f4020) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:56,372 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a0f5b8bc2c01d52b595fc4301ecc9a38) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:56,373 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (251dad84bbd0cdfd36e6dd5845925231) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:56,374 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (dcba9058b8b1323d726246f33de13a78) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:56,375 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (80f9ee6b489b0afaa4f0539d5d15297a) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:56,376 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (471d42853a1b07df565bdb20dadc5168) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:56,380 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (c8546b82cdb3269499d7580845e5d039) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:56,381 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (c12496053493ec64c7d3ca72a79abbf4) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:56,382 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (cee78d5edb7d161f1be2742a46a52f2f) switched from DEPLOYING to RUNNING.
2017-05-15 13:59:56,453 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (251dad84bbd0cdfd36e6dd5845925231) switched from RUNNING to FAILED.
java.lang.IllegalStateException: Could not initialize keyed state backend.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85)
at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)
... 6 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)
... 28 more
2017-05-15 13:59:56,454 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING.
java.lang.IllegalStateException: Could not initialize keyed state backend.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85)
at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)
... 6 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)
... 28 more
2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (6b4d03aa9e6202c7dac653e1d6b34047) switched from RUNNING to CANCELING.
2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (4d136e14c501f525db8d65b3fa800cd9) switched from RUNNING to CANCELING.
2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (492faa5407fe4b7e3d9a84d886180b0e) switched from RUNNING to CANCELING.
2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (6be80899f60475fc6b3a25f5e78d51fa) switched from RUNNING to CANCELING.
2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (22665b351c702c2d90d601b97c8f4020) switched from RUNNING to CANCELING.
2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a0f5b8bc2c01d52b595fc4301ecc9a38) switched from RUNNING to CANCELING.
2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (dcba9058b8b1323d726246f33de13a78) switched from RUNNING to CANCELING.
2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (80f9ee6b489b0afaa4f0539d5d15297a) switched from RUNNING to CANCELING.
2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (471d42853a1b07df565bdb20dadc5168) switched from RUNNING to CANCELING.
2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (c8546b82cdb3269499d7580845e5d039) switched from RUNNING to CANCELING.
2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (c12496053493ec64c7d3ca72a79abbf4) switched from RUNNING to CANCELING.
2017-05-15 13:59:56,457 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (cee78d5edb7d161f1be2742a46a52f2f) switched from RUNNING to CANCELING.
2017-05-15 13:59:56,475 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (6b4d03aa9e6202c7dac653e1d6b34047) switched from CANCELING to CANCELED.
2017-05-15 13:59:56,477 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (6be80899f60475fc6b3a25f5e78d51fa) switched from CANCELING to CANCELED.
2017-05-15 13:59:56,477 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a0f5b8bc2c01d52b595fc4301ecc9a38) switched from CANCELING to CANCELED.
2017-05-15 13:59:56,478 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (dcba9058b8b1323d726246f33de13a78) switched from CANCELING to CANCELED.
2017-05-15 13:59:56,479 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (80f9ee6b489b0afaa4f0539d5d15297a) switched from CANCELING to CANCELED.
2017-05-15 13:59:56,479 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (c8546b82cdb3269499d7580845e5d039) switched from CANCELING to CANCELED.
2017-05-15 13:59:56,480 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (471d42853a1b07df565bdb20dadc5168) switched from CANCELING to CANCELED.
2017-05-15 13:59:56,480 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (c12496053493ec64c7d3ca72a79abbf4) switched from CANCELING to CANCELED.
2017-05-15 13:59:56,480 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (cee78d5edb7d161f1be2742a46a52f2f) switched from CANCELING to CANCELED.
2017-05-15 13:59:57,029 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (492faa5407fe4b7e3d9a84d886180b0e) switched from CANCELING to CANCELED.
2017-05-15 13:59:57,044 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (4d136e14c501f525db8d65b3fa800cd9) switched from CANCELING to CANCELED.
2017-05-15 13:59:57,228 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (22665b351c702c2d90d601b97c8f4020) switched from CANCELING to CANCELED.
2017-05-15 13:59:57,228 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible.
2017-05-15 13:59:57,228 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING.
2017-05-15 13:59:57,228 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e).
2017-05-15 13:59:57,228 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ...
2017-05-15 14:00:00,513 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 14:00:05,534 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 14:00:07,228 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED.
2017-05-15 14:00:07,228 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper.
2017-05-15 14:00:07,230 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 1 checkpoints in ZooKeeper.
2017-05-15 14:00:07,230 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29.
2017-05-15 14:00:07,236 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29.
2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 29 @ 1494849550373 for 1f387e6136f6faf1824808898750878e.
2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore
2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING.
2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (9a3aa7d3fc1ee24194edc1f4323d19d2) switched from CREATED to SCHEDULED.
2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96f3c203d94359531e06f628016ada20) switched from CREATED to SCHEDULED.
2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d0a5fda1fc993891b3cb688365ebe1f8) switched from CREATED to SCHEDULED.
2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (352737692e7062ccf73d6f3bca65ec57) switched from CREATED to SCHEDULED.
2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (c02268503a51fb91c81a455663598720) switched from CREATED to SCHEDULED.
2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (92d265fad55c908b077782341a0cf51f) switched from CREATED to SCHEDULED.
2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (d4c3b674b6eef5254d07f67204b4ac4b) switched from CREATED to SCHEDULED.
2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (9d336b1b9fb2b473ff1101f0b7aef44b) switched from CREATED to SCHEDULED.
2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (b38c36dc40c13d520c16b73364fc693b) switched from CREATED to SCHEDULED.
2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (bfe2339eb2ad2ccd3ed65fea417766af) switched from CREATED to SCHEDULED.
2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (4d610b0414eb5ac25e8c36cb0ee89ef7) switched from CREATED to SCHEDULED.
2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (7fa7764d87ad8cc4affa6aa5873762a4) switched from CREATED to SCHEDULED.
2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7c46499da6faf343f474bebb0bd85c4b) switched from CREATED to SCHEDULED.
2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (9a3aa7d3fc1ee24194edc1f4323d19d2) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #5) to splat34
2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96f3c203d94359531e06f628016ada20) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #5) to splat34
2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d0a5fda1fc993891b3cb688365ebe1f8) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #5) to splat34
2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (352737692e7062ccf73d6f3bca65ec57) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #5) to splat34
2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (c02268503a51fb91c81a455663598720) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #5) to splat34
2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (92d265fad55c908b077782341a0cf51f) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #5) to splat34
2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (d4c3b674b6eef5254d07f67204b4ac4b) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #5) to splat34
2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (9d336b1b9fb2b473ff1101f0b7aef44b) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #5) to splat34
2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (b38c36dc40c13d520c16b73364fc693b) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #5) to splat34
2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (bfe2339eb2ad2ccd3ed65fea417766af) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #5) to splat34
2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (4d610b0414eb5ac25e8c36cb0ee89ef7) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #5) to splat34
2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (7fa7764d87ad8cc4affa6aa5873762a4) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #5) to splat34
2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7c46499da6faf343f474bebb0bd85c4b) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #5) to splat34
2017-05-15 14:00:07,251 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (9a3aa7d3fc1ee24194edc1f4323d19d2) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:07,251 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96f3c203d94359531e06f628016ada20) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:07,253 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d0a5fda1fc993891b3cb688365ebe1f8) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:07,253 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (352737692e7062ccf73d6f3bca65ec57) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:07,254 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (c02268503a51fb91c81a455663598720) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:07,255 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (92d265fad55c908b077782341a0cf51f) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:07,255 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (d4c3b674b6eef5254d07f67204b4ac4b) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:07,256 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (9d336b1b9fb2b473ff1101f0b7aef44b) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:07,261 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (b38c36dc40c13d520c16b73364fc693b) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:07,263 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (bfe2339eb2ad2ccd3ed65fea417766af) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:07,267 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (4d610b0414eb5ac25e8c36cb0ee89ef7) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:07,269 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (7fa7764d87ad8cc4affa6aa5873762a4) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:07,269 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7c46499da6faf343f474bebb0bd85c4b) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:07,335 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (d4c3b674b6eef5254d07f67204b4ac4b) switched from RUNNING to FAILED.
java.lang.IllegalStateException: Could not initialize keyed state backend.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85)
at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)
... 6 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)
... 28 more
2017-05-15 14:00:07,336 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING.
java.lang.IllegalStateException: Could not initialize keyed state backend.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85)
at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)
... 6 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)
... 28 more
2017-05-15 14:00:07,337 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (9a3aa7d3fc1ee24194edc1f4323d19d2) switched from RUNNING to CANCELING.
2017-05-15 14:00:07,337 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96f3c203d94359531e06f628016ada20) switched from RUNNING to CANCELING.
2017-05-15 14:00:07,337 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d0a5fda1fc993891b3cb688365ebe1f8) switched from RUNNING to CANCELING.
2017-05-15 14:00:07,337 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (352737692e7062ccf73d6f3bca65ec57) switched from RUNNING to CANCELING.
2017-05-15 14:00:07,337 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (c02268503a51fb91c81a455663598720) switched from RUNNING to CANCELING.
2017-05-15 14:00:07,338 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (92d265fad55c908b077782341a0cf51f) switched from RUNNING to CANCELING.
2017-05-15 14:00:07,338 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (9d336b1b9fb2b473ff1101f0b7aef44b) switched from RUNNING to CANCELING.
2017-05-15 14:00:07,338 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (b38c36dc40c13d520c16b73364fc693b) switched from RUNNING to CANCELING.
2017-05-15 14:00:07,338 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (bfe2339eb2ad2ccd3ed65fea417766af) switched from RUNNING to CANCELING.
2017-05-15 14:00:07,338 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (4d610b0414eb5ac25e8c36cb0ee89ef7) switched from RUNNING to CANCELING.
2017-05-15 14:00:07,338 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (7fa7764d87ad8cc4affa6aa5873762a4) switched from RUNNING to CANCELING.
2017-05-15 14:00:07,338 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7c46499da6faf343f474bebb0bd85c4b) switched from RUNNING to CANCELING.
2017-05-15 14:00:07,375 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (9a3aa7d3fc1ee24194edc1f4323d19d2) switched from CANCELING to CANCELED.
2017-05-15 14:00:07,376 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (352737692e7062ccf73d6f3bca65ec57) switched from CANCELING to CANCELED.
2017-05-15 14:00:07,377 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (92d265fad55c908b077782341a0cf51f) switched from CANCELING to CANCELED.
2017-05-15 14:00:07,377 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (9d336b1b9fb2b473ff1101f0b7aef44b) switched from CANCELING to CANCELED.
2017-05-15 14:00:07,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (b38c36dc40c13d520c16b73364fc693b) switched from CANCELING to CANCELED.
2017-05-15 14:00:07,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (4d610b0414eb5ac25e8c36cb0ee89ef7) switched from CANCELING to CANCELED.
2017-05-15 14:00:07,379 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (bfe2339eb2ad2ccd3ed65fea417766af) switched from CANCELING to CANCELED.
2017-05-15 14:00:07,380 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (7fa7764d87ad8cc4affa6aa5873762a4) switched from CANCELING to CANCELED.
2017-05-15 14:00:07,380 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7c46499da6faf343f474bebb0bd85c4b) switched from CANCELING to CANCELED.
2017-05-15 14:00:08,275 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96f3c203d94359531e06f628016ada20) switched from CANCELING to CANCELED.
2017-05-15 14:00:08,475 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d0a5fda1fc993891b3cb688365ebe1f8) switched from CANCELING to CANCELED.
2017-05-15 14:00:08,476 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (c02268503a51fb91c81a455663598720) switched from CANCELING to CANCELED.
2017-05-15 14:00:08,476 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible.
2017-05-15 14:00:08,476 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING.
2017-05-15 14:00:08,476 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e).
2017-05-15 14:00:08,476 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ...
2017-05-15 14:00:10,554 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 14:00:15,574 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 14:00:18,476 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED.
2017-05-15 14:00:18,476 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper.
2017-05-15 14:00:18,478 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 1 checkpoints in ZooKeeper.
2017-05-15 14:00:18,478 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29.
2017-05-15 14:00:18,484 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29.
2017-05-15 14:00:18,486 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 29 @ 1494849550373 for 1f387e6136f6faf1824808898750878e.
2017-05-15 14:00:18,487 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore
2017-05-15 14:00:18,487 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING.
2017-05-15 14:00:18,487 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (aa487e650bf0d4ea44955c44c585c488) switched from CREATED to SCHEDULED.
2017-05-15 14:00:18,487 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (1e0cf112d2d80c488174a95ae7785245) switched from CREATED to SCHEDULED.
2017-05-15 14:00:18,487 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d711c61450623bb20a8738f82f086b3c) switched from CREATED to SCHEDULED.
2017-05-15 14:00:18,487 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (1adca9fbb82bdb66bce22857e11358b9) switched from CREATED to SCHEDULED.
2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (e32a60ff98ce85f35bd77d8559f884a4) switched from CREATED to SCHEDULED.
2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (694359a780a126d897e15b2fefbf02be) switched from CREATED to SCHEDULED.
2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (abfa36c27ff9ef6b40ec1dc41833ea2a) switched from CREATED to SCHEDULED.
2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (ecb484995ca035fc697cb5b7f2d6432f) switched from CREATED to SCHEDULED.
2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (892d0d66545aa33a8a31d323f86d8d3c) switched from CREATED to SCHEDULED.
2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (fea28107146851d415f3b6976474212c) switched from CREATED to SCHEDULED.
2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (ef1c63edc779c463800fbb272f4ef3ca) switched from CREATED to SCHEDULED.
2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (8d9b795aa5a8d586e3b9edc827e07c96) switched from CREATED to SCHEDULED.
2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (31aa1ccc883d064e494045e0ef9cc4c9) switched from CREATED to SCHEDULED.
2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (aa487e650bf0d4ea44955c44c585c488) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #6) to splat34
2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (1e0cf112d2d80c488174a95ae7785245) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #6) to splat34
2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d711c61450623bb20a8738f82f086b3c) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #6) to splat34
2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (1adca9fbb82bdb66bce22857e11358b9) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #6) to splat34
2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (e32a60ff98ce85f35bd77d8559f884a4) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #6) to splat34
2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (694359a780a126d897e15b2fefbf02be) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #6) to splat34
2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (abfa36c27ff9ef6b40ec1dc41833ea2a) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #6) to splat34
2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (ecb484995ca035fc697cb5b7f2d6432f) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #6) to splat34
2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (892d0d66545aa33a8a31d323f86d8d3c) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #6) to splat34
2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (fea28107146851d415f3b6976474212c) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #6) to splat34
2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (ef1c63edc779c463800fbb272f4ef3ca) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #6) to splat34
2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (8d9b795aa5a8d586e3b9edc827e07c96) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #6) to splat34
2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (31aa1ccc883d064e494045e0ef9cc4c9) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:18,491 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #6) to splat34
2017-05-15 14:00:18,508 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (aa487e650bf0d4ea44955c44c585c488) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:18,509 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (1e0cf112d2d80c488174a95ae7785245) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:18,509 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d711c61450623bb20a8738f82f086b3c) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:18,510 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (1adca9fbb82bdb66bce22857e11358b9) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:18,511 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (e32a60ff98ce85f35bd77d8559f884a4) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:18,511 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (694359a780a126d897e15b2fefbf02be) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:18,511 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (abfa36c27ff9ef6b40ec1dc41833ea2a) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:18,512 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (ecb484995ca035fc697cb5b7f2d6432f) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:18,512 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (892d0d66545aa33a8a31d323f86d8d3c) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:18,512 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (fea28107146851d415f3b6976474212c) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:18,512 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (ef1c63edc779c463800fbb272f4ef3ca) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:18,513 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (8d9b795aa5a8d586e3b9edc827e07c96) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:18,513 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (31aa1ccc883d064e494045e0ef9cc4c9) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:18,559 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (abfa36c27ff9ef6b40ec1dc41833ea2a) switched from RUNNING to FAILED.
java.lang.IllegalStateException: Could not initialize keyed state backend.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85)
at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)
... 6 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)
... 28 more
2017-05-15 14:00:18,560 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING.
java.lang.IllegalStateException: Could not initialize keyed state backend.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85)
at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)
... 6 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)
... 28 more
2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (aa487e650bf0d4ea44955c44c585c488) switched from RUNNING to CANCELING.
2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (1e0cf112d2d80c488174a95ae7785245) switched from RUNNING to CANCELING.
2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d711c61450623bb20a8738f82f086b3c) switched from RUNNING to CANCELING.
2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (1adca9fbb82bdb66bce22857e11358b9) switched from RUNNING to CANCELING.
2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (e32a60ff98ce85f35bd77d8559f884a4) switched from RUNNING to CANCELING.
2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (694359a780a126d897e15b2fefbf02be) switched from RUNNING to CANCELING.
2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (ecb484995ca035fc697cb5b7f2d6432f) switched from RUNNING to CANCELING.
2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (892d0d66545aa33a8a31d323f86d8d3c) switched from RUNNING to CANCELING.
2017-05-15 14:00:18,562 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (fea28107146851d415f3b6976474212c) switched from RUNNING to CANCELING.
2017-05-15 14:00:18,562 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (ef1c63edc779c463800fbb272f4ef3ca) switched from RUNNING to CANCELING.
2017-05-15 14:00:18,562 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (8d9b795aa5a8d586e3b9edc827e07c96) switched from RUNNING to CANCELING.
2017-05-15 14:00:18,562 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (31aa1ccc883d064e494045e0ef9cc4c9) switched from RUNNING to CANCELING.
2017-05-15 14:00:18,564 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (aa487e650bf0d4ea44955c44c585c488) switched from CANCELING to CANCELED.
2017-05-15 14:00:18,570 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (1adca9fbb82bdb66bce22857e11358b9) switched from CANCELING to CANCELED.
2017-05-15 14:00:18,572 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (694359a780a126d897e15b2fefbf02be) switched from CANCELING to CANCELED.
2017-05-15 14:00:18,573 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (ecb484995ca035fc697cb5b7f2d6432f) switched from CANCELING to CANCELED.
2017-05-15 14:00:18,574 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (892d0d66545aa33a8a31d323f86d8d3c) switched from CANCELING to CANCELED.
2017-05-15 14:00:18,574 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (8d9b795aa5a8d586e3b9edc827e07c96) switched from CANCELING to CANCELED.
2017-05-15 14:00:18,574 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (ef1c63edc779c463800fbb272f4ef3ca) switched from CANCELING to CANCELED.
2017-05-15 14:00:18,574 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (fea28107146851d415f3b6976474212c) switched from CANCELING to CANCELED.
2017-05-15 14:00:18,574 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (31aa1ccc883d064e494045e0ef9cc4c9) switched from CANCELING to CANCELED.
2017-05-15 14:00:19,373 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (e32a60ff98ce85f35bd77d8559f884a4) switched from CANCELING to CANCELED.
2017-05-15 14:00:19,373 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d711c61450623bb20a8738f82f086b3c) switched from CANCELING to CANCELED.
2017-05-15 14:00:19,640 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (1e0cf112d2d80c488174a95ae7785245) switched from CANCELING to CANCELED.
2017-05-15 14:00:19,641 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible.
2017-05-15 14:00:19,641 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING.
2017-05-15 14:00:19,641 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e).
2017-05-15 14:00:19,641 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ...
2017-05-15 14:00:20,594 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 14:00:25,615 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 14:00:29,641 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED.
2017-05-15 14:00:29,641 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper.
2017-05-15 14:00:29,643 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 1 checkpoints in ZooKeeper.
2017-05-15 14:00:29,643 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29.
2017-05-15 14:00:29,651 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29.
2017-05-15 14:00:29,655 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 29 @ 1494849550373 for 1f387e6136f6faf1824808898750878e.
2017-05-15 14:00:29,655 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore
2017-05-15 14:00:29,655 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING.
2017-05-15 14:00:29,655 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (01143eb487fbceddc5e0531b9a20e296) switched from CREATED to SCHEDULED.
2017-05-15 14:00:29,655 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96d5f86d669ad54950e9747d48b064b2) switched from CREATED to SCHEDULED.
2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7bcaeb685ce42ec560e2ecb4df1f7216) switched from CREATED to SCHEDULED.
2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (c797dfa92d7b0f02070e67b87ad8a203) switched from CREATED to SCHEDULED.
2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (a9eb5578023189d2f414c58c8219d046) switched from CREATED to SCHEDULED.
2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (34c8fe5375c64b0e4f343e64517371bd) switched from CREATED to SCHEDULED.
2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (035f512337eb68ba9f2433ece44a2d19) switched from CREATED to SCHEDULED.
2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3525adaf53e5b366d25443e77ad2bbe4) switched from CREATED to SCHEDULED.
2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (4df4c3554d2628d630716ff5a0fc57c5) switched from CREATED to SCHEDULED.
2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (78b22ef5e7356ce532756eb9063e3803) switched from CREATED to SCHEDULED.
2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3643843191636ca6c6317a1b0280e347) switched from CREATED to SCHEDULED.
2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (ed0c35d4812b8fcbcf872ee928a459a5) switched from CREATED to SCHEDULED.
2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (097d118f9a601c4a119751a4dfb05add) switched from CREATED to SCHEDULED.
2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (01143eb487fbceddc5e0531b9a20e296) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #7) to splat34
2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96d5f86d669ad54950e9747d48b064b2) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #7) to splat34
2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7bcaeb685ce42ec560e2ecb4df1f7216) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #7) to splat34
2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (c797dfa92d7b0f02070e67b87ad8a203) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #7) to splat34
2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (a9eb5578023189d2f414c58c8219d046) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #7) to splat34
2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (34c8fe5375c64b0e4f343e64517371bd) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #7) to splat34
2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (035f512337eb68ba9f2433ece44a2d19) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #7) to splat34
2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3525adaf53e5b366d25443e77ad2bbe4) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #7) to splat34
2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (4df4c3554d2628d630716ff5a0fc57c5) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #7) to splat34
2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (78b22ef5e7356ce532756eb9063e3803) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #7) to splat34
2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3643843191636ca6c6317a1b0280e347) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #7) to splat34
2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (ed0c35d4812b8fcbcf872ee928a459a5) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #7) to splat34
2017-05-15 14:00:29,659 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (097d118f9a601c4a119751a4dfb05add) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:29,659 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #7) to splat34
2017-05-15 14:00:29,690 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (01143eb487fbceddc5e0531b9a20e296) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:29,692 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96d5f86d669ad54950e9747d48b064b2) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:29,692 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7bcaeb685ce42ec560e2ecb4df1f7216) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:29,692 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (c797dfa92d7b0f02070e67b87ad8a203) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:29,693 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (a9eb5578023189d2f414c58c8219d046) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:29,693 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (34c8fe5375c64b0e4f343e64517371bd) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:29,693 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (035f512337eb68ba9f2433ece44a2d19) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:29,694 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3525adaf53e5b366d25443e77ad2bbe4) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:29,694 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (4df4c3554d2628d630716ff5a0fc57c5) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:29,695 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (78b22ef5e7356ce532756eb9063e3803) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:29,695 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3643843191636ca6c6317a1b0280e347) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:29,695 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (ed0c35d4812b8fcbcf872ee928a459a5) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:29,695 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (097d118f9a601c4a119751a4dfb05add) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:29,779 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (035f512337eb68ba9f2433ece44a2d19) switched from RUNNING to FAILED.
java.lang.IllegalStateException: Could not initialize keyed state backend.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85)
at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)
... 6 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)
... 28 more
2017-05-15 14:00:29,779 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING.
java.lang.IllegalStateException: Could not initialize keyed state backend.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85)
at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)
... 6 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)
... 28 more
2017-05-15 14:00:29,780 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (01143eb487fbceddc5e0531b9a20e296) switched from RUNNING to CANCELING.
2017-05-15 14:00:29,780 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96d5f86d669ad54950e9747d48b064b2) switched from RUNNING to CANCELING.
2017-05-15 14:00:29,780 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7bcaeb685ce42ec560e2ecb4df1f7216) switched from RUNNING to CANCELING.
2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (c797dfa92d7b0f02070e67b87ad8a203) switched from RUNNING to CANCELING.
2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (a9eb5578023189d2f414c58c8219d046) switched from RUNNING to CANCELING.
2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (34c8fe5375c64b0e4f343e64517371bd) switched from RUNNING to CANCELING.
2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3525adaf53e5b366d25443e77ad2bbe4) switched from RUNNING to CANCELING.
2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (4df4c3554d2628d630716ff5a0fc57c5) switched from RUNNING to CANCELING.
2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (78b22ef5e7356ce532756eb9063e3803) switched from RUNNING to CANCELING.
2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3643843191636ca6c6317a1b0280e347) switched from RUNNING to CANCELING.
2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (ed0c35d4812b8fcbcf872ee928a459a5) switched from RUNNING to CANCELING.
2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (097d118f9a601c4a119751a4dfb05add) switched from RUNNING to CANCELING.
2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (01143eb487fbceddc5e0531b9a20e296) switched from CANCELING to CANCELED.
2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (c797dfa92d7b0f02070e67b87ad8a203) switched from CANCELING to CANCELED.
2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (34c8fe5375c64b0e4f343e64517371bd) switched from CANCELING to CANCELED.
2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3525adaf53e5b366d25443e77ad2bbe4) switched from CANCELING to CANCELED.
2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (4df4c3554d2628d630716ff5a0fc57c5) switched from CANCELING to CANCELED.
2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3643843191636ca6c6317a1b0280e347) switched from CANCELING to CANCELED.
2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (ed0c35d4812b8fcbcf872ee928a459a5) switched from CANCELING to CANCELED.
2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (78b22ef5e7356ce532756eb9063e3803) switched from CANCELING to CANCELED.
2017-05-15 14:00:29,793 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (097d118f9a601c4a119751a4dfb05add) switched from CANCELING to CANCELED.
2017-05-15 14:00:30,451 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (a9eb5578023189d2f414c58c8219d046) switched from CANCELING to CANCELED.
2017-05-15 14:00:30,550 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7bcaeb685ce42ec560e2ecb4df1f7216) switched from CANCELING to CANCELED.
2017-05-15 14:00:30,617 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96d5f86d669ad54950e9747d48b064b2) switched from CANCELING to CANCELED.
2017-05-15 14:00:30,617 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible.
2017-05-15 14:00:30,634 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 14:00:30,647 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING.
2017-05-15 14:00:30,647 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e).
2017-05-15 14:00:30,647 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ...
2017-05-15 14:00:35,655 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 14:00:40,647 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED.
2017-05-15 14:00:40,647 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper.
2017-05-15 14:00:40,651 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 1 checkpoints in ZooKeeper.
2017-05-15 14:00:40,651 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29.
2017-05-15 14:00:40,658 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29.
2017-05-15 14:00:40,661 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 29 @ 1494849550373 for 1f387e6136f6faf1824808898750878e.
2017-05-15 14:00:40,662 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore
2017-05-15 14:00:40,662 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING.
2017-05-15 14:00:40,662 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (557342eb5035b03d66578c9928cc4cdd) switched from CREATED to SCHEDULED.
2017-05-15 14:00:40,662 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (f5901179c871f609f87d7b2dbdf08689) switched from CREATED to SCHEDULED.
2017-05-15 14:00:40,662 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (a891ce4e388ef6c253be967cdc1b02bf) switched from CREATED to SCHEDULED.
2017-05-15 14:00:40,662 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (b849a4add5d6aecea749d725dfedbf95) switched from CREATED to SCHEDULED.
2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (bd514206cb90c8c7af5920616d3d4de4) switched from CREATED to SCHEDULED.
2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (515e891fa796a2a6e205c851fd7a550a) switched from CREATED to SCHEDULED.
2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (ef42c4911c40011dc26cd1a03e490eea) switched from CREATED to SCHEDULED.
2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (1e860c7508dd154388e4aa46bd460037) switched from CREATED to SCHEDULED.
2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (2a66d35d8618b87698cf83ac982770f2) switched from CREATED to SCHEDULED.
2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (c410ce8973e8027cd1b297f2ad5f6930) switched from CREATED to SCHEDULED.
2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3d5dad52595a9deb8aebf08d815a3620) switched from CREATED to SCHEDULED.
2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (67a553e88df17ecbace5821563b1bb4c) switched from CREATED to SCHEDULED.
2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (684a8d45c657df79a93db15a866e141f) switched from CREATED to SCHEDULED.
2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (557342eb5035b03d66578c9928cc4cdd) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #8) to splat34
2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (f5901179c871f609f87d7b2dbdf08689) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #8) to splat34
2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (a891ce4e388ef6c253be967cdc1b02bf) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #8) to splat34
2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (b849a4add5d6aecea749d725dfedbf95) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #8) to splat34
2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (bd514206cb90c8c7af5920616d3d4de4) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #8) to splat34
2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (515e891fa796a2a6e205c851fd7a550a) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #8) to splat34
2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (ef42c4911c40011dc26cd1a03e490eea) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #8) to splat34
2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (1e860c7508dd154388e4aa46bd460037) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #8) to splat34
2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (2a66d35d8618b87698cf83ac982770f2) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #8) to splat34
2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (c410ce8973e8027cd1b297f2ad5f6930) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #8) to splat34
2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3d5dad52595a9deb8aebf08d815a3620) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #8) to splat34
2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (67a553e88df17ecbace5821563b1bb4c) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #8) to splat34
2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (684a8d45c657df79a93db15a866e141f) switched from SCHEDULED to DEPLOYING.
2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #8) to splat34
2017-05-15 14:00:40,668 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (557342eb5035b03d66578c9928cc4cdd) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:40,668 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (f5901179c871f609f87d7b2dbdf08689) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:40,669 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (a891ce4e388ef6c253be967cdc1b02bf) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:40,670 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (b849a4add5d6aecea749d725dfedbf95) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:40,671 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (bd514206cb90c8c7af5920616d3d4de4) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:40,671 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (515e891fa796a2a6e205c851fd7a550a) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:40,674 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 14:00:40,676 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (ef42c4911c40011dc26cd1a03e490eea) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:40,676 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (1e860c7508dd154388e4aa46bd460037) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:40,677 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (2a66d35d8618b87698cf83ac982770f2) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:40,678 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (c410ce8973e8027cd1b297f2ad5f6930) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:40,679 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3d5dad52595a9deb8aebf08d815a3620) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:40,680 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (67a553e88df17ecbace5821563b1bb4c) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:40,681 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (684a8d45c657df79a93db15a866e141f) switched from DEPLOYING to RUNNING.
2017-05-15 14:00:40,745 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (ef42c4911c40011dc26cd1a03e490eea) switched from RUNNING to FAILED.
java.lang.IllegalStateException: Could not initialize keyed state backend.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85)
at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)
... 6 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)
... 28 more
2017-05-15 14:00:40,745 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING.
java.lang.IllegalStateException: Could not initialize keyed state backend.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48)
at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85)
at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)
... 6 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220)
... 28 more
2017-05-15 14:00:40,746 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (557342eb5035b03d66578c9928cc4cdd) switched from RUNNING to CANCELING.
2017-05-15 14:00:40,746 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (f5901179c871f609f87d7b2dbdf08689) switched from RUNNING to CANCELING.
2017-05-15 14:00:40,746 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (a891ce4e388ef6c253be967cdc1b02bf) switched from RUNNING to CANCELING.
2017-05-15 14:00:40,746 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (b849a4add5d6aecea749d725dfedbf95) switched from RUNNING to CANCELING.
2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (bd514206cb90c8c7af5920616d3d4de4) switched from RUNNING to CANCELING.
2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (515e891fa796a2a6e205c851fd7a550a) switched from RUNNING to CANCELING.
2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (1e860c7508dd154388e4aa46bd460037) switched from RUNNING to CANCELING.
2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (2a66d35d8618b87698cf83ac982770f2) switched from RUNNING to CANCELING.
2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (c410ce8973e8027cd1b297f2ad5f6930) switched from RUNNING to CANCELING.
2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3d5dad52595a9deb8aebf08d815a3620) switched from RUNNING to CANCELING.
2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (67a553e88df17ecbace5821563b1bb4c) switched from RUNNING to CANCELING.
2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (684a8d45c657df79a93db15a866e141f) switched from RUNNING to CANCELING.
2017-05-15 14:00:40,756 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (557342eb5035b03d66578c9928cc4cdd) switched from CANCELING to CANCELED.
2017-05-15 14:00:40,757 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (b849a4add5d6aecea749d725dfedbf95) switched from CANCELING to CANCELED.
2017-05-15 14:00:40,757 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (515e891fa796a2a6e205c851fd7a550a) switched from CANCELING to CANCELED.
2017-05-15 14:00:40,757 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (1e860c7508dd154388e4aa46bd460037) switched from CANCELING to CANCELED.
2017-05-15 14:00:40,757 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (2a66d35d8618b87698cf83ac982770f2) switched from CANCELING to CANCELED.
2017-05-15 14:00:40,757 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3d5dad52595a9deb8aebf08d815a3620) switched from CANCELING to CANCELED.
2017-05-15 14:00:40,758 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (67a553e88df17ecbace5821563b1bb4c) switched from CANCELING to CANCELED.
2017-05-15 14:00:40,758 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (c410ce8973e8027cd1b297f2ad5f6930) switched from CANCELING to CANCELED.
2017-05-15 14:00:40,758 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (684a8d45c657df79a93db15a866e141f) switched from CANCELING to CANCELED.
2017-05-15 14:00:41,540 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (bd514206cb90c8c7af5920616d3d4de4) switched from CANCELING to CANCELED.
2017-05-15 14:00:41,639 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (a891ce4e388ef6c253be967cdc1b02bf) switched from CANCELING to CANCELED.
2017-05-15 14:00:41,904 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (f5901179c871f609f87d7b2dbdf08689) switched from CANCELING to CANCELED.
2017-05-15 14:00:41,904 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible.
2017-05-15 14:00:41,904 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING.
2017-05-15 14:00:41,904 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e).
2017-05-15 14:00:41,904 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ...
2017-05-15 14:00:43,929 INFO org.apache.flink.yarn.YarnJobManager - Trying to cancel job with ID 1f387e6136f6faf1824808898750878e.
2017-05-15 14:00:43,930 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CANCELED.
2017-05-15 14:00:43,930 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Stopping checkpoint coordinator for job 1f387e6136f6faf1824808898750878e
2017-05-15 14:00:43,930 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Shutting down
2017-05-15 14:00:43,931 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Removing /flink-splat/application_1494426363399_0021/checkpoints/1f387e6136f6faf1824808898750878e from ZooKeeper
2017-05-15 14:00:43,933 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29.
2017-05-15 14:00:43,936 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter - Shutting down.
2017-05-15 14:00:43,936 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter - Removing /checkpoint-counter/1f387e6136f6faf1824808898750878e from ZooKeeper
2017-05-15 14:00:43,936 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Canceled during restart.
2017-05-15 14:00:43,938 INFO org.apache.flink.runtime.checkpoint.CompletedCheckpoint - Persistent checkpoint with ID 29 at 'hdfs:/flink/external-checkpoints/bifrost/checkpoint_metadata-88c22ff47ca0' not discarded.
2017-05-15 14:00:43,982 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Removed job graph 1f387e6136f6faf1824808898750878e from ZooKeeper.
2017-05-15 14:00:45,694 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378]
2017-05-15 14:00:46,661 INFO org.apache.flink.yarn.YarnJobManager - Job with ID 1f387e6136f6faf1824808898750878e is in terminal state CANCELED. Shutting down session
2017-05-15 14:00:46,662 INFO org.apache.flink.yarn.YarnJobManager - Stopping JobManager with final application status FAILED and diagnostics: The monitored job with ID 1f387e6136f6faf1824808898750878e has failed to complete.
2017-05-15 14:00:46,663 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Shutting down cluster with status FAILED : The monitored job with ID 1f387e6136f6faf1824808898750878e has failed to complete.
2017-05-15 14:00:46,663 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Unregistering application from the YARN Resource Manager
2017-05-15 14:00:46,669 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Waiting for application to be successfully unregistered.
2017-05-15 14:00:46,771 INFO org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl - Interrupted while waiting for queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
2017-05-15 14:00:46,771 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : splat35.sto.midasplayer.com:8041
2017-05-15 14:00:46,787 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : splat34.sto.midasplayer.com:8041
2017-05-15 14:00:46,798 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Stopping ZooKeeperLeaderRetrievalService.
2017-05-15 14:00:46,798 INFO org.apache.flink.yarn.YarnJobManager - Task manager akka.tcp://flink@splat34.sto.midasplayer.com:53480/user/taskmanager wants to disconnect, because TaskManager akka://flink/user/taskmanager is disassociating.
2017-05-15 14:00:46,798 INFO org.apache.flink.runtime.instance.InstanceManager - Unregistered task manager splat34.sto.midasplayer.com/172.26.83.103. Number of registered task managers 0. Number of available slots 0.
2017-05-15 14:00:46,799 INFO org.apache.flink.yarn.YarnJobManager - Stopping JobManager akka.tcp://flink@splat34.sto.midasplayer.com:33785/user/jobmanager.
2017-05-15 14:00:46,801 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Stopping ZooKeeperLeaderElectionService org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService@5ebb8b4b.
2017-05-15 14:00:46,804 INFO org.apache.flink.runtime.blob.BlobServer - Stopped BLOB server at 0.0.0.0:41349
2017-05-15 14:00:46,807 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon.
2017-05-15 14:00:46,809 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports.
2017-05-15 14:00:46,829 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting shut down.
2017-05-15 14:00:46,845 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Stopping ZooKeeperLeaderRetrievalService.
2017-05-15 14:00:46,845 INFO org.apache.flink.yarn.YarnJobManager - Shutdown completed. Stopping JVM.
2017-05-15 14:00:46,847 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Removing web dashboard root cache directory /tmp/flink-web-cb0ef96f-0a47-412c-8e82-43b4bdb6dc61
2017-05-15 14:00:46,848 INFO org.apache.flink.runtime.webmonitor.StackTraceSampleCoordinator - Shutting down stack trace sample coordinator.
2017-05-15 14:00:46,851 INFO org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2017-05-15 14:00:46,852 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Removing web dashboard jar upload directory /tmp/flink-web-d05454bc-860d-4b6e-a694-2306a5b962ef
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment