Created
May 15, 2017 12:01
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Logged in as: dr.who | |
| Application | |
| About | |
| Jobs | |
| Tools | |
| Log Type: jobmanager.log | |
| Log Upload Time: Mon May 15 14:00:48 +0200 2017 | |
| Log Length: 438303 | |
| 2017-05-15 13:29:57,186 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -------------------------------------------------------------------------------- | |
| 2017-05-15 13:29:57,187 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Starting YARN ApplicationMaster / ResourceManager / JobManager (Version: 1.3-SNAPSHOT, Rev:29a221b, Date:15.05.2017 @ 11:22:12 CEST) | |
| 2017-05-15 13:29:57,198 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Current user: splat | |
| 2017-05-15 13:29:57,198 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.131-b11 | |
| 2017-05-15 13:29:57,198 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Maximum heap size: 406 MiBytes | |
| 2017-05-15 13:29:57,198 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - JAVA_HOME: /fjord/java/ | |
| 2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Hadoop version: 2.6.0 | |
| 2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - JVM Options: | |
| 2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -Xmx424m | |
| 2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -Dlog.file=/fjord/hadoop/data/3/yarn/container-logs/application_1494426363399_0021/container_1494426363399_0021_01_000001/jobmanager.log | |
| 2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -Dlogback.configurationFile=file:logback.xml | |
| 2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -Dlog4j.configuration=file:log4j.properties | |
| 2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Program Arguments: (none) | |
| 2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Classpath: lib/flink-connector-kafka-0.8_2.10-1.3-SNAPSHOT.jar:lib/flink-connector-kafka-base_2.10-1.3-SNAPSHOT.jar:lib/flink-dist_2.10-1.3-SNAPSHOT.jar:lib/flink-python_2.10-1.3-SNAPSHOT.jar:lib/flink-shaded-hadoop2-1.3-SNAPSHOT.jar:lib/kafka-clients-0.8.2.2.jar:lib/kafka_2.10-0.8.2.2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:rbea-on-flink-2.0-SNAPSHOT.jar:flink.jar:flink-conf.yaml::/etc/hadoop/conf.cloudera.yarn2:/run/cloudera-scm-agent/process/948-yarn-NODEMANAGER:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-protobuf.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-generator.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-cascading.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-auth.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-auth-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-aws-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-thrift.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-azure-datalake.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-nfs.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-avro.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-annotations-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-azure-datalake-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-test-hadoop2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-encoding.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-nfs-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-jackson.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-pig-bundle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-column.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-format.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-scrooge_2.10.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-format-sources.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-annotations.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-hadoop-bundle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-tools.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common-2.6.0-cdh5.11.0-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-hadoop.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-pig.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-aws.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-scala_2.10.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-format-javadoc.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-codec-1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jersey-core-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/slf4j-log4j12.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/servlet-api-2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-digester-1.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/curator-client-2.7.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/xz-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jetty-util-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/activation-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/curator-recipes-2.7.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/asm-3.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/stax-api-1.0-2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-dynamodb-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/guava-11.0.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jsp-api-2.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/logredactor-1.0.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/httpcore-4.2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-kms-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/hamcrest-core-1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jets3t-0.9.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-s3-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/avro.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/slf4j-api-1.7.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-el-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/netty-3.10.5.Final.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-math3-3.1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/log4j-1.2.17.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/gson-2.2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/azure-data-lake-store-sdk-2.1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-io-2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-core-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-compress-1.4.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-logging-1.1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/curator-framework-2.7.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jsch-0.1.42.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-beanutils-1.9.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jaxb-api-2.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-collections-3.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/protobuf-java-2.5.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-sts-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jetty-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/httpclient-4.2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-lang-2.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jersey-server-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-configuration-1.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jsr305-3.0.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jackson-xc-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jettison-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/hue-plugins-3.9.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/zookeeper.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/junit-4.11.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-net-3.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/htrace-core4-4.0.1-incubating.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-cli-1.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/api-util-1.0.0-M20.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/paranamer-2.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jersey-json-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/xmlenc-0.52.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/mockito-all-1.8.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-httpclient-3.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-2.6.0-cdh5.11.0-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/asm-3.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/netty-3.10.5.Final.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/htrace-core4-4.0.1-incubating.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-client.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-api-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-registry-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-registry.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-client-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-api.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-common-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-tests-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-common-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/javax.inject-1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/xz-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jetty-util-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/activation-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/asm-3.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/guava-11.0.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/spark-yarn-shuffle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-core-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jline-2.11.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-io-2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/guice-3.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-mapper-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/spark-1.6.0-cdh5.11.0-yarn-shuffle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jetty-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-xc-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-jaxrs-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jettison-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/zookeeper.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-json-1.9.jar | |
| 2017-05-15 13:29:57,200 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -------------------------------------------------------------------------------- | |
| 2017-05-15 13:29:57,201 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Registered UNIX signal handlers for [TERM, HUP, INT] | |
| 2017-05-15 13:29:57,206 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - remoteKeytabPrincipal obtained null | |
| 2017-05-15 13:29:57,206 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - YARN daemon is running as: splat Yarn client user obtainer: splat | |
| 2017-05-15 13:29:57,210 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Loading config from directory /fjord/hadoop/data/nvme/splat/yarn-local-dir/usercache/splat/appcache/application_1494426363399_0021/container_1494426363399_0021_01_000001 | |
| 2017-05-15 13:29:57,212 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.ask.timeout, 120 s | |
| 2017-05-15 13:29:57,212 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.client.timeout, 1200 s | |
| 2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.lookup.timeout, 120 s | |
| 2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.log.dir, /home/splat/flink/log/ | |
| 2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.hdfs.hadoopconf, /home/splat/rbea-on-flink/deployment/environments/test/yarn-conf | |
| 2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 | |
| 2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.mode, zookeeper | |
| 2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.zookeeper.path.root, /flink-splat | |
| 2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.zookeeper.quorum, zk04.sto.midasplayer.com:2181,zk05.sto.midasplayer.com:2181,zk06.sto.midasplayer.com:2181 | |
| 2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.zookeeper.storageDir, hdfs:///flink/ha | |
| 2017-05-15 13:29:57,213 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.checkpoints.dir, hdfs:///flink/external-checkpoints/bifrost | |
| 2017-05-15 13:29:57,214 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.savepoints.dir, hdfs:///flink/external-checkpoints/bifrost | |
| 2017-05-15 13:29:57,214 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.application-attempts, 10 | |
| 2017-05-15 13:29:57,214 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.containers.vcores, 4 | |
| 2017-05-15 13:29:57,214 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.heap-cutoff-ratio, 0.1 | |
| 2017-05-15 13:29:57,214 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.maximum-failed-containers, 100 | |
| 2017-05-15 13:29:57,244 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to splat (auth:SIMPLE) | |
| 2017-05-15 13:29:57,265 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - YARN assigned hostname for application master: splat34.sto.midasplayer.com | |
| 2017-05-15 13:29:57,270 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TaskManagers will be created with 1 task slots | |
| 2017-05-15 13:29:57,270 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TaskManagers will be started with container size 5000 MB, JVM heap size 4400 MB, JVM direct memory limit -1 MB | |
| 2017-05-15 13:29:57,274 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Trying to start actor system at splat34.sto.midasplayer.com:33785 | |
| 2017-05-15 13:29:57,618 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started | |
| 2017-05-15 13:29:57,651 INFO Remoting - Starting remoting | |
| 2017-05-15 13:29:57,846 INFO Remoting - Remoting started; listening on addresses :[akka.tcp://flink@splat34.sto.midasplayer.com:33785] | |
| 2017-05-15 13:29:57,850 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Actor system started at akka.tcp://flink@splat34.sto.midasplayer.com:33785 | |
| 2017-05-15 13:29:57,850 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Actor system bound to hostname splat34.sto.midasplayer.com. | |
| 2017-05-15 13:29:57,853 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TM:remote keytab path obtained null | |
| 2017-05-15 13:29:57,853 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TM:remote keytab principal obtained null | |
| 2017-05-15 13:29:57,853 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TM:remote yarn conf path obtained null | |
| 2017-05-15 13:29:57,853 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TM:remote krb5 path obtained null | |
| 2017-05-15 13:29:58,321 INFO org.apache.flink.yarn.Utils - Copying from file:/fjord/hadoop/data/nvme/splat/yarn-local-dir/usercache/splat/appcache/application_1494426363399_0021/container_1494426363399_0021_01_000001/b87cbc44-9ced-4a61-a7df-a790571dbf9e-taskmanager-conf.yaml to hdfs://splat34.sto.midasplayer.com:8020/user/splat/.flink/application_1494426363399_0021/b87cbc44-9ced-4a61-a7df-a790571dbf9e-taskmanager-conf.yaml | |
| 2017-05-15 13:29:58,612 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Prepared local resource for modified yaml: resource { scheme: "hdfs" host: "splat34.sto.midasplayer.com" port: 8020 file: "/user/splat/.flink/application_1494426363399_0021/b87cbc44-9ced-4a61-a7df-a790571dbf9e-taskmanager-conf.yaml" } size: 987 timestamp: 1494847798535 type: FILE visibility: APPLICATION | |
| 2017-05-15 13:29:58,616 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Creating container launch context for TaskManagers | |
| 2017-05-15 13:29:58,617 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Starting TaskManagers with command: $JAVA_HOME/bin/java -Xms4400m -Xmx4400m -Dlog.file=<LOG_DIR>/taskmanager.log -Dlogback.configurationFile=file:./logback.xml -Dlog4j.configuration=file:./log4j.properties org.apache.flink.yarn.YarnTaskManager --configDir . 1> <LOG_DIR>/taskmanager.out 2> <LOG_DIR>/taskmanager.err | |
| 2017-05-15 13:29:58,628 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.mode' instead of proper key 'high-availability' | |
| 2017-05-15 13:29:58,631 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.zookeeper.quorum' instead of proper key 'high-availability.zookeeper.quorum' | |
| 2017-05-15 13:29:58,631 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.zookeeper.path.root' instead of proper key 'high-availability.zookeeper.path.root' | |
| 2017-05-15 13:29:58,631 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections | |
| 2017-05-15 13:29:58,632 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink-splat/application_1494426363399_0021' as Zookeeper namespace. | |
| 2017-05-15 13:29:58,676 INFO org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting | |
| 2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT | |
| 2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=splat34.sto.midasplayer.com | |
| 2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_131 | |
| 2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation | |
| 2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre | |
| 2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=lib/flink-connector-kafka-0.8_2.10-1.3-SNAPSHOT.jar:lib/flink-connector-kafka-base_2.10-1.3-SNAPSHOT.jar:lib/flink-dist_2.10-1.3-SNAPSHOT.jar:lib/flink-python_2.10-1.3-SNAPSHOT.jar:lib/flink-shaded-hadoop2-1.3-SNAPSHOT.jar:lib/kafka-clients-0.8.2.2.jar:lib/kafka_2.10-0.8.2.2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:rbea-on-flink-2.0-SNAPSHOT.jar:flink.jar:flink-conf.yaml::/etc/hadoop/conf.cloudera.yarn2:/run/cloudera-scm-agent/process/948-yarn-NODEMANAGER:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-protobuf.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-generator.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-cascading.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-auth.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-auth-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-aws-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-thrift.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-azure-datalake.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-nfs.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-avro.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-annotations-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-azure-datalake-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-test-hadoop2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-encoding.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-nfs-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-jackson.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-pig-bundle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-column.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-format.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-scrooge_2.10.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-format-sources.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-annotations.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-hadoop-bundle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-tools.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-common-2.6.0-cdh5.11.0-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-hadoop.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-pig.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/hadoop-aws.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-scala_2.10.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/parquet-format-javadoc.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-codec-1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jersey-core-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/slf4j-log4j12.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/servlet-api-2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-digester-1.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/curator-client-2.7.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/xz-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jetty-util-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/activation-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/curator-recipes-2.7.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/asm-3.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/stax-api-1.0-2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-dynamodb-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/guava-11.0.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jsp-api-2.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/logredactor-1.0.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/httpcore-4.2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-kms-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/hamcrest-core-1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jets3t-0.9.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-s3-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/avro.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/slf4j-api-1.7.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-el-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/netty-3.10.5.Final.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-math3-3.1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/log4j-1.2.17.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/gson-2.2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/azure-data-lake-store-sdk-2.1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-io-2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-core-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-compress-1.4.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-logging-1.1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/curator-framework-2.7.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jsch-0.1.42.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-beanutils-1.9.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jaxb-api-2.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-collections-3.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/protobuf-java-2.5.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/aws-java-sdk-sts-1.10.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jetty-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/httpclient-4.2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-lang-2.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jersey-server-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-configuration-1.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jsr305-3.0.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jackson-xc-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jettison-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/hue-plugins-3.9.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/zookeeper.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/junit-4.11.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-net-3.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/htrace-core4-4.0.1-incubating.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-cli-1.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/api-util-1.0.0-M20.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/paranamer-2.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/jersey-json-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/xmlenc-0.52.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/mockito-all-1.8.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/commons-httpclient-3.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/hadoop-hdfs-2.6.0-cdh5.11.0-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/asm-3.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/netty-3.10.5.Final.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/htrace-core4-4.0.1-incubating.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-client.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-api-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-registry-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-registry.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-client-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-api.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-common-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-server-tests-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-common-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-common.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.6.0-cdh5.11.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/javax.inject-1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/xz-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jetty-util-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/activation-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/asm-3.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/guava-11.0.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/spark-yarn-shuffle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-core-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jline-2.11.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-io-2.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/guice-3.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-mapper-asl-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/spark-1.6.0-cdh5.11.0-yarn-shuffle.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jetty-6.1.26.cloudera.4.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-xc-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jackson-jaxrs-1.8.8.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jettison-1.1.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/zookeeper.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-yarn/lib/jersey-json-1.9.jar | |
| 2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=:/fjord/hadoop/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib | |
| 2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp | |
| 2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> | |
| 2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux | |
| 2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 | |
| 2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.16.0-4-amd64 | |
| 2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=yarn | |
| 2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/var/lib/hadoop-yarn | |
| 2017-05-15 13:29:58,682 INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/fjord/hadoop/data/nvme/splat/yarn-local-dir/usercache/splat/appcache/application_1494426363399_0021/container_1494426363399_0021_01_000001 | |
| 2017-05-15 13:29:58,683 INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zk04.sto.midasplayer.com:2181,zk05.sto.midasplayer.com:2181,zk06.sto.midasplayer.com:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@781a9412 | |
| 2017-05-15 13:29:58,702 WARN org.apache.zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-5646645451050700926.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. | |
| 2017-05-15 13:29:58,703 INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zk05.sto.midasplayer.com/172.26.82.243:2181 | |
| 2017-05-15 13:29:58,703 ERROR org.apache.flink.shaded.org.apache.curator.ConnectionState - Authentication failed | |
| 2017-05-15 13:29:58,704 INFO org.apache.zookeeper.ClientCnxn - Socket connection established to zk05.sto.midasplayer.com/172.26.82.243:2181, initiating session | |
| 2017-05-15 13:29:58,709 INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zk05.sto.midasplayer.com/172.26.82.243:2181, sessionid = 0x25aad587316ddb2, negotiated timeout = 40000 | |
| 2017-05-15 13:29:58,710 INFO org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED | |
| 2017-05-15 13:29:58,721 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.mode' instead of proper key 'high-availability' | |
| 2017-05-15 13:29:58,721 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.zookeeper.storageDir' instead of proper key 'high-availability.storageDir' | |
| 2017-05-15 13:29:58,754 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at hdfs:///flink/ha/application_1494426363399_0021/blob | |
| 2017-05-15 13:29:58,769 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-50fb85c1-d862-4d21-bfc7-eb1ac920f2f1 | |
| 2017-05-15 13:29:58,771 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:41349 - max concurrent requests: 50 - max backlog: 1000 | |
| 2017-05-15 13:29:58,790 INFO org.apache.flink.runtime.metrics.MetricRegistry - No metrics reporter configured, no metrics will be exposed/reported. | |
| 2017-05-15 13:29:58,801 INFO org.apache.flink.runtime.jobmanager.MemoryArchivist - Started memory archivist akka://flink/user/$a | |
| 2017-05-15 13:29:58,801 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.zookeeper.storageDir' instead of proper key 'high-availability.storageDir' | |
| 2017-05-15 13:29:58,825 WARN org.apache.flink.shaded.org.apache.curator.utils.ZKPaths - The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead. | |
| 2017-05-15 13:29:58,846 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.mode' instead of proper key 'high-availability' | |
| 2017-05-15 13:29:58,847 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Starting JobManager Web Frontend | |
| 2017-05-15 13:29:58,850 INFO org.apache.flink.yarn.YarnJobManager - Starting JobManager at akka.tcp://flink@splat34.sto.midasplayer.com:33785/user/jobmanager. | |
| 2017-05-15 13:29:58,850 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService@5ebb8b4b. | |
| 2017-05-15 13:29:58,854 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of JobManager log file: /fjord/hadoop/data/3/yarn/container-logs/application_1494426363399_0021/container_1494426363399_0021_01_000001/jobmanager.log | |
| 2017-05-15 13:29:58,854 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of JobManager stdout file: /fjord/hadoop/data/3/yarn/container-logs/application_1494426363399_0021/container_1494426363399_0021_01_000001/jobmanager.out | |
| 2017-05-15 13:29:58,854 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-cb0ef96f-0a47-412c-8e82-43b4bdb6dc61 for the web interface files | |
| 2017-05-15 13:29:58,854 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-d05454bc-860d-4b6e-a694-2306a5b962ef for web frontend JAR file uploads | |
| 2017-05-15 13:29:58,956 INFO org.apache.flink.yarn.YarnJobManager - JobManager akka.tcp://flink@splat34.sto.midasplayer.com:33785/user/jobmanager was granted leadership with leader session ID Some(c33ca0bb-8d46-47ab-a450-a39471007f79). | |
| 2017-05-15 13:29:58,965 INFO org.apache.flink.yarn.YarnJobManager - Delaying recovery of all jobs by 120000 milliseconds. | |
| 2017-05-15 13:29:59,110 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web frontend listening at 0.0.0.0:34993 | |
| 2017-05-15 13:29:59,110 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Starting with JobManager akka.tcp://flink@splat34.sto.midasplayer.com:33785/user/jobmanager on port 34993 | |
| 2017-05-15 13:29:59,110 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService. | |
| 2017-05-15 13:29:59,113 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - YARN application tolerates 100 failed TaskManager containers before giving up | |
| 2017-05-15 13:29:59,116 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - YARN Application Master started | |
| 2017-05-15 13:29:59,117 INFO org.apache.flink.runtime.webmonitor.JobManagerRetriever - New leader reachable under akka.tcp://flink@splat34.sto.midasplayer.com:33785/user/jobmanager:c33ca0bb-8d46-47ab-a450-a39471007f79. | |
| 2017-05-15 13:29:59,123 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService. | |
| 2017-05-15 13:29:59,124 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Initializing YARN resource master | |
| 2017-05-15 13:29:59,131 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at splat34.sto.midasplayer.com/172.26.83.103:8030 | |
| 2017-05-15 13:29:59,148 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 2017-05-15 13:29:59,148 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Registering Application Master with tracking url http://splat34.sto.midasplayer.com:34993 | |
| 2017-05-15 13:29:59,245 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Trying to associate with JobManager leader akka.tcp://flink@splat34.sto.midasplayer.com:33785/user/jobmanager | |
| 2017-05-15 13:29:59,252 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Resource Manager associating with leading JobManager Actor[akka://flink/user/jobmanager#-193256653] - leader session c33ca0bb-8d46-47ab-a450-a39471007f79 | |
| 2017-05-15 13:29:59,252 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Requesting new TaskManager container with 5000 megabytes memory. Pending requests: 1 | |
| 2017-05-15 13:30:00,297 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : splat35.sto.midasplayer.com:8041 | |
| 2017-05-15 13:30:00,309 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Received new container: container_1494426363399_0021_01_000002 - Remaining pending container requests: 0 | |
| 2017-05-15 13:30:00,310 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Launching TaskManager in container ContainerInLaunch @ 1494847800310: Container: [ContainerId: container_1494426363399_0021_01_000002, NodeId: splat35.sto.midasplayer.com:8041, NodeHttpAddress: splat35.sto.midasplayer.com:8042, Resource: <memory:5120, vCores:4>, Priority: 0, Token: Token { kind: ContainerToken, service: 172.26.83.104:8041 }, ] on host splat35.sto.midasplayer.com | |
| 2017-05-15 13:30:00,312 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : splat35.sto.midasplayer.com:8041 | |
| 2017-05-15 13:30:01,645 INFO org.apache.flink.yarn.YarnJobManager - ApplicationMaster will shut down session when job 1f387e6136f6faf1824808898750878e has finished. | |
| 2017-05-15 13:30:03,672 INFO org.apache.flink.yarn.YarnFlinkResourceManager - TaskManager container_1494426363399_0021_01_000002 has started. | |
| 2017-05-15 13:30:03,674 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at splat35 (akka.tcp://flink@splat35.sto.midasplayer.com:37378/user/taskmanager) as 983ddd2d6460628883d44134215e7987. Current number of registered hosts is 1. Current number of alive task slots is 1. | |
| 2017-05-15 13:30:03,901 INFO org.apache.flink.yarn.YarnJobManager - Submitting job 1f387e6136f6faf1824808898750878e (event.bifrost.log). | |
| 2017-05-15 13:30:03,915 INFO org.apache.flink.yarn.YarnJobManager - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, delayBetweenRestartAttempts=10000) for 1f387e6136f6faf1824808898750878e. | |
| 2017-05-15 13:30:03,929 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart | |
| 2017-05-15 13:30:03,944 INFO org.apache.flink.yarn.YarnJobManager - Running initialization on master for job event.bifrost.log (1f387e6136f6faf1824808898750878e). | |
| 2017-05-15 13:30:03,944 INFO org.apache.flink.yarn.YarnJobManager - Successfully ran initialization on master in 0 ms. | |
| 2017-05-15 13:30:03,960 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.zookeeper.storageDir' instead of proper key 'high-availability.storageDir' | |
| 2017-05-15 13:30:03,968 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Initialized in '/checkpoints/1f387e6136f6faf1824808898750878e'. | |
| 2017-05-15 13:30:03,977 INFO org.apache.flink.yarn.YarnJobManager - Using application-defined state backend for checkpoint/savepoint metadata: RocksDB State Backend {isInitialized=false, configuredDbBasePaths=null, initializedDbBasePaths=null, checkpointStreamBackend=File State Backend @ hdfs://splat34.sto.midasplayer.com:8020/flink/checkpoints/event.bifrost.log}. | |
| 2017-05-15 13:30:03,979 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Persisting periodic checkpoints externally at hdfs:///flink/external-checkpoints/bifrost. | |
| 2017-05-15 13:30:04,118 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Added SubmittedJobGraph(1f387e6136f6faf1824808898750878e, JobInfo(clients: Set((Actor[akka.tcp://flink@deploy.sto.midasplayer.com:38106/temp/$v],DETACHED)), start: 1494847803889)) to ZooKeeper. | |
| 2017-05-15 13:30:04,119 INFO org.apache.flink.yarn.YarnJobManager - Scheduling job 1f387e6136f6faf1824808898750878e (event.bifrost.log). | |
| 2017-05-15 13:30:04,119 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING. | |
| 2017-05-15 13:30:04,122 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (83626b435e5ba85b71abff97fba20cff) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:04,128 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2a3b9cd8726849c2c652a4e6d8fb59d5) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:04,129 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7b3e5fc47473abba62c4353c10c6a4b3) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:04,129 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bee6085558210aa1501d42432bdf7246) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:04,129 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (2c79d2f52e79f021b284dacf341adc9e) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:04,129 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (59574c2c8369b21866db827d30ea80be) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:04,130 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (65a61cfe9e7b4cbba965b6140bbf4a86) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:04,130 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (b44566133816ba074a4dfdea4b6c326b) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:04,130 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (c4fe2994919c03d68f629c611c70f832) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:04,130 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (1539ced5d6fc47263117e1fa3b68fbea) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:04,130 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (53a9ec178aa90a8e156a800ceab190c0) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:04,130 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (d4f53929056c373d92cd21814812afe1) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:04,130 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (a9b57a05d728fbb5e377f1c2442ac13e) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:04,134 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (83626b435e5ba85b71abff97fba20cff) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:04,135 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #0) to splat35 | |
| 2017-05-15 13:30:04,141 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2a3b9cd8726849c2c652a4e6d8fb59d5) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:04,141 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #0) to splat35 | |
| 2017-05-15 13:30:04,142 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7b3e5fc47473abba62c4353c10c6a4b3) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:04,142 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #0) to splat35 | |
| 2017-05-15 13:30:04,144 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bee6085558210aa1501d42432bdf7246) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:04,144 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #0) to splat35 | |
| 2017-05-15 13:30:04,145 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (2c79d2f52e79f021b284dacf341adc9e) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:04,145 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #0) to splat35 | |
| 2017-05-15 13:30:04,146 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (59574c2c8369b21866db827d30ea80be) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:04,146 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #0) to splat35 | |
| 2017-05-15 13:30:04,151 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (65a61cfe9e7b4cbba965b6140bbf4a86) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:04,151 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #0) to splat35 | |
| 2017-05-15 13:30:04,152 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (b44566133816ba074a4dfdea4b6c326b) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:04,153 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #0) to splat35 | |
| 2017-05-15 13:30:04,156 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (c4fe2994919c03d68f629c611c70f832) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:04,156 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #0) to splat35 | |
| 2017-05-15 13:30:04,157 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (1539ced5d6fc47263117e1fa3b68fbea) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:04,157 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #0) to splat35 | |
| 2017-05-15 13:30:04,158 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (53a9ec178aa90a8e156a800ceab190c0) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:04,158 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #0) to splat35 | |
| 2017-05-15 13:30:04,159 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (d4f53929056c373d92cd21814812afe1) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:04,160 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #0) to splat35 | |
| 2017-05-15 13:30:04,160 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (a9b57a05d728fbb5e377f1c2442ac13e) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:04,160 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #0) to splat35 | |
| 2017-05-15 13:30:04,231 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (65a61cfe9e7b4cbba965b6140bbf4a86) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:04,231 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7b3e5fc47473abba62c4353c10c6a4b3) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:04,232 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (59574c2c8369b21866db827d30ea80be) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:04,233 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (53a9ec178aa90a8e156a800ceab190c0) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:04,234 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (1539ced5d6fc47263117e1fa3b68fbea) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:04,234 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2a3b9cd8726849c2c652a4e6d8fb59d5) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:04,235 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (c4fe2994919c03d68f629c611c70f832) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:04,236 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (83626b435e5ba85b71abff97fba20cff) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:04,236 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (b44566133816ba074a4dfdea4b6c326b) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:04,237 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (d4f53929056c373d92cd21814812afe1) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:04,238 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (2c79d2f52e79f021b284dacf341adc9e) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:04,239 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bee6085558210aa1501d42432bdf7246) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:04,239 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (a9b57a05d728fbb5e377f1c2442ac13e) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:44,550 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 1 @ 1494847844481 | |
| 2017-05-15 13:30:46,364 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7b3e5fc47473abba62c4353c10c6a4b3) switched from RUNNING to FAILED. | |
| AsynchronousException{java.lang.Exception: Could not materialize checkpoint 1 for operator Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1).} | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:966) | |
| at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) | |
| at java.util.concurrent.FutureTask.run(FutureTask.java:266) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.Exception: Could not materialize checkpoint 1 for operator Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1). | |
| ... 6 more | |
| Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Could not open output stream for state backend | |
| at java.util.concurrent.FutureTask.report(FutureTask.java:122) | |
| at java.util.concurrent.FutureTask.get(FutureTask.java:192) | |
| at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:902) | |
| ... 5 more | |
| Suppressed: java.lang.Exception: Could not properly cancel managed operator state future. | |
| at org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:98) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.cleanup(StreamTask.java:1018) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:957) | |
| ... 5 more | |
| Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Could not open output stream for state backend | |
| at java.util.concurrent.FutureTask.report(FutureTask.java:122) | |
| at java.util.concurrent.FutureTask.get(FutureTask.java:192) | |
| at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43) | |
| at org.apache.flink.runtime.state.StateUtil.discardStateFuture(StateUtil.java:85) | |
| at org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:96) | |
| ... 7 more | |
| Caused by: java.io.IOException: Could not open output stream for state backend | |
| at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:371) | |
| at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.flush(FsCheckpointStreamFactory.java:228) | |
| at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.write(FsCheckpointStreamFactory.java:203) | |
| at java.io.DataOutputStream.write(DataOutputStream.java:107) | |
| at org.apache.flink.api.java.typeutils.runtime.DataOutputViewStream.write(DataOutputViewStream.java:41) | |
| at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877) | |
| at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786) | |
| at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1286) | |
| at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231) | |
| at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1427) | |
| at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) | |
| at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1577) | |
| at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:351) | |
| at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:323) | |
| at org.apache.flink.runtime.state.JavaSerializer.serialize(JavaSerializer.java:69) | |
| at org.apache.flink.runtime.state.JavaSerializer.serialize(JavaSerializer.java:33) | |
| at org.apache.flink.runtime.state.DefaultOperatorStateBackend$PartitionableListState.write(DefaultOperatorStateBackend.java:432) | |
| at org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:233) | |
| at org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:203) | |
| at org.apache.flink.runtime.io.async.AbstractAsyncIOCallable.call(AbstractAsyncIOCallable.java:72) | |
| at java.util.concurrent.FutureTask.run(FutureTask.java:266) | |
| at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:40) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:902) | |
| ... 5 more | |
| Caused by: java.io.IOException: Cannot register Closeable, registry is already closed. Closing argument. | |
| at org.apache.flink.util.AbstractCloseableRegistry.registerClosable(AbstractCloseableRegistry.java:66) | |
| at org.apache.flink.core.fs.ClosingFSDataOutputStream.wrapSafe(ClosingFSDataOutputStream.java:101) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:125) | |
| at org.apache.flink.core.fs.FileSystem.create(FileSystem.java:621) | |
| at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:362) | |
| ... 27 more | |
| [CIRCULAR REFERENCE:java.io.IOException: Could not open output stream for state backend] | |
| 2017-05-15 13:30:46,366 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING. | |
| AsynchronousException{java.lang.Exception: Could not materialize checkpoint 1 for operator Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1).} | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:966) | |
| at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) | |
| at java.util.concurrent.FutureTask.run(FutureTask.java:266) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.Exception: Could not materialize checkpoint 1 for operator Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1). | |
| ... 6 more | |
| Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Could not open output stream for state backend | |
| at java.util.concurrent.FutureTask.report(FutureTask.java:122) | |
| at java.util.concurrent.FutureTask.get(FutureTask.java:192) | |
| at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:902) | |
| ... 5 more | |
| Suppressed: java.lang.Exception: Could not properly cancel managed operator state future. | |
| at org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:98) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.cleanup(StreamTask.java:1018) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:957) | |
| ... 5 more | |
| Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Could not open output stream for state backend | |
| at java.util.concurrent.FutureTask.report(FutureTask.java:122) | |
| at java.util.concurrent.FutureTask.get(FutureTask.java:192) | |
| at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43) | |
| at org.apache.flink.runtime.state.StateUtil.discardStateFuture(StateUtil.java:85) | |
| at org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:96) | |
| ... 7 more | |
| Caused by: java.io.IOException: Could not open output stream for state backend | |
| at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:371) | |
| at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.flush(FsCheckpointStreamFactory.java:228) | |
| at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.write(FsCheckpointStreamFactory.java:203) | |
| at java.io.DataOutputStream.write(DataOutputStream.java:107) | |
| at org.apache.flink.api.java.typeutils.runtime.DataOutputViewStream.write(DataOutputViewStream.java:41) | |
| at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877) | |
| at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786) | |
| at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1286) | |
| at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231) | |
| at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1427) | |
| at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) | |
| at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1577) | |
| at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:351) | |
| at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:323) | |
| at org.apache.flink.runtime.state.JavaSerializer.serialize(JavaSerializer.java:69) | |
| at org.apache.flink.runtime.state.JavaSerializer.serialize(JavaSerializer.java:33) | |
| at org.apache.flink.runtime.state.DefaultOperatorStateBackend$PartitionableListState.write(DefaultOperatorStateBackend.java:432) | |
| at org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:233) | |
| at org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:203) | |
| at org.apache.flink.runtime.io.async.AbstractAsyncIOCallable.call(AbstractAsyncIOCallable.java:72) | |
| at java.util.concurrent.FutureTask.run(FutureTask.java:266) | |
| at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:40) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:902) | |
| ... 5 more | |
| Caused by: java.io.IOException: Cannot register Closeable, registry is already closed. Closing argument. | |
| at org.apache.flink.util.AbstractCloseableRegistry.registerClosable(AbstractCloseableRegistry.java:66) | |
| at org.apache.flink.core.fs.ClosingFSDataOutputStream.wrapSafe(ClosingFSDataOutputStream.java:101) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:125) | |
| at org.apache.flink.core.fs.FileSystem.create(FileSystem.java:621) | |
| at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:362) | |
| ... 27 more | |
| [CIRCULAR REFERENCE:java.io.IOException: Could not open output stream for state backend] | |
| 2017-05-15 13:30:46,373 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (83626b435e5ba85b71abff97fba20cff) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:30:46,376 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2a3b9cd8726849c2c652a4e6d8fb59d5) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:30:46,377 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bee6085558210aa1501d42432bdf7246) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:30:46,377 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (2c79d2f52e79f021b284dacf341adc9e) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:30:46,377 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (59574c2c8369b21866db827d30ea80be) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:30:46,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (65a61cfe9e7b4cbba965b6140bbf4a86) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:30:46,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (b44566133816ba074a4dfdea4b6c326b) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:30:46,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (c4fe2994919c03d68f629c611c70f832) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:30:46,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (1539ced5d6fc47263117e1fa3b68fbea) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:30:46,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (53a9ec178aa90a8e156a800ceab190c0) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:30:46,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (d4f53929056c373d92cd21814812afe1) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:30:46,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (a9b57a05d728fbb5e377f1c2442ac13e) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:30:46,396 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (83626b435e5ba85b71abff97fba20cff) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:30:46,406 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bee6085558210aa1501d42432bdf7246) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:30:46,407 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (59574c2c8369b21866db827d30ea80be) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:30:46,408 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (1539ced5d6fc47263117e1fa3b68fbea) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:30:46,409 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (65a61cfe9e7b4cbba965b6140bbf4a86) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:30:46,410 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (53a9ec178aa90a8e156a800ceab190c0) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:30:46,410 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (b44566133816ba074a4dfdea4b6c326b) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:30:46,411 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (c4fe2994919c03d68f629c611c70f832) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:30:46,412 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (d4f53929056c373d92cd21814812afe1) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:30:46,412 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (a9b57a05d728fbb5e377f1c2442ac13e) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:30:46,470 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2a3b9cd8726849c2c652a4e6d8fb59d5) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:30:46,614 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (2c79d2f52e79f021b284dacf341adc9e) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:30:46,614 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible. | |
| 2017-05-15 13:30:46,614 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING. | |
| 2017-05-15 13:30:46,614 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e). | |
| 2017-05-15 13:30:46,615 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ... | |
| 2017-05-15 13:30:56,616 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED. | |
| 2017-05-15 13:30:56,616 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper. | |
| 2017-05-15 13:30:56,621 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 0 checkpoints in ZooKeeper. | |
| 2017-05-15 13:30:56,621 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING. | |
| 2017-05-15 13:30:56,621 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (a805efa3b73f0379008da6d15fafc4d7) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2856c0d36955d3e90435ce1b1a2a9bb0) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d876b4b2d0f567a09562ce0e8720abd6) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (5359a06611829c1d9bfdecf0b66540a4) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (5eeb7c066ee255a446cc70edfbea2b42) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (61cb52d8e9d07d6645fe2d6f12db676e) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (08c882c354263ddfeb8711b12249c87e) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (92affefa971d217fda6a8c94fb64f2a5) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:56,622 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (279e8a00c652f78c5938d9b432852475) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:56,623 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (8ae81acf282ca346d76b8bb6694994a4) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:56,623 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (5af3e14dc574394bf24fe7e3b9186812) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:56,623 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (a46f9e0b28f10f3d4a02e6fa6699840a) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:56,623 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7534412a727ea1a1daebef412095c0c7) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:30:56,623 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (a805efa3b73f0379008da6d15fafc4d7) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:56,624 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #1) to splat35 | |
| 2017-05-15 13:30:56,624 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2856c0d36955d3e90435ce1b1a2a9bb0) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:56,624 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #1) to splat35 | |
| 2017-05-15 13:30:56,625 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d876b4b2d0f567a09562ce0e8720abd6) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:56,625 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #1) to splat35 | |
| 2017-05-15 13:30:56,625 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (5359a06611829c1d9bfdecf0b66540a4) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:56,625 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #1) to splat35 | |
| 2017-05-15 13:30:56,625 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (5eeb7c066ee255a446cc70edfbea2b42) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:56,625 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #1) to splat35 | |
| 2017-05-15 13:30:56,626 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (61cb52d8e9d07d6645fe2d6f12db676e) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:56,626 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #1) to splat35 | |
| 2017-05-15 13:30:56,626 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (08c882c354263ddfeb8711b12249c87e) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:56,626 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #1) to splat35 | |
| 2017-05-15 13:30:56,626 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (92affefa971d217fda6a8c94fb64f2a5) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:56,626 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #1) to splat35 | |
| 2017-05-15 13:30:56,626 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (279e8a00c652f78c5938d9b432852475) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:56,627 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #1) to splat35 | |
| 2017-05-15 13:30:56,627 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (8ae81acf282ca346d76b8bb6694994a4) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:56,627 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #1) to splat35 | |
| 2017-05-15 13:30:56,627 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (5af3e14dc574394bf24fe7e3b9186812) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:56,627 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #1) to splat35 | |
| 2017-05-15 13:30:56,627 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (a46f9e0b28f10f3d4a02e6fa6699840a) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:56,627 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #1) to splat35 | |
| 2017-05-15 13:30:56,628 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7534412a727ea1a1daebef412095c0c7) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:30:56,628 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #1) to splat35 | |
| 2017-05-15 13:30:56,640 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (a805efa3b73f0379008da6d15fafc4d7) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:56,642 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2856c0d36955d3e90435ce1b1a2a9bb0) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:56,643 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d876b4b2d0f567a09562ce0e8720abd6) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:56,644 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (5359a06611829c1d9bfdecf0b66540a4) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:56,645 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (5eeb7c066ee255a446cc70edfbea2b42) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:56,649 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (61cb52d8e9d07d6645fe2d6f12db676e) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:56,649 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (08c882c354263ddfeb8711b12249c87e) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:56,668 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (92affefa971d217fda6a8c94fb64f2a5) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:56,670 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (279e8a00c652f78c5938d9b432852475) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:56,671 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (8ae81acf282ca346d76b8bb6694994a4) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:56,671 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (5af3e14dc574394bf24fe7e3b9186812) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:56,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (a46f9e0b28f10f3d4a02e6fa6699840a) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:30:56,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7534412a727ea1a1daebef412095c0c7) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:31:56,624 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 2 @ 1494847916621 | |
| 2017-05-15 13:31:57,135 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 2 (1108645 bytes in 462 ms). | |
| 2017-05-15 13:31:58,980 INFO org.apache.flink.yarn.YarnJobManager - Attempting to recover all jobs. | |
| 2017-05-15 13:31:58,984 INFO org.apache.flink.yarn.YarnJobManager - There are 1 jobs to recover. Starting the job recovery. | |
| 2017-05-15 13:31:58,987 INFO org.apache.flink.yarn.YarnJobManager - Attempting to recover job 1f387e6136f6faf1824808898750878e. | |
| 2017-05-15 13:31:59,042 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Recovered SubmittedJobGraph(1f387e6136f6faf1824808898750878e, JobInfo(clients: Set((Actor[akka.tcp://flink@deploy.sto.midasplayer.com:38106/temp/$v],DETACHED)), start: 1494847803889)). | |
| 2017-05-15 13:31:59,043 INFO org.apache.flink.yarn.YarnJobManager - Ignoring job recovery for 1f387e6136f6faf1824808898750878e, because it is already submitted. | |
| 2017-05-15 13:32:57,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 3 @ 1494847977134 | |
| 2017-05-15 13:32:57,816 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 3 (1120696 bytes in 624 ms). | |
| 2017-05-15 13:32:57,819 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 2. | |
| 2017-05-15 13:32:57,878 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 2. | |
| java.lang.Exception: Could not execute callable action for checkpoint 2. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:33:57,818 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 4 @ 1494848037815 | |
| 2017-05-15 13:33:58,319 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 4 (1132754 bytes in 460 ms). | |
| 2017-05-15 13:33:58,321 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 3. | |
| 2017-05-15 13:33:58,351 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 3. | |
| java.lang.Exception: Could not execute callable action for checkpoint 3. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:34:58,321 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 5 @ 1494848098319 | |
| 2017-05-15 13:34:58,894 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 5 (1145382 bytes in 530 ms). | |
| 2017-05-15 13:34:58,896 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 4. | |
| 2017-05-15 13:34:58,927 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 4. | |
| java.lang.Exception: Could not execute callable action for checkpoint 4. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:35:58,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 6 @ 1494848158893 | |
| 2017-05-15 13:35:59,401 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 6 (1124229 bytes in 464 ms). | |
| 2017-05-15 13:35:59,403 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 5. | |
| 2017-05-15 13:35:59,426 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 5. | |
| java.lang.Exception: Could not execute callable action for checkpoint 5. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:36:59,403 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 7 @ 1494848219401 | |
| 2017-05-15 13:36:59,893 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 7 (1139094 bytes in 448 ms). | |
| 2017-05-15 13:36:59,895 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 6. | |
| 2017-05-15 13:36:59,926 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 6. | |
| java.lang.Exception: Could not execute callable action for checkpoint 6. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:37:59,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 8 @ 1494848279893 | |
| 2017-05-15 13:38:00,779 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 8 (1154702 bytes in 733 ms). | |
| 2017-05-15 13:38:00,780 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 7. | |
| 2017-05-15 13:38:00,878 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 7. | |
| java.lang.Exception: Could not execute callable action for checkpoint 7. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:39:00,780 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 9 @ 1494848340778 | |
| 2017-05-15 13:39:01,353 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 9 (1170306 bytes in 531 ms). | |
| 2017-05-15 13:39:01,355 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 8. | |
| 2017-05-15 13:39:01,385 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 8. | |
| java.lang.Exception: Could not execute callable action for checkpoint 8. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:40:01,356 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 10 @ 1494848401352 | |
| 2017-05-15 13:40:01,861 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 10 (1131407 bytes in 465 ms). | |
| 2017-05-15 13:40:01,862 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 9. | |
| 2017-05-15 13:40:01,885 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 9. | |
| java.lang.Exception: Could not execute callable action for checkpoint 9. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:41:01,863 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 11 @ 1494848461860 | |
| 2017-05-15 13:41:02,411 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 11 (1149375 bytes in 507 ms). | |
| 2017-05-15 13:41:02,413 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 10. | |
| 2017-05-15 13:41:02,443 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 10. | |
| java.lang.Exception: Could not execute callable action for checkpoint 10. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:42:02,412 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 12 @ 1494848522410 | |
| 2017-05-15 13:42:02,986 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 12 (1167347 bytes in 532 ms). | |
| 2017-05-15 13:42:02,988 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 11. | |
| 2017-05-15 13:42:03,017 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 11. | |
| java.lang.Exception: Could not execute callable action for checkpoint 11. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:43:02,988 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 13 @ 1494848582986 | |
| 2017-05-15 13:43:03,442 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 13 (1185638 bytes in 411 ms). | |
| 2017-05-15 13:43:03,444 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 12. | |
| 2017-05-15 13:43:03,473 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 12. | |
| java.lang.Exception: Could not execute callable action for checkpoint 12. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:44:03,444 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 14 @ 1494848643442 | |
| 2017-05-15 13:44:03,824 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 14 (1135623 bytes in 355 ms). | |
| 2017-05-15 13:44:03,826 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 13. | |
| 2017-05-15 13:44:03,847 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 13. | |
| java.lang.Exception: Could not execute callable action for checkpoint 13. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:45:03,826 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 15 @ 1494848703824 | |
| 2017-05-15 13:45:04,266 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 15 (1154541 bytes in 414 ms). | |
| 2017-05-15 13:45:04,268 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 14. | |
| 2017-05-15 13:45:04,298 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 14. | |
| java.lang.Exception: Could not execute callable action for checkpoint 14. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:46:04,268 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 16 @ 1494848764266 | |
| 2017-05-15 13:46:04,741 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 16 (1174659 bytes in 447 ms). | |
| 2017-05-15 13:46:04,743 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 15. | |
| 2017-05-15 13:46:04,772 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 15. | |
| java.lang.Exception: Could not execute callable action for checkpoint 15. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:47:04,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 17 @ 1494848824740 | |
| 2017-05-15 13:47:05,157 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 17 (1194777 bytes in 389 ms). | |
| 2017-05-15 13:47:05,158 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 16. | |
| 2017-05-15 13:47:05,189 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 16. | |
| java.lang.Exception: Could not execute callable action for checkpoint 16. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:48:05,158 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 18 @ 1494848885156 | |
| 2017-05-15 13:48:05,600 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 18 (1140489 bytes in 416 ms). | |
| 2017-05-15 13:48:05,601 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 17. | |
| 2017-05-15 13:48:05,624 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 17. | |
| java.lang.Exception: Could not execute callable action for checkpoint 17. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:49:05,601 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 19 @ 1494848945599 | |
| 2017-05-15 13:49:05,999 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 19 (1161882 bytes in 373 ms). | |
| 2017-05-15 13:49:06,001 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 18. | |
| 2017-05-15 13:49:06,023 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 18. | |
| java.lang.Exception: Could not execute callable action for checkpoint 18. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:50:06,001 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 20 @ 1494849005998 | |
| 2017-05-15 13:50:06,417 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 20 (1183918 bytes in 391 ms). | |
| 2017-05-15 13:50:06,419 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 19. | |
| 2017-05-15 13:50:06,448 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 19. | |
| java.lang.Exception: Could not execute callable action for checkpoint 19. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:51:06,419 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 21 @ 1494849066417 | |
| 2017-05-15 13:51:06,858 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 21 (1206982 bytes in 405 ms). | |
| 2017-05-15 13:51:06,859 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 20. | |
| 2017-05-15 13:51:06,889 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 20. | |
| java.lang.Exception: Could not execute callable action for checkpoint 20. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:52:06,859 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 22 @ 1494849126857 | |
| 2017-05-15 13:52:07,274 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 22 (1146260 bytes in 390 ms). | |
| 2017-05-15 13:52:07,275 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 21. | |
| 2017-05-15 13:52:07,298 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 21. | |
| java.lang.Exception: Could not execute callable action for checkpoint 21. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:53:07,276 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 23 @ 1494849187274 | |
| 2017-05-15 13:53:07,757 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 23 (1170386 bytes in 447 ms). | |
| 2017-05-15 13:53:07,759 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 22. | |
| 2017-05-15 13:53:07,780 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 22. | |
| java.lang.Exception: Could not execute callable action for checkpoint 22. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:54:07,758 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 24 @ 1494849247756 | |
| 2017-05-15 13:54:08,290 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 24 (1194506 bytes in 498 ms). | |
| 2017-05-15 13:54:08,291 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 23. | |
| 2017-05-15 13:54:08,313 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 23. | |
| java.lang.Exception: Could not execute callable action for checkpoint 23. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:55:08,291 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 25 @ 1494849308289 | |
| 2017-05-15 13:55:08,864 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 25 (1218646 bytes in 540 ms). | |
| 2017-05-15 13:55:08,866 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 24. | |
| 2017-05-15 13:55:08,896 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 24. | |
| java.lang.Exception: Could not execute callable action for checkpoint 24. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:56:08,866 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 26 @ 1494849368864 | |
| 2017-05-15 13:56:09,422 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 26 (1148346 bytes in 523 ms). | |
| 2017-05-15 13:56:09,424 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 25. | |
| 2017-05-15 13:56:09,446 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 25. | |
| java.lang.Exception: Could not execute callable action for checkpoint 25. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:57:09,425 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 27 @ 1494849429422 | |
| 2017-05-15 13:57:09,881 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 27 (1173336 bytes in 423 ms). | |
| 2017-05-15 13:57:09,883 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 26. | |
| 2017-05-15 13:57:09,904 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 26. | |
| java.lang.Exception: Could not execute callable action for checkpoint 26. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:58:09,883 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 28 @ 1494849489880 | |
| 2017-05-15 13:58:10,374 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 28 (1198326 bytes in 459 ms). | |
| 2017-05-15 13:58:10,376 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 27. | |
| 2017-05-15 13:58:10,406 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 27. | |
| java.lang.Exception: Could not execute callable action for checkpoint 27. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:59:10,376 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 29 @ 1494849550373 | |
| 2017-05-15 13:59:10,982 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 29 (1223316 bytes in 574 ms). | |
| 2017-05-15 13:59:10,984 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 28. | |
| 2017-05-15 13:59:11,014 WARN org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Failed to discard checkpoint 28. | |
| java.lang.Exception: Could not execute callable action for checkpoint 28. | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:380) | |
| at org.apache.flink.shaded.org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.lang.IllegalStateException: The state handle has not registered its shared states yet. | |
| at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) | |
| at org.apache.flink.runtime.state.IncrementalKeyedStateHandle.unregisterSharedStates(IncrementalKeyedStateHandle.java:251) | |
| at org.apache.flink.runtime.checkpoint.OperatorSubtaskState.unregisterSharedStates(OperatorSubtaskState.java:163) | |
| at org.apache.flink.runtime.checkpoint.OperatorState.unregisterSharedStates(OperatorState.java:131) | |
| at org.apache.flink.runtime.state.SharedStateRegistry.unregisterAll(SharedStateRegistry.java:187) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$StoredDiscardStrategy.doDiscardSharedState(CompletedCheckpoint.java:364) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint$DiscardStrategy.discard(CompletedCheckpoint.java:288) | |
| at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:186) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:324) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$1.call(ZooKeeperCompletedCheckpointStore.java:318) | |
| at org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$3.processResult(ZooKeeperCompletedCheckpointStore.java:378) | |
| ... 4 more | |
| 2017-05-15 13:59:20,335 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Disassociated] | |
| 2017-05-15 13:59:21,286 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Container container_1494426363399_0021_01_000002 failed. Exit status: 143 | |
| 2017-05-15 13:59:21,287 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Diagnostics for container container_1494426363399_0021_01_000002 in state COMPLETE : exitStatus=143 diagnostics=Container killed on request. Exit code is 143 | |
| Container exited with a non-zero exit code 143 | |
| Killed by external signal | |
| 2017-05-15 13:59:21,287 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Total number of failed containers so far: 1 | |
| 2017-05-15 13:59:21,287 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Requesting new TaskManager container with 5000 megabytes memory. Pending requests: 1 | |
| 2017-05-15 13:59:21,287 INFO org.apache.flink.yarn.YarnJobManager - Task manager akka.tcp://flink@splat35.sto.midasplayer.com:37378/user/taskmanager terminated. | |
| 2017-05-15 13:59:21,288 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (5359a06611829c1d9bfdecf0b66540a4) switched from RUNNING to FAILED. | |
| java.lang.Exception: TaskManager was lost/killed: container_1494426363399_0021_01_000002 @ splat35.sto.midasplayer.com (dataPort=41610) | |
| at org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:217) | |
| at org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:533) | |
| at org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:192) | |
| at org.apache.flink.runtime.instance.Instance.markDead(Instance.java:167) | |
| at org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:212) | |
| at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$handleTaskManagerTerminated(JobManager.scala:1228) | |
| at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:474) | |
| at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) | |
| at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) | |
| at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) | |
| at org.apache.flink.runtime.clusterframework.ContaineredJobManager$$anonfun$handleContainerMessage$1.applyOrElse(ContaineredJobManager.scala:100) | |
| at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162) | |
| at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:38) | |
| at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) | |
| at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) | |
| at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) | |
| at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33) | |
| at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28) | |
| at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118) | |
| at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28) | |
| at akka.actor.Actor$class.aroundReceive(Actor.scala:467) | |
| at org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:125) | |
| at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) | |
| at akka.actor.ActorCell.invoke(ActorCell.scala:487) | |
| at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) | |
| at akka.dispatch.Mailbox.run(Mailbox.scala:220) | |
| at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) | |
| at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) | |
| at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) | |
| at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) | |
| at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) | |
| 2017-05-15 13:59:21,288 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING. | |
| java.lang.Exception: TaskManager was lost/killed: container_1494426363399_0021_01_000002 @ splat35.sto.midasplayer.com (dataPort=41610) | |
| at org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:217) | |
| at org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:533) | |
| at org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:192) | |
| at org.apache.flink.runtime.instance.Instance.markDead(Instance.java:167) | |
| at org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:212) | |
| at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$handleTaskManagerTerminated(JobManager.scala:1228) | |
| at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:474) | |
| at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) | |
| at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) | |
| at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) | |
| at org.apache.flink.runtime.clusterframework.ContaineredJobManager$$anonfun$handleContainerMessage$1.applyOrElse(ContaineredJobManager.scala:100) | |
| at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162) | |
| at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:38) | |
| at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) | |
| at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) | |
| at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) | |
| at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33) | |
| at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28) | |
| at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118) | |
| at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28) | |
| at akka.actor.Actor$class.aroundReceive(Actor.scala:467) | |
| at org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:125) | |
| at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) | |
| at akka.actor.ActorCell.invoke(ActorCell.scala:487) | |
| at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) | |
| at akka.dispatch.Mailbox.run(Mailbox.scala:220) | |
| at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) | |
| at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) | |
| at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) | |
| at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) | |
| at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) | |
| 2017-05-15 13:59:21,292 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (a805efa3b73f0379008da6d15fafc4d7) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2856c0d36955d3e90435ce1b1a2a9bb0) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d876b4b2d0f567a09562ce0e8720abd6) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (5eeb7c066ee255a446cc70edfbea2b42) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (61cb52d8e9d07d6645fe2d6f12db676e) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (08c882c354263ddfeb8711b12249c87e) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (92affefa971d217fda6a8c94fb64f2a5) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (279e8a00c652f78c5938d9b432852475) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (8ae81acf282ca346d76b8bb6694994a4) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (5af3e14dc574394bf24fe7e3b9186812) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:21,293 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (a46f9e0b28f10f3d4a02e6fa6699840a) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7534412a727ea1a1daebef412095c0c7) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (61cb52d8e9d07d6645fe2d6f12db676e) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d876b4b2d0f567a09562ce0e8720abd6) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (5af3e14dc574394bf24fe7e3b9186812) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (08c882c354263ddfeb8711b12249c87e) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (2856c0d36955d3e90435ce1b1a2a9bb0) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (279e8a00c652f78c5938d9b432852475) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (a805efa3b73f0379008da6d15fafc4d7) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:21,294 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (a46f9e0b28f10f3d4a02e6fa6699840a) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (5eeb7c066ee255a446cc70edfbea2b42) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (92affefa971d217fda6a8c94fb64f2a5) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (8ae81acf282ca346d76b8bb6694994a4) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7534412a727ea1a1daebef412095c0c7) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible. | |
| 2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING. | |
| 2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e). | |
| 2017-05-15 13:59:21,295 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ... | |
| 2017-05-15 13:59:21,296 INFO org.apache.flink.runtime.instance.InstanceManager - Unregistered task manager splat35.sto.midasplayer.com/172.26.83.104. Number of registered task managers 0. Number of available slots 0. | |
| 2017-05-15 13:59:25,371 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 13:59:27,291 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : splat34.sto.midasplayer.com:8041 | |
| 2017-05-15 13:59:27,293 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Received new container: container_1494426363399_0021_01_000003 - Remaining pending container requests: 0 | |
| 2017-05-15 13:59:27,293 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Launching TaskManager in container ContainerInLaunch @ 1494849567293: Container: [ContainerId: container_1494426363399_0021_01_000003, NodeId: splat34.sto.midasplayer.com:8041, NodeHttpAddress: splat34.sto.midasplayer.com:8042, Resource: <memory:5120, vCores:4>, Priority: 0, Token: Token { kind: ContainerToken, service: 172.26.83.103:8041 }, ] on host splat34.sto.midasplayer.com | |
| 2017-05-15 13:59:27,293 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : splat34.sto.midasplayer.com:8041 | |
| 2017-05-15 13:59:27,305 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Received new container: container_1494426363399_0021_01_000004 - Remaining pending container requests: 0 | |
| 2017-05-15 13:59:27,305 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Returning excess container container_1494426363399_0021_01_000004 | |
| 2017-05-15 13:59:27,799 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Container container_1494426363399_0021_01_000004 completed successfully with diagnostics: Container released by application | |
| 2017-05-15 13:59:29,962 INFO org.apache.flink.yarn.YarnFlinkResourceManager - TaskManager container_1494426363399_0021_01_000003 has started. | |
| 2017-05-15 13:59:29,962 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at splat34 (akka.tcp://flink@splat34.sto.midasplayer.com:53480/user/taskmanager) as 326104999a117409c1609bdd0b38403a. Current number of registered hosts is 1. Current number of alive task slots is 1. | |
| 2017-05-15 13:59:30,394 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 13:59:31,300 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED. | |
| 2017-05-15 13:59:31,300 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper. | |
| 2017-05-15 13:59:31,303 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 1 checkpoints in ZooKeeper. | |
| 2017-05-15 13:59:31,304 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29. | |
| 2017-05-15 13:59:31,309 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29. | |
| 2017-05-15 13:59:31,311 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 29 @ 1494849550373 for 1f387e6136f6faf1824808898750878e. | |
| 2017-05-15 13:59:31,314 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore | |
| 2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING. | |
| 2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (0bd17588f4674f754ced2562c447ed0a) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (006621199f1282f7ec623fadb1cb65dc) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (79ded85532660a95ac8280f21ae9d8d4) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bf80bf5edfc2c072fe36a61156ff935a) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (fbb19bea4823fc2f0fd1482d61054a20) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a55516edcbcc448edc5fd77e31d55d64) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (114fe598aed8f758e58fef63caca9d72) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3e600d83298e0b81658d57bef7e8536c) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (84b8b824b499518947ae477dad73924d) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (56c3498a9b9ed624d608bc75ece2142b) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (484eae4801564b1c5064fc74408327bc) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (e89114ab129b9a766bae9394d08b721a) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:31,315 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (e0ef852e975307d3794663d38f7488ca) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:31,316 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (0bd17588f4674f754ced2562c447ed0a) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:31,316 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #2) to splat34 | |
| 2017-05-15 13:59:31,316 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (006621199f1282f7ec623fadb1cb65dc) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:31,316 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #2) to splat34 | |
| 2017-05-15 13:59:31,316 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (79ded85532660a95ac8280f21ae9d8d4) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:31,316 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #2) to splat34 | |
| 2017-05-15 13:59:31,317 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bf80bf5edfc2c072fe36a61156ff935a) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:31,317 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #2) to splat34 | |
| 2017-05-15 13:59:31,317 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (fbb19bea4823fc2f0fd1482d61054a20) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:31,317 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #2) to splat34 | |
| 2017-05-15 13:59:31,317 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a55516edcbcc448edc5fd77e31d55d64) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:31,317 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #2) to splat34 | |
| 2017-05-15 13:59:31,317 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (114fe598aed8f758e58fef63caca9d72) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:31,318 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #2) to splat34 | |
| 2017-05-15 13:59:31,318 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3e600d83298e0b81658d57bef7e8536c) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:31,318 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #2) to splat34 | |
| 2017-05-15 13:59:31,318 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (84b8b824b499518947ae477dad73924d) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:31,318 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #2) to splat34 | |
| 2017-05-15 13:59:31,318 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (56c3498a9b9ed624d608bc75ece2142b) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:31,318 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #2) to splat34 | |
| 2017-05-15 13:59:31,319 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (484eae4801564b1c5064fc74408327bc) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:31,319 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #2) to splat34 | |
| 2017-05-15 13:59:31,319 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (e89114ab129b9a766bae9394d08b721a) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:31,319 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #2) to splat34 | |
| 2017-05-15 13:59:31,319 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (e0ef852e975307d3794663d38f7488ca) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:31,319 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #2) to splat34 | |
| 2017-05-15 13:59:31,399 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (114fe598aed8f758e58fef63caca9d72) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:31,399 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (79ded85532660a95ac8280f21ae9d8d4) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:31,400 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (fbb19bea4823fc2f0fd1482d61054a20) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:31,401 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (0bd17588f4674f754ced2562c447ed0a) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:31,401 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (006621199f1282f7ec623fadb1cb65dc) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:31,402 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bf80bf5edfc2c072fe36a61156ff935a) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:31,403 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a55516edcbcc448edc5fd77e31d55d64) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:31,404 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3e600d83298e0b81658d57bef7e8536c) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:31,405 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (84b8b824b499518947ae477dad73924d) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:31,406 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (56c3498a9b9ed624d608bc75ece2142b) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:31,407 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (484eae4801564b1c5064fc74408327bc) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:31,407 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (e89114ab129b9a766bae9394d08b721a) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:31,408 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (e0ef852e975307d3794663d38f7488ca) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:31,887 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (114fe598aed8f758e58fef63caca9d72) switched from RUNNING to FAILED. | |
| java.lang.IllegalStateException: Could not initialize keyed state backend. | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251) | |
| at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) | |
| at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) | |
| at java.lang.reflect.Constructor.newInstance(Constructor.java:423) | |
| at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) | |
| at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200) | |
| at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271) | |
| at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238) | |
| at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231) | |
| at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85) | |
| at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311) | |
| ... 6 more | |
| Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1468) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1399) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) | |
| at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) | |
| at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220) | |
| ... 28 more | |
| 2017-05-15 13:59:31,888 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING. | |
| java.lang.IllegalStateException: Could not initialize keyed state backend. | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251) | |
| at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) | |
| at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) | |
| at java.lang.reflect.Constructor.newInstance(Constructor.java:423) | |
| at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) | |
| at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200) | |
| at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271) | |
| at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238) | |
| at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231) | |
| at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85) | |
| at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311) | |
| ... 6 more | |
| Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1468) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1399) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) | |
| at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) | |
| at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220) | |
| ... 28 more | |
| 2017-05-15 13:59:31,890 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (0bd17588f4674f754ced2562c447ed0a) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:31,890 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (006621199f1282f7ec623fadb1cb65dc) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:31,890 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (79ded85532660a95ac8280f21ae9d8d4) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:31,890 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bf80bf5edfc2c072fe36a61156ff935a) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (fbb19bea4823fc2f0fd1482d61054a20) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a55516edcbcc448edc5fd77e31d55d64) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3e600d83298e0b81658d57bef7e8536c) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (84b8b824b499518947ae477dad73924d) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (56c3498a9b9ed624d608bc75ece2142b) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (484eae4801564b1c5064fc74408327bc) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (e89114ab129b9a766bae9394d08b721a) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:31,891 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (e0ef852e975307d3794663d38f7488ca) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:31,932 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (0bd17588f4674f754ced2562c447ed0a) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:31,933 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (bf80bf5edfc2c072fe36a61156ff935a) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:31,933 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a55516edcbcc448edc5fd77e31d55d64) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:31,934 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3e600d83298e0b81658d57bef7e8536c) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:31,934 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (84b8b824b499518947ae477dad73924d) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:31,935 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (56c3498a9b9ed624d608bc75ece2142b) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:31,935 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (484eae4801564b1c5064fc74408327bc) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:31,935 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (e89114ab129b9a766bae9394d08b721a) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:31,935 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (e0ef852e975307d3794663d38f7488ca) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:34,078 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (006621199f1282f7ec623fadb1cb65dc) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:34,290 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (79ded85532660a95ac8280f21ae9d8d4) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:35,199 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (fbb19bea4823fc2f0fd1482d61054a20) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:35,199 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible. | |
| 2017-05-15 13:59:35,199 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING. | |
| 2017-05-15 13:59:35,199 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e). | |
| 2017-05-15 13:59:35,199 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ... | |
| 2017-05-15 13:59:35,414 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 13:59:40,434 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 13:59:45,200 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED. | |
| 2017-05-15 13:59:45,200 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper. | |
| 2017-05-15 13:59:45,204 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 1 checkpoints in ZooKeeper. | |
| 2017-05-15 13:59:45,204 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29. | |
| 2017-05-15 13:59:45,211 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29. | |
| 2017-05-15 13:59:45,214 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 29 @ 1494849550373 for 1f387e6136f6faf1824808898750878e. | |
| 2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore | |
| 2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING. | |
| 2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (31c20e3773238419f0ad99e2fb2da7fa) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (368276f9f3101752a52a9b3ef4ab090a) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (6f3eadbb122384d3c186417e7c359d4d) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (7507653dacef8da0f710f6eafeb5310f) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (3da695760bd3750f38048853db4b3ada) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (1f70d3824513abaa0150c020f134daf5) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (4bbfb4d9e406c459222e1ea8304ecf2d) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:45,215 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (94d2b1c4903a17c029b343c02afe5ed7) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (ddb73b0e50b2b9e6115edd93a2ee8314) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (0faff6f59295a616585130b79b40cf56) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (9b9e596b89cc450adb0d8d8106c5de23) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (96710b7b73e8b145284d469017c94619) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (48ccd602fbdf5dbf99f2fcb035395b31) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (31c20e3773238419f0ad99e2fb2da7fa) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #3) to splat34 | |
| 2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (368276f9f3101752a52a9b3ef4ab090a) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #3) to splat34 | |
| 2017-05-15 13:59:45,216 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (6f3eadbb122384d3c186417e7c359d4d) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #3) to splat34 | |
| 2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (7507653dacef8da0f710f6eafeb5310f) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #3) to splat34 | |
| 2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (3da695760bd3750f38048853db4b3ada) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #3) to splat34 | |
| 2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (1f70d3824513abaa0150c020f134daf5) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #3) to splat34 | |
| 2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (4bbfb4d9e406c459222e1ea8304ecf2d) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #3) to splat34 | |
| 2017-05-15 13:59:45,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (94d2b1c4903a17c029b343c02afe5ed7) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #3) to splat34 | |
| 2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (ddb73b0e50b2b9e6115edd93a2ee8314) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #3) to splat34 | |
| 2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (0faff6f59295a616585130b79b40cf56) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #3) to splat34 | |
| 2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (9b9e596b89cc450adb0d8d8106c5de23) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #3) to splat34 | |
| 2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (96710b7b73e8b145284d469017c94619) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:45,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #3) to splat34 | |
| 2017-05-15 13:59:45,219 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (48ccd602fbdf5dbf99f2fcb035395b31) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:45,219 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #3) to splat34 | |
| 2017-05-15 13:59:45,233 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (368276f9f3101752a52a9b3ef4ab090a) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:45,233 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (31c20e3773238419f0ad99e2fb2da7fa) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:45,235 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (6f3eadbb122384d3c186417e7c359d4d) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:45,236 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (7507653dacef8da0f710f6eafeb5310f) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:45,237 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (3da695760bd3750f38048853db4b3ada) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:45,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (1f70d3824513abaa0150c020f134daf5) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:45,246 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (4bbfb4d9e406c459222e1ea8304ecf2d) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:45,253 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (94d2b1c4903a17c029b343c02afe5ed7) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:45,255 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (ddb73b0e50b2b9e6115edd93a2ee8314) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:45,261 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (0faff6f59295a616585130b79b40cf56) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:45,263 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (9b9e596b89cc450adb0d8d8106c5de23) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:45,264 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (96710b7b73e8b145284d469017c94619) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:45,264 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (48ccd602fbdf5dbf99f2fcb035395b31) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:45,329 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (4bbfb4d9e406c459222e1ea8304ecf2d) switched from RUNNING to FAILED. | |
| java.lang.IllegalStateException: Could not initialize keyed state backend. | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251) | |
| at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) | |
| at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) | |
| at java.lang.reflect.Constructor.newInstance(Constructor.java:423) | |
| at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) | |
| at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200) | |
| at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271) | |
| at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238) | |
| at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231) | |
| at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85) | |
| at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311) | |
| ... 6 more | |
| Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1468) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1399) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) | |
| at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) | |
| at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) | |
| at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220) | |
| ... 28 more | |
| 2017-05-15 13:59:45,329 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING. | |
| java.lang.IllegalStateException: Could not initialize keyed state backend. | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251) | |
| at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) | |
| at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) | |
| at java.lang.reflect.Constructor.newInstance(Constructor.java:423) | |
| at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) | |
| at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200) | |
| at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271) | |
| at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238) | |
| at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231) | |
| at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85) | |
| at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311) | |
| ... 6 more | |
| Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1468) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1399) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) | |
| at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) | |
| at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) | |
| at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220) | |
| ... 28 more | |
| 2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (31c20e3773238419f0ad99e2fb2da7fa) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (368276f9f3101752a52a9b3ef4ab090a) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (6f3eadbb122384d3c186417e7c359d4d) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (7507653dacef8da0f710f6eafeb5310f) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (3da695760bd3750f38048853db4b3ada) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (1f70d3824513abaa0150c020f134daf5) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (94d2b1c4903a17c029b343c02afe5ed7) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (ddb73b0e50b2b9e6115edd93a2ee8314) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (0faff6f59295a616585130b79b40cf56) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (9b9e596b89cc450adb0d8d8106c5de23) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:45,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (96710b7b73e8b145284d469017c94619) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:45,332 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (48ccd602fbdf5dbf99f2fcb035395b31) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:45,360 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (31c20e3773238419f0ad99e2fb2da7fa) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:45,361 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (7507653dacef8da0f710f6eafeb5310f) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:45,362 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (1f70d3824513abaa0150c020f134daf5) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:45,362 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (94d2b1c4903a17c029b343c02afe5ed7) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:45,363 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (ddb73b0e50b2b9e6115edd93a2ee8314) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:45,363 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (0faff6f59295a616585130b79b40cf56) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:45,364 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (9b9e596b89cc450adb0d8d8106c5de23) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:45,365 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (96710b7b73e8b145284d469017c94619) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:45,365 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (48ccd602fbdf5dbf99f2fcb035395b31) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:45,453 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 13:59:46,029 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (6f3eadbb122384d3c186417e7c359d4d) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:46,253 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (368276f9f3101752a52a9b3ef4ab090a) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:46,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (3da695760bd3750f38048853db4b3ada) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:46,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible. | |
| 2017-05-15 13:59:46,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING. | |
| 2017-05-15 13:59:46,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e). | |
| 2017-05-15 13:59:46,332 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ... | |
| 2017-05-15 13:59:50,474 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 13:59:55,494 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 13:59:56,332 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED. | |
| 2017-05-15 13:59:56,332 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper. | |
| 2017-05-15 13:59:56,334 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 1 checkpoints in ZooKeeper. | |
| 2017-05-15 13:59:56,334 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29. | |
| 2017-05-15 13:59:56,341 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29. | |
| 2017-05-15 13:59:56,345 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 29 @ 1494849550373 for 1f387e6136f6faf1824808898750878e. | |
| 2017-05-15 13:59:56,345 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore | |
| 2017-05-15 13:59:56,345 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING. | |
| 2017-05-15 13:59:56,345 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (6b4d03aa9e6202c7dac653e1d6b34047) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:56,345 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (4d136e14c501f525db8d65b3fa800cd9) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:56,345 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (492faa5407fe4b7e3d9a84d886180b0e) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (6be80899f60475fc6b3a25f5e78d51fa) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (22665b351c702c2d90d601b97c8f4020) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a0f5b8bc2c01d52b595fc4301ecc9a38) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (251dad84bbd0cdfd36e6dd5845925231) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (dcba9058b8b1323d726246f33de13a78) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (80f9ee6b489b0afaa4f0539d5d15297a) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (471d42853a1b07df565bdb20dadc5168) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (c8546b82cdb3269499d7580845e5d039) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (c12496053493ec64c7d3ca72a79abbf4) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (cee78d5edb7d161f1be2742a46a52f2f) switched from CREATED to SCHEDULED. | |
| 2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (6b4d03aa9e6202c7dac653e1d6b34047) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:56,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #4) to splat34 | |
| 2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (4d136e14c501f525db8d65b3fa800cd9) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #4) to splat34 | |
| 2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (492faa5407fe4b7e3d9a84d886180b0e) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #4) to splat34 | |
| 2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (6be80899f60475fc6b3a25f5e78d51fa) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #4) to splat34 | |
| 2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (22665b351c702c2d90d601b97c8f4020) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #4) to splat34 | |
| 2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a0f5b8bc2c01d52b595fc4301ecc9a38) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #4) to splat34 | |
| 2017-05-15 13:59:56,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (251dad84bbd0cdfd36e6dd5845925231) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #4) to splat34 | |
| 2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (dcba9058b8b1323d726246f33de13a78) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #4) to splat34 | |
| 2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (80f9ee6b489b0afaa4f0539d5d15297a) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #4) to splat34 | |
| 2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (471d42853a1b07df565bdb20dadc5168) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #4) to splat34 | |
| 2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (c8546b82cdb3269499d7580845e5d039) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #4) to splat34 | |
| 2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (c12496053493ec64c7d3ca72a79abbf4) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #4) to splat34 | |
| 2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (cee78d5edb7d161f1be2742a46a52f2f) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 13:59:56,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #4) to splat34 | |
| 2017-05-15 13:59:56,360 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (6b4d03aa9e6202c7dac653e1d6b34047) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:56,360 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (4d136e14c501f525db8d65b3fa800cd9) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:56,366 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (492faa5407fe4b7e3d9a84d886180b0e) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:56,369 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (6be80899f60475fc6b3a25f5e78d51fa) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:56,370 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (22665b351c702c2d90d601b97c8f4020) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:56,372 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a0f5b8bc2c01d52b595fc4301ecc9a38) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:56,373 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (251dad84bbd0cdfd36e6dd5845925231) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:56,374 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (dcba9058b8b1323d726246f33de13a78) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:56,375 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (80f9ee6b489b0afaa4f0539d5d15297a) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:56,376 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (471d42853a1b07df565bdb20dadc5168) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:56,380 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (c8546b82cdb3269499d7580845e5d039) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:56,381 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (c12496053493ec64c7d3ca72a79abbf4) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:56,382 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (cee78d5edb7d161f1be2742a46a52f2f) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 13:59:56,453 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (251dad84bbd0cdfd36e6dd5845925231) switched from RUNNING to FAILED. | |
| java.lang.IllegalStateException: Could not initialize keyed state backend. | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251) | |
| at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) | |
| at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) | |
| at java.lang.reflect.Constructor.newInstance(Constructor.java:423) | |
| at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) | |
| at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200) | |
| at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271) | |
| at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238) | |
| at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231) | |
| at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85) | |
| at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311) | |
| ... 6 more | |
| Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1468) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1399) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) | |
| at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) | |
| at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) | |
| at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220) | |
| ... 28 more | |
| 2017-05-15 13:59:56,454 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING. | |
| java.lang.IllegalStateException: Could not initialize keyed state backend. | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251) | |
| at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) | |
| at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) | |
| at java.lang.reflect.Constructor.newInstance(Constructor.java:423) | |
| at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) | |
| at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200) | |
| at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271) | |
| at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238) | |
| at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231) | |
| at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85) | |
| at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311) | |
| ... 6 more | |
| Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1468) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1399) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) | |
| at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) | |
| at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) | |
| at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220) | |
| ... 28 more | |
| 2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (6b4d03aa9e6202c7dac653e1d6b34047) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (4d136e14c501f525db8d65b3fa800cd9) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (492faa5407fe4b7e3d9a84d886180b0e) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (6be80899f60475fc6b3a25f5e78d51fa) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (22665b351c702c2d90d601b97c8f4020) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a0f5b8bc2c01d52b595fc4301ecc9a38) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (dcba9058b8b1323d726246f33de13a78) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (80f9ee6b489b0afaa4f0539d5d15297a) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (471d42853a1b07df565bdb20dadc5168) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (c8546b82cdb3269499d7580845e5d039) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:56,456 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (c12496053493ec64c7d3ca72a79abbf4) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:56,457 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (cee78d5edb7d161f1be2742a46a52f2f) switched from RUNNING to CANCELING. | |
| 2017-05-15 13:59:56,475 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (6b4d03aa9e6202c7dac653e1d6b34047) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:56,477 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (6be80899f60475fc6b3a25f5e78d51fa) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:56,477 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (a0f5b8bc2c01d52b595fc4301ecc9a38) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:56,478 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (dcba9058b8b1323d726246f33de13a78) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:56,479 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (80f9ee6b489b0afaa4f0539d5d15297a) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:56,479 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (c8546b82cdb3269499d7580845e5d039) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:56,480 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (471d42853a1b07df565bdb20dadc5168) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:56,480 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (c12496053493ec64c7d3ca72a79abbf4) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:56,480 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (cee78d5edb7d161f1be2742a46a52f2f) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:57,029 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (492faa5407fe4b7e3d9a84d886180b0e) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:57,044 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (4d136e14c501f525db8d65b3fa800cd9) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:57,228 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (22665b351c702c2d90d601b97c8f4020) switched from CANCELING to CANCELED. | |
| 2017-05-15 13:59:57,228 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible. | |
| 2017-05-15 13:59:57,228 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING. | |
| 2017-05-15 13:59:57,228 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e). | |
| 2017-05-15 13:59:57,228 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ... | |
| 2017-05-15 14:00:00,513 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 14:00:05,534 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 14:00:07,228 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED. | |
| 2017-05-15 14:00:07,228 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper. | |
| 2017-05-15 14:00:07,230 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 1 checkpoints in ZooKeeper. | |
| 2017-05-15 14:00:07,230 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29. | |
| 2017-05-15 14:00:07,236 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29. | |
| 2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 29 @ 1494849550373 for 1f387e6136f6faf1824808898750878e. | |
| 2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore | |
| 2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING. | |
| 2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (9a3aa7d3fc1ee24194edc1f4323d19d2) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96f3c203d94359531e06f628016ada20) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d0a5fda1fc993891b3cb688365ebe1f8) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (352737692e7062ccf73d6f3bca65ec57) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (c02268503a51fb91c81a455663598720) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (92d265fad55c908b077782341a0cf51f) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (d4c3b674b6eef5254d07f67204b4ac4b) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (9d336b1b9fb2b473ff1101f0b7aef44b) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (b38c36dc40c13d520c16b73364fc693b) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:07,240 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (bfe2339eb2ad2ccd3ed65fea417766af) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (4d610b0414eb5ac25e8c36cb0ee89ef7) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (7fa7764d87ad8cc4affa6aa5873762a4) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7c46499da6faf343f474bebb0bd85c4b) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (9a3aa7d3fc1ee24194edc1f4323d19d2) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #5) to splat34 | |
| 2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96f3c203d94359531e06f628016ada20) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #5) to splat34 | |
| 2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d0a5fda1fc993891b3cb688365ebe1f8) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #5) to splat34 | |
| 2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (352737692e7062ccf73d6f3bca65ec57) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #5) to splat34 | |
| 2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (c02268503a51fb91c81a455663598720) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #5) to splat34 | |
| 2017-05-15 14:00:07,241 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (92d265fad55c908b077782341a0cf51f) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #5) to splat34 | |
| 2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (d4c3b674b6eef5254d07f67204b4ac4b) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #5) to splat34 | |
| 2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (9d336b1b9fb2b473ff1101f0b7aef44b) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #5) to splat34 | |
| 2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (b38c36dc40c13d520c16b73364fc693b) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #5) to splat34 | |
| 2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (bfe2339eb2ad2ccd3ed65fea417766af) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #5) to splat34 | |
| 2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (4d610b0414eb5ac25e8c36cb0ee89ef7) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #5) to splat34 | |
| 2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (7fa7764d87ad8cc4affa6aa5873762a4) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #5) to splat34 | |
| 2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7c46499da6faf343f474bebb0bd85c4b) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:07,242 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #5) to splat34 | |
| 2017-05-15 14:00:07,251 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (9a3aa7d3fc1ee24194edc1f4323d19d2) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:07,251 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96f3c203d94359531e06f628016ada20) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:07,253 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d0a5fda1fc993891b3cb688365ebe1f8) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:07,253 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (352737692e7062ccf73d6f3bca65ec57) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:07,254 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (c02268503a51fb91c81a455663598720) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:07,255 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (92d265fad55c908b077782341a0cf51f) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:07,255 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (d4c3b674b6eef5254d07f67204b4ac4b) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:07,256 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (9d336b1b9fb2b473ff1101f0b7aef44b) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:07,261 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (b38c36dc40c13d520c16b73364fc693b) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:07,263 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (bfe2339eb2ad2ccd3ed65fea417766af) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:07,267 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (4d610b0414eb5ac25e8c36cb0ee89ef7) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:07,269 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (7fa7764d87ad8cc4affa6aa5873762a4) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:07,269 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7c46499da6faf343f474bebb0bd85c4b) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:07,335 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (d4c3b674b6eef5254d07f67204b4ac4b) switched from RUNNING to FAILED. | |
| java.lang.IllegalStateException: Could not initialize keyed state backend. | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251) | |
| at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) | |
| at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) | |
| at java.lang.reflect.Constructor.newInstance(Constructor.java:423) | |
| at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) | |
| at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200) | |
| at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271) | |
| at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238) | |
| at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231) | |
| at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85) | |
| at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311) | |
| ... 6 more | |
| Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1468) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1399) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) | |
| at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) | |
| at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) | |
| at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220) | |
| ... 28 more | |
| 2017-05-15 14:00:07,336 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING. | |
| java.lang.IllegalStateException: Could not initialize keyed state backend. | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251) | |
| at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) | |
| at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) | |
| at java.lang.reflect.Constructor.newInstance(Constructor.java:423) | |
| at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) | |
| at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200) | |
| at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271) | |
| at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238) | |
| at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231) | |
| at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85) | |
| at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311) | |
| ... 6 more | |
| Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1468) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1399) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) | |
| at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) | |
| at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) | |
| at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220) | |
| ... 28 more | |
| 2017-05-15 14:00:07,337 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (9a3aa7d3fc1ee24194edc1f4323d19d2) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:07,337 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96f3c203d94359531e06f628016ada20) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:07,337 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d0a5fda1fc993891b3cb688365ebe1f8) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:07,337 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (352737692e7062ccf73d6f3bca65ec57) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:07,337 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (c02268503a51fb91c81a455663598720) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:07,338 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (92d265fad55c908b077782341a0cf51f) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:07,338 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (9d336b1b9fb2b473ff1101f0b7aef44b) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:07,338 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (b38c36dc40c13d520c16b73364fc693b) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:07,338 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (bfe2339eb2ad2ccd3ed65fea417766af) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:07,338 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (4d610b0414eb5ac25e8c36cb0ee89ef7) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:07,338 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (7fa7764d87ad8cc4affa6aa5873762a4) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:07,338 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7c46499da6faf343f474bebb0bd85c4b) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:07,375 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (9a3aa7d3fc1ee24194edc1f4323d19d2) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:07,376 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (352737692e7062ccf73d6f3bca65ec57) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:07,377 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (92d265fad55c908b077782341a0cf51f) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:07,377 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (9d336b1b9fb2b473ff1101f0b7aef44b) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:07,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (b38c36dc40c13d520c16b73364fc693b) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:07,378 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (4d610b0414eb5ac25e8c36cb0ee89ef7) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:07,379 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (bfe2339eb2ad2ccd3ed65fea417766af) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:07,380 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (7fa7764d87ad8cc4affa6aa5873762a4) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:07,380 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (7c46499da6faf343f474bebb0bd85c4b) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:08,275 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96f3c203d94359531e06f628016ada20) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:08,475 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d0a5fda1fc993891b3cb688365ebe1f8) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:08,476 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (c02268503a51fb91c81a455663598720) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:08,476 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible. | |
| 2017-05-15 14:00:08,476 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING. | |
| 2017-05-15 14:00:08,476 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e). | |
| 2017-05-15 14:00:08,476 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ... | |
| 2017-05-15 14:00:10,554 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 14:00:15,574 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 14:00:18,476 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED. | |
| 2017-05-15 14:00:18,476 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper. | |
| 2017-05-15 14:00:18,478 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 1 checkpoints in ZooKeeper. | |
| 2017-05-15 14:00:18,478 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29. | |
| 2017-05-15 14:00:18,484 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29. | |
| 2017-05-15 14:00:18,486 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 29 @ 1494849550373 for 1f387e6136f6faf1824808898750878e. | |
| 2017-05-15 14:00:18,487 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore | |
| 2017-05-15 14:00:18,487 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING. | |
| 2017-05-15 14:00:18,487 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (aa487e650bf0d4ea44955c44c585c488) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:18,487 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (1e0cf112d2d80c488174a95ae7785245) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:18,487 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d711c61450623bb20a8738f82f086b3c) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:18,487 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (1adca9fbb82bdb66bce22857e11358b9) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (e32a60ff98ce85f35bd77d8559f884a4) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (694359a780a126d897e15b2fefbf02be) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (abfa36c27ff9ef6b40ec1dc41833ea2a) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (ecb484995ca035fc697cb5b7f2d6432f) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (892d0d66545aa33a8a31d323f86d8d3c) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (fea28107146851d415f3b6976474212c) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (ef1c63edc779c463800fbb272f4ef3ca) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (8d9b795aa5a8d586e3b9edc827e07c96) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (31aa1ccc883d064e494045e0ef9cc4c9) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (aa487e650bf0d4ea44955c44c585c488) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #6) to splat34 | |
| 2017-05-15 14:00:18,488 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (1e0cf112d2d80c488174a95ae7785245) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #6) to splat34 | |
| 2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d711c61450623bb20a8738f82f086b3c) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #6) to splat34 | |
| 2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (1adca9fbb82bdb66bce22857e11358b9) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #6) to splat34 | |
| 2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (e32a60ff98ce85f35bd77d8559f884a4) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #6) to splat34 | |
| 2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (694359a780a126d897e15b2fefbf02be) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #6) to splat34 | |
| 2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (abfa36c27ff9ef6b40ec1dc41833ea2a) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:18,489 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #6) to splat34 | |
| 2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (ecb484995ca035fc697cb5b7f2d6432f) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #6) to splat34 | |
| 2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (892d0d66545aa33a8a31d323f86d8d3c) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #6) to splat34 | |
| 2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (fea28107146851d415f3b6976474212c) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #6) to splat34 | |
| 2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (ef1c63edc779c463800fbb272f4ef3ca) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #6) to splat34 | |
| 2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (8d9b795aa5a8d586e3b9edc827e07c96) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #6) to splat34 | |
| 2017-05-15 14:00:18,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (31aa1ccc883d064e494045e0ef9cc4c9) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:18,491 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #6) to splat34 | |
| 2017-05-15 14:00:18,508 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (aa487e650bf0d4ea44955c44c585c488) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:18,509 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (1e0cf112d2d80c488174a95ae7785245) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:18,509 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d711c61450623bb20a8738f82f086b3c) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:18,510 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (1adca9fbb82bdb66bce22857e11358b9) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:18,511 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (e32a60ff98ce85f35bd77d8559f884a4) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:18,511 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (694359a780a126d897e15b2fefbf02be) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:18,511 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (abfa36c27ff9ef6b40ec1dc41833ea2a) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:18,512 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (ecb484995ca035fc697cb5b7f2d6432f) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:18,512 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (892d0d66545aa33a8a31d323f86d8d3c) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:18,512 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (fea28107146851d415f3b6976474212c) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:18,512 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (ef1c63edc779c463800fbb272f4ef3ca) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:18,513 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (8d9b795aa5a8d586e3b9edc827e07c96) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:18,513 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (31aa1ccc883d064e494045e0ef9cc4c9) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:18,559 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (abfa36c27ff9ef6b40ec1dc41833ea2a) switched from RUNNING to FAILED. | |
| java.lang.IllegalStateException: Could not initialize keyed state backend. | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251) | |
| at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) | |
| at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) | |
| at java.lang.reflect.Constructor.newInstance(Constructor.java:423) | |
| at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) | |
| at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200) | |
| at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271) | |
| at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238) | |
| at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231) | |
| at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85) | |
| at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311) | |
| ... 6 more | |
| Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1468) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1399) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) | |
| at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) | |
| at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) | |
| at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220) | |
| ... 28 more | |
| 2017-05-15 14:00:18,560 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING. | |
| java.lang.IllegalStateException: Could not initialize keyed state backend. | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251) | |
| at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) | |
| at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) | |
| at java.lang.reflect.Constructor.newInstance(Constructor.java:423) | |
| at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) | |
| at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200) | |
| at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271) | |
| at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238) | |
| at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231) | |
| at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85) | |
| at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311) | |
| ... 6 more | |
| Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1468) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1399) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) | |
| at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) | |
| at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) | |
| at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220) | |
| ... 28 more | |
| 2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (aa487e650bf0d4ea44955c44c585c488) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (1e0cf112d2d80c488174a95ae7785245) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d711c61450623bb20a8738f82f086b3c) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (1adca9fbb82bdb66bce22857e11358b9) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (e32a60ff98ce85f35bd77d8559f884a4) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (694359a780a126d897e15b2fefbf02be) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (ecb484995ca035fc697cb5b7f2d6432f) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:18,561 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (892d0d66545aa33a8a31d323f86d8d3c) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:18,562 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (fea28107146851d415f3b6976474212c) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:18,562 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (ef1c63edc779c463800fbb272f4ef3ca) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:18,562 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (8d9b795aa5a8d586e3b9edc827e07c96) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:18,562 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (31aa1ccc883d064e494045e0ef9cc4c9) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:18,564 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (aa487e650bf0d4ea44955c44c585c488) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:18,570 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (1adca9fbb82bdb66bce22857e11358b9) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:18,572 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (694359a780a126d897e15b2fefbf02be) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:18,573 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (ecb484995ca035fc697cb5b7f2d6432f) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:18,574 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (892d0d66545aa33a8a31d323f86d8d3c) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:18,574 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (8d9b795aa5a8d586e3b9edc827e07c96) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:18,574 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (ef1c63edc779c463800fbb272f4ef3ca) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:18,574 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (fea28107146851d415f3b6976474212c) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:18,574 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (31aa1ccc883d064e494045e0ef9cc4c9) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:19,373 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (e32a60ff98ce85f35bd77d8559f884a4) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:19,373 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (d711c61450623bb20a8738f82f086b3c) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:19,640 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (1e0cf112d2d80c488174a95ae7785245) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:19,641 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible. | |
| 2017-05-15 14:00:19,641 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING. | |
| 2017-05-15 14:00:19,641 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e). | |
| 2017-05-15 14:00:19,641 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ... | |
| 2017-05-15 14:00:20,594 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 14:00:25,615 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 14:00:29,641 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED. | |
| 2017-05-15 14:00:29,641 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper. | |
| 2017-05-15 14:00:29,643 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 1 checkpoints in ZooKeeper. | |
| 2017-05-15 14:00:29,643 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29. | |
| 2017-05-15 14:00:29,651 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29. | |
| 2017-05-15 14:00:29,655 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 29 @ 1494849550373 for 1f387e6136f6faf1824808898750878e. | |
| 2017-05-15 14:00:29,655 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore | |
| 2017-05-15 14:00:29,655 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING. | |
| 2017-05-15 14:00:29,655 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (01143eb487fbceddc5e0531b9a20e296) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:29,655 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96d5f86d669ad54950e9747d48b064b2) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7bcaeb685ce42ec560e2ecb4df1f7216) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (c797dfa92d7b0f02070e67b87ad8a203) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (a9eb5578023189d2f414c58c8219d046) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (34c8fe5375c64b0e4f343e64517371bd) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (035f512337eb68ba9f2433ece44a2d19) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3525adaf53e5b366d25443e77ad2bbe4) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (4df4c3554d2628d630716ff5a0fc57c5) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (78b22ef5e7356ce532756eb9063e3803) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3643843191636ca6c6317a1b0280e347) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (ed0c35d4812b8fcbcf872ee928a459a5) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (097d118f9a601c4a119751a4dfb05add) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (01143eb487fbceddc5e0531b9a20e296) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:29,656 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #7) to splat34 | |
| 2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96d5f86d669ad54950e9747d48b064b2) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #7) to splat34 | |
| 2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7bcaeb685ce42ec560e2ecb4df1f7216) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #7) to splat34 | |
| 2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (c797dfa92d7b0f02070e67b87ad8a203) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #7) to splat34 | |
| 2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (a9eb5578023189d2f414c58c8219d046) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #7) to splat34 | |
| 2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (34c8fe5375c64b0e4f343e64517371bd) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #7) to splat34 | |
| 2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (035f512337eb68ba9f2433ece44a2d19) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:29,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #7) to splat34 | |
| 2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3525adaf53e5b366d25443e77ad2bbe4) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #7) to splat34 | |
| 2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (4df4c3554d2628d630716ff5a0fc57c5) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #7) to splat34 | |
| 2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (78b22ef5e7356ce532756eb9063e3803) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #7) to splat34 | |
| 2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3643843191636ca6c6317a1b0280e347) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #7) to splat34 | |
| 2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (ed0c35d4812b8fcbcf872ee928a459a5) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:29,658 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #7) to splat34 | |
| 2017-05-15 14:00:29,659 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (097d118f9a601c4a119751a4dfb05add) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:29,659 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #7) to splat34 | |
| 2017-05-15 14:00:29,690 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (01143eb487fbceddc5e0531b9a20e296) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:29,692 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96d5f86d669ad54950e9747d48b064b2) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:29,692 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7bcaeb685ce42ec560e2ecb4df1f7216) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:29,692 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (c797dfa92d7b0f02070e67b87ad8a203) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:29,693 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (a9eb5578023189d2f414c58c8219d046) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:29,693 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (34c8fe5375c64b0e4f343e64517371bd) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:29,693 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (035f512337eb68ba9f2433ece44a2d19) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:29,694 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3525adaf53e5b366d25443e77ad2bbe4) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:29,694 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (4df4c3554d2628d630716ff5a0fc57c5) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:29,695 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (78b22ef5e7356ce532756eb9063e3803) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:29,695 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3643843191636ca6c6317a1b0280e347) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:29,695 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (ed0c35d4812b8fcbcf872ee928a459a5) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:29,695 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (097d118f9a601c4a119751a4dfb05add) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:29,779 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (035f512337eb68ba9f2433ece44a2d19) switched from RUNNING to FAILED. | |
| java.lang.IllegalStateException: Could not initialize keyed state backend. | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251) | |
| at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) | |
| at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) | |
| at java.lang.reflect.Constructor.newInstance(Constructor.java:423) | |
| at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) | |
| at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200) | |
| at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271) | |
| at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238) | |
| at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231) | |
| at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85) | |
| at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311) | |
| ... 6 more | |
| Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1468) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1399) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) | |
| at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) | |
| at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) | |
| at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220) | |
| ... 28 more | |
| 2017-05-15 14:00:29,779 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING. | |
| java.lang.IllegalStateException: Could not initialize keyed state backend. | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251) | |
| at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) | |
| at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) | |
| at java.lang.reflect.Constructor.newInstance(Constructor.java:423) | |
| at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) | |
| at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200) | |
| at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271) | |
| at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238) | |
| at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231) | |
| at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85) | |
| at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311) | |
| ... 6 more | |
| Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1468) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1399) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) | |
| at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) | |
| at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) | |
| at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220) | |
| ... 28 more | |
| 2017-05-15 14:00:29,780 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (01143eb487fbceddc5e0531b9a20e296) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:29,780 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96d5f86d669ad54950e9747d48b064b2) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:29,780 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7bcaeb685ce42ec560e2ecb4df1f7216) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (c797dfa92d7b0f02070e67b87ad8a203) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (a9eb5578023189d2f414c58c8219d046) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (34c8fe5375c64b0e4f343e64517371bd) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3525adaf53e5b366d25443e77ad2bbe4) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (4df4c3554d2628d630716ff5a0fc57c5) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (78b22ef5e7356ce532756eb9063e3803) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3643843191636ca6c6317a1b0280e347) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (ed0c35d4812b8fcbcf872ee928a459a5) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:29,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (097d118f9a601c4a119751a4dfb05add) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (01143eb487fbceddc5e0531b9a20e296) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (c797dfa92d7b0f02070e67b87ad8a203) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (34c8fe5375c64b0e4f343e64517371bd) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (3525adaf53e5b366d25443e77ad2bbe4) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (4df4c3554d2628d630716ff5a0fc57c5) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3643843191636ca6c6317a1b0280e347) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (ed0c35d4812b8fcbcf872ee928a459a5) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:29,792 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (78b22ef5e7356ce532756eb9063e3803) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:29,793 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (097d118f9a601c4a119751a4dfb05add) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:30,451 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (a9eb5578023189d2f414c58c8219d046) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:30,550 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (7bcaeb685ce42ec560e2ecb4df1f7216) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:30,617 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (96d5f86d669ad54950e9747d48b064b2) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:30,617 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible. | |
| 2017-05-15 14:00:30,634 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 14:00:30,647 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING. | |
| 2017-05-15 14:00:30,647 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e). | |
| 2017-05-15 14:00:30,647 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ... | |
| 2017-05-15 14:00:35,655 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 14:00:40,647 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CREATED. | |
| 2017-05-15 14:00:40,647 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper. | |
| 2017-05-15 14:00:40,651 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 1 checkpoints in ZooKeeper. | |
| 2017-05-15 14:00:40,651 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29. | |
| 2017-05-15 14:00:40,658 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29. | |
| 2017-05-15 14:00:40,661 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 29 @ 1494849550373 for 1f387e6136f6faf1824808898750878e. | |
| 2017-05-15 14:00:40,662 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore | |
| 2017-05-15 14:00:40,662 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state CREATED to RUNNING. | |
| 2017-05-15 14:00:40,662 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (557342eb5035b03d66578c9928cc4cdd) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:40,662 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (f5901179c871f609f87d7b2dbdf08689) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:40,662 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (a891ce4e388ef6c253be967cdc1b02bf) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:40,662 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (b849a4add5d6aecea749d725dfedbf95) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (bd514206cb90c8c7af5920616d3d4de4) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (515e891fa796a2a6e205c851fd7a550a) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (ef42c4911c40011dc26cd1a03e490eea) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (1e860c7508dd154388e4aa46bd460037) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (2a66d35d8618b87698cf83ac982770f2) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (c410ce8973e8027cd1b297f2ad5f6930) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3d5dad52595a9deb8aebf08d815a3620) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (67a553e88df17ecbace5821563b1bb4c) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (684a8d45c657df79a93db15a866e141f) switched from CREATED to SCHEDULED. | |
| 2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (557342eb5035b03d66578c9928cc4cdd) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSource-15 (1/1) (attempt #8) to splat34 | |
| 2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (f5901179c871f609f87d7b2dbdf08689) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:40,663 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (attempt #8) to splat34 | |
| 2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (a891ce4e388ef6c253be967cdc1b02bf) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (attempt #8) to splat34 | |
| 2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (b849a4add5d6aecea749d725dfedbf95) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Query Job Info (1/1) (attempt #8) to splat34 | |
| 2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (bd514206cb90c8c7af5920616d3d4de4) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (attempt #8) to splat34 | |
| 2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (515e891fa796a2a6e205c851fd7a550a) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Fields and Ids -> Filter Errors and Notifications (1/1) (attempt #8) to splat34 | |
| 2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (ef42c4911c40011dc26cd1a03e490eea) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (attempt #8) to splat34 | |
| 2017-05-15 14:00:40,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (1e860c7508dd154388e4aa46bd460037) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Window aggregator (1/1) (attempt #8) to splat34 | |
| 2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (2a66d35d8618b87698cf83ac982770f2) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keep last -> NoOp -> Create Aggrigato events (1/1) (attempt #8) to splat34 | |
| 2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (c410ce8973e8027cd1b297f2ad5f6930) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying MySql output info -> Filter (1/1) (attempt #8) to splat34 | |
| 2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3d5dad52595a9deb8aebf08d815a3620) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying To DeploymentInfo (1/1) (attempt #8) to splat34 | |
| 2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (67a553e88df17ecbace5821563b1bb4c) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying IterationSink-15 (1/1) (attempt #8) to splat34 | |
| 2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (684a8d45c657df79a93db15a866e141f) switched from SCHEDULED to DEPLOYING. | |
| 2017-05-15 14:00:40,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Create Job information (1/1) (attempt #8) to splat34 | |
| 2017-05-15 14:00:40,668 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (557342eb5035b03d66578c9928cc4cdd) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:40,668 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (f5901179c871f609f87d7b2dbdf08689) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:40,669 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (a891ce4e388ef6c253be967cdc1b02bf) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:40,670 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (b849a4add5d6aecea749d725dfedbf95) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:40,671 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (bd514206cb90c8c7af5920616d3d4de4) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:40,671 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (515e891fa796a2a6e205c851fd7a550a) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:40,674 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 14:00:40,676 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (ef42c4911c40011dc26cd1a03e490eea) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:40,676 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (1e860c7508dd154388e4aa46bd460037) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:40,677 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (2a66d35d8618b87698cf83ac982770f2) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:40,678 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (c410ce8973e8027cd1b297f2ad5f6930) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:40,679 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3d5dad52595a9deb8aebf08d815a3620) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:40,680 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (67a553e88df17ecbace5821563b1bb4c) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:40,681 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (684a8d45c657df79a93db15a866e141f) switched from DEPLOYING to RUNNING. | |
| 2017-05-15 14:00:40,745 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Execute processors -> (Filter processor info -> Filter Failures, Filter BEA) (1/1) (ef42c4911c40011dc26cd1a03e490eea) switched from RUNNING to FAILED. | |
| java.lang.IllegalStateException: Could not initialize keyed state backend. | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251) | |
| at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) | |
| at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) | |
| at java.lang.reflect.Constructor.newInstance(Constructor.java:423) | |
| at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) | |
| at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200) | |
| at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271) | |
| at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238) | |
| at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231) | |
| at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85) | |
| at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311) | |
| ... 6 more | |
| Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1468) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1399) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) | |
| at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) | |
| at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) | |
| at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220) | |
| ... 28 more | |
| 2017-05-15 14:00:40,745 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RUNNING to FAILING. | |
| java.lang.IllegalStateException: Could not initialize keyed state backend. | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251) | |
| at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: java.io.FileNotFoundException: File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) | |
| at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) | |
| at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) | |
| at java.lang.reflect.Constructor.newInstance(Constructor.java:423) | |
| at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) | |
| at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) | |
| at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1200) | |
| at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:271) | |
| at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:238) | |
| at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:231) | |
| at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1498) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) | |
| at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) | |
| at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:404) | |
| at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:48) | |
| at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85) | |
| at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readStateData(RocksDBKeyedStateBackend.java:1250) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.readAllStateData(RocksDBKeyedStateBackend.java:1439) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restoreInstance(RocksDBKeyedStateBackend.java:1295) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalRestoreOperation.restore(RocksDBKeyedStateBackend.java:1474) | |
| at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.restore(RocksDBKeyedStateBackend.java:968) | |
| at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:771) | |
| at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311) | |
| ... 6 more | |
| Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink/checkpoints/event.bifrost.log/1f387e6136f6faf1824808898750878e/chk-29/ad381141-3c79-42d4-99f7-2e4a47adf81c | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) | |
| at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2008) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1978) | |
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1891) | |
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:572) | |
| at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:89) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) | |
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at javax.security.auth.Subject.doAs(Subject.java:422) | |
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) | |
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1468) | |
| at org.apache.hadoop.ipc.Client.call(Client.java:1399) | |
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) | |
| at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) | |
| at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) | |
| at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) | |
| at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source) | |
| at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1220) | |
| ... 28 more | |
| 2017-05-15 14:00:40,746 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (557342eb5035b03d66578c9928cc4cdd) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:40,746 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (f5901179c871f609f87d7b2dbdf08689) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:40,746 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (a891ce4e388ef6c253be967cdc1b02bf) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:40,746 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (b849a4add5d6aecea749d725dfedbf95) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (bd514206cb90c8c7af5920616d3d4de4) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (515e891fa796a2a6e205c851fd7a550a) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (1e860c7508dd154388e4aa46bd460037) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (2a66d35d8618b87698cf83ac982770f2) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (c410ce8973e8027cd1b297f2ad5f6930) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3d5dad52595a9deb8aebf08d815a3620) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (67a553e88df17ecbace5821563b1bb4c) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:40,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (684a8d45c657df79a93db15a866e141f) switched from RUNNING to CANCELING. | |
| 2017-05-15 14:00:40,756 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSource-15 (1/1) (557342eb5035b03d66578c9928cc4cdd) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:40,757 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Query Job Info (1/1) (b849a4add5d6aecea749d725dfedbf95) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:40,757 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Fields and Ids -> Filter Errors and Notifications (1/1) (515e891fa796a2a6e205c851fd7a550a) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:40,757 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window aggregator (1/1) (1e860c7508dd154388e4aa46bd460037) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:40,757 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Keep last -> NoOp -> Create Aggrigato events (1/1) (2a66d35d8618b87698cf83ac982770f2) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:40,757 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - To DeploymentInfo (1/1) (3d5dad52595a9deb8aebf08d815a3620) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:40,758 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - IterationSink-15 (1/1) (67a553e88df17ecbace5821563b1bb4c) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:40,758 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - MySql output info -> Filter (1/1) (c410ce8973e8027cd1b297f2ad5f6930) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:40,758 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Create Job information (1/1) (684a8d45c657df79a93db15a866e141f) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:41,540 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbeaDeploymentsplattest1] -> Max watermark -> Drop errors -> Filter for topic (1/1) (bd514206cb90c8c7af5920616d3d4de4) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:41,639 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[rbea.state.event.bifrost.log] -> Timestamp assigner -> Filter state updates -> Wrap events (1/1) (a891ce4e388ef6c253be967cdc1b02bf) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:41,904 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Kafka[event.bifrost.log] -> Timestamp assigner -> Wrap events (1/1) (f5901179c871f609f87d7b2dbdf08689) switched from CANCELING to CANCELED. | |
| 2017-05-15 14:00:41,904 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job event.bifrost.log (1f387e6136f6faf1824808898750878e) if no longer possible. | |
| 2017-05-15 14:00:41,904 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state FAILING to RESTARTING. | |
| 2017-05-15 14:00:41,904 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job event.bifrost.log (1f387e6136f6faf1824808898750878e). | |
| 2017-05-15 14:00:41,904 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ... | |
| 2017-05-15 14:00:43,929 INFO org.apache.flink.yarn.YarnJobManager - Trying to cancel job with ID 1f387e6136f6faf1824808898750878e. | |
| 2017-05-15 14:00:43,930 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job event.bifrost.log (1f387e6136f6faf1824808898750878e) switched from state RESTARTING to CANCELED. | |
| 2017-05-15 14:00:43,930 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Stopping checkpoint coordinator for job 1f387e6136f6faf1824808898750878e | |
| 2017-05-15 14:00:43,930 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Shutting down | |
| 2017-05-15 14:00:43,931 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Removing /flink-splat/application_1494426363399_0021/checkpoints/1f387e6136f6faf1824808898750878e from ZooKeeper | |
| 2017-05-15 14:00:43,933 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 29. | |
| 2017-05-15 14:00:43,936 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter - Shutting down. | |
| 2017-05-15 14:00:43,936 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter - Removing /checkpoint-counter/1f387e6136f6faf1824808898750878e from ZooKeeper | |
| 2017-05-15 14:00:43,936 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Canceled during restart. | |
| 2017-05-15 14:00:43,938 INFO org.apache.flink.runtime.checkpoint.CompletedCheckpoint - Persistent checkpoint with ID 29 at 'hdfs:/flink/external-checkpoints/bifrost/checkpoint_metadata-88c22ff47ca0' not discarded. | |
| 2017-05-15 14:00:43,982 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Removed job graph 1f387e6136f6faf1824808898750878e from ZooKeeper. | |
| 2017-05-15 14:00:45,694 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@splat35.sto.midasplayer.com:37378] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@splat35.sto.midasplayer.com:37378]] Caused by: [Connection refused: splat35.sto.midasplayer.com/172.26.83.104:37378] | |
| 2017-05-15 14:00:46,661 INFO org.apache.flink.yarn.YarnJobManager - Job with ID 1f387e6136f6faf1824808898750878e is in terminal state CANCELED. Shutting down session | |
| 2017-05-15 14:00:46,662 INFO org.apache.flink.yarn.YarnJobManager - Stopping JobManager with final application status FAILED and diagnostics: The monitored job with ID 1f387e6136f6faf1824808898750878e has failed to complete. | |
| 2017-05-15 14:00:46,663 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Shutting down cluster with status FAILED : The monitored job with ID 1f387e6136f6faf1824808898750878e has failed to complete. | |
| 2017-05-15 14:00:46,663 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Unregistering application from the YARN Resource Manager | |
| 2017-05-15 14:00:46,669 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Waiting for application to be successfully unregistered. | |
| 2017-05-15 14:00:46,771 INFO org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl - Interrupted while waiting for queue | |
| java.lang.InterruptedException | |
| at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) | |
| at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) | |
| at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) | |
| at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274) | |
| 2017-05-15 14:00:46,771 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : splat35.sto.midasplayer.com:8041 | |
| 2017-05-15 14:00:46,787 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : splat34.sto.midasplayer.com:8041 | |
| 2017-05-15 14:00:46,798 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Stopping ZooKeeperLeaderRetrievalService. | |
| 2017-05-15 14:00:46,798 INFO org.apache.flink.yarn.YarnJobManager - Task manager akka.tcp://flink@splat34.sto.midasplayer.com:53480/user/taskmanager wants to disconnect, because TaskManager akka://flink/user/taskmanager is disassociating. | |
| 2017-05-15 14:00:46,798 INFO org.apache.flink.runtime.instance.InstanceManager - Unregistered task manager splat34.sto.midasplayer.com/172.26.83.103. Number of registered task managers 0. Number of available slots 0. | |
| 2017-05-15 14:00:46,799 INFO org.apache.flink.yarn.YarnJobManager - Stopping JobManager akka.tcp://flink@splat34.sto.midasplayer.com:33785/user/jobmanager. | |
| 2017-05-15 14:00:46,801 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Stopping ZooKeeperLeaderElectionService org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService@5ebb8b4b. | |
| 2017-05-15 14:00:46,804 INFO org.apache.flink.runtime.blob.BlobServer - Stopped BLOB server at 0.0.0.0:41349 | |
| 2017-05-15 14:00:46,807 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. | |
| 2017-05-15 14:00:46,809 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. | |
| 2017-05-15 14:00:46,829 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting shut down. | |
| 2017-05-15 14:00:46,845 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Stopping ZooKeeperLeaderRetrievalService. | |
| 2017-05-15 14:00:46,845 INFO org.apache.flink.yarn.YarnJobManager - Shutdown completed. Stopping JVM. | |
| 2017-05-15 14:00:46,847 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Removing web dashboard root cache directory /tmp/flink-web-cb0ef96f-0a47-412c-8e82-43b4bdb6dc61 | |
| 2017-05-15 14:00:46,848 INFO org.apache.flink.runtime.webmonitor.StackTraceSampleCoordinator - Shutting down stack trace sample coordinator. | |
| 2017-05-15 14:00:46,851 INFO org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting | |
| 2017-05-15 14:00:46,852 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Removing web dashboard jar upload directory /tmp/flink-web-d05454bc-860d-4b6e-a694-2306a5b962ef |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment