Skip to content

Instantly share code, notes, and snippets.

@miko-code

miko-code/log

Created May 31, 2018 16:08
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save miko-code/28d57b32cb9c4f1aa96fa9873e10e53c to your computer and use it in GitHub Desktop.
Save miko-code/28d57b32cb9c4f1aa96fa9873e10e53c to your computer and use it in GitHub Desktop.
Download ZIP
JM
Raw
log
2018-05-31 11:56:22,953 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2018-05-31 11:56:22,954 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @ 14:54:44 UTC)
2018-05-31 11:56:22,954 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: beam1
2018-05-31 11:56:23,382 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-05-31 11:56:23,448 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: beam1
2018-05-31 11:56:23,448 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b11
2018-05-31 11:56:23,448 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 11776 MiBytes
2018-05-31 11:56:23,448 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /usr/lib/jvm/java-1.8.0-openjdk-amd64
2018-05-31 11:56:23,452 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Hadoop version: 2.8.3
2018-05-31 11:56:23,452 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
2018-05-31 11:56:23,452 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms12288m
2018-05-31 11:56:23,452 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx12288m
2018-05-31 11:56:23,452 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog.file=/home/beam1/flink-1.5.0/log/flink-beam1-standalonesession-7-beam1.log
2018-05-31 11:56:23,452 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:/home/beam1/flink-1.5.0/conf/log4j.properties
2018-05-31 11:56:23,452 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:/home/beam1/flink-1.5.0/conf/logback.xml
2018-05-31 11:56:23,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments:
2018-05-31 11:56:23,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --configDir
2018-05-31 11:56:23,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - /home/beam1/flink-1.5.0/conf
2018-05-31 11:56:23,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --executionMode
2018-05-31 11:56:23,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster
2018-05-31 11:56:23,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --host
2018-05-31 11:56:23,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - 10.1.70.108
2018-05-31 11:56:23,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --webui-port
2018-05-31 11:56:23,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - 8082
2018-05-31 11:56:23,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: /home/beam1/flink-1.5.0/lib/flink-hadoop-compatibility_2.11-1.5.0.jar:/home/beam1/flink-1.5.0/lib/flink-python_2.11-1.5.0.jar:/home/beam1/flink-1.5.0/lib/flink-shaded-hadoop2-uber-1.5.0.jar:/home/beam1/flink-1.5.0/lib/log4j-1.2.17.jar:/home/beam1/flink-1.5.0/lib/slf4j-log4j12-1.7.7.jar:/home/beam1/flink-1.5.0/lib/flink-dist_2.11-1.5.0.jar:::
2018-05-31 11:56:23,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2018-05-31 11:56:23,454 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]
2018-05-31 11:56:23,460 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, 10.1.70.108
2018-05-31 11:56:23,460 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2018-05-31 11:56:23,460 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 12288
2018-05-31 11:56:23,460 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 12228
2018-05-31 11:56:23,460 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 8
2018-05-31 11:56:23,460 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2018-05-31 11:56:23,461 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper
2018-05-31 11:56:23,461 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, hdfs://10.1.70.31:9000/flink1.5/ha
2018-05-31 11:56:23,461 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, localhost:2181,10.1.70.116:2181
2018-05-31 11:56:23,461 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.checkpoints.dir, hdfs://10.1.70.31:9000/flink1.5/flink-checkpoints
2018-05-31 11:56:23,461 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.savepoints.dir, hdfs://10.1.70.31:9000/flink1.5/flink-savepoint
2018-05-31 11:56:23,461 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081
2018-05-31 11:56:23,478 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint.
2018-05-31 11:56:23,478 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem.
2018-05-31 11:56:23,488 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install security context.
2018-05-31 11:56:23,530 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to beam1 (auth:SIMPLE)
2018-05-31 11:56:23,545 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services.
2018-05-31 11:56:23,553 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at 10.1.70.108:46044
2018-05-31 11:56:24,015 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2018-05-31 11:56:24,063 INFO akka.remote.Remoting - Starting remoting
2018-05-31 11:56:24,253 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@10.1.70.108:46044]
2018-05-31 11:56:24,259 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink@10.1.70.108:46044
2018-05-31 11:56:24,718 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at hdfs://10.1.70.31:9000/flink1.5/ha//default/blob
2018-05-31 11:56:24,771 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections
2018-05-31 11:56:24,771 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink/default' as Zookeeper namespace.
2018-05-31 11:56:24,822 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2018-05-31 11:56:24,828 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2018-05-31 11:56:24,828 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=beam1.tlv.gov.il
2018-05-31 11:56:24,828 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_171
2018-05-31 11:56:24,829 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2018-05-31 11:56:24,829 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2018-05-31 11:56:24,829 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/home/beam1/flink-1.5.0/lib/flink-hadoop-compatibility_2.11-1.5.0.jar:/home/beam1/flink-1.5.0/lib/flink-python_2.11-1.5.0.jar:/home/beam1/flink-1.5.0/lib/flink-shaded-hadoop2-uber-1.5.0.jar:/home/beam1/flink-1.5.0/lib/log4j-1.2.17.jar:/home/beam1/flink-1.5.0/lib/slf4j-log4j12-1.7.7.jar:/home/beam1/flink-1.5.0/lib/flink-dist_2.11-1.5.0.jar:::
2018-05-31 11:56:24,829 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2018-05-31 11:56:24,829 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2018-05-31 11:56:24,829 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2018-05-31 11:56:24,829 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2018-05-31 11:56:24,829 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2018-05-31 11:56:24,829 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.4.0-87-generic
2018-05-31 11:56:24,829 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=beam1
2018-05-31 11:56:24,829 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/beam1
2018-05-31 11:56:24,829 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/beam1
2018-05-31 11:56:24,830 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2181,10.1.70.116:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@5432050b
2018-05-31 11:56:24,840 WARN org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-6147667929831012942.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2018-05-31 11:56:24,842 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181
2018-05-31 11:56:24,842 ERROR org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - Authentication failed
2018-05-31 11:56:24,842 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-05-31 11:56:24,848 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-67fdcccd-bb4e-4a3b-8aef-ddb930b759fc
2018-05-31 11:56:24,850 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x163b1cdee640005, negotiated timeout = 40000
2018-05-31 11:56:24,850 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:40083 - max concurrent requests: 50 - max backlog: 1000
2018-05-31 11:56:24,851 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2018-05-31 11:56:24,864 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
2018-05-31 11:56:24,868 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/executionGraphStore-c85fc293-ec14-462e-bc84-8276e509e9f6, expiration time 3600000, maximum cache size 52428800 bytes.
2018-05-31 11:56:24,891 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-c3b229f4-eb60-468d-bcad-3c7ffc83a1e8
2018-05-31 11:56:24,904 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'
2018-05-31 11:56:24,905 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-2c37266f-98be-4583-9564-fa120029a57c/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.
2018-05-31 11:56:24,906 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-2c37266f-98be-4583-9564-fa120029a57c/flink-web-upload for file uploads.
2018-05-31 11:56:24,909 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint.
2018-05-31 11:56:25,217 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /home/beam1/flink-1.5.0/log/flink-beam1-standalonesession-7-beam1.log
2018-05-31 11:56:25,217 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /home/beam1/flink-1.5.0/log/flink-beam1-standalonesession-7-beam1.out
2018-05-31 11:56:25,289 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at 10.1.70.108:8082
2018-05-31 11:56:25,289 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}.
2018-05-31 11:56:25,307 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend listening at http://10.1.70.108:8082.
2018-05-31 11:56:25,318 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager .
2018-05-31 11:56:25,391 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
2018-05-31 11:56:25,404 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}.
2018-05-31 11:56:25,405 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2018-05-31 11:56:25,407 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}.
2018-05-31 11:56:25,407 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock.
2018-05-31 11:56:25,487 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.108:44541
2018-05-31 11:56:25,491 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.108:44541] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.108:44541]] Caused by: [Connection refused: /10.1.70.108:44541]
2018-05-31 11:56:25,566 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.108:44541
2018-05-31 11:56:25,566 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.108:44541] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.108:44541]] Caused by: [Connection refused: /10.1.70.108:44541]
2018-05-31 11:56:25,607 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Recovered SubmittedJobGraph(5c545fc3f43d69325fb9966b8dd4c8f3, null).
2018-05-31 11:56:25,620 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Recovered SubmittedJobGraph(3f3b45be54a2e5ea987ced475ff9b8bc, null).
2018-05-31 11:56:25,632 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Recovered SubmittedJobGraph(56cd7f998a2ca3df192a3213f3f3bf4c, null).
2018-05-31 11:56:25,636 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.108:44541
2018-05-31 11:56:25,637 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.108:44541] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.108:44541]] Caused by: [Connection refused: /10.1.70.108:44541]
2018-05-31 11:56:25,648 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Recovered SubmittedJobGraph(a4f39270cc6efcf5698d4db2f66d40a7, null).
2018-05-31 11:56:25,705 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.108:44541
2018-05-31 11:56:25,705 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.108:44541] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.108:44541]] Caused by: [Connection refused: /10.1.70.108:44541]
2018-05-31 11:56:25,775 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.108:44541
2018-05-31 11:56:25,776 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.108:44541] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.108:44541]] Caused by: [Connection refused: /10.1.70.108:44541]
2018-05-31 11:56:25,845 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.108:44541
2018-05-31 11:56:25,845 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.108:44541] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.108:44541]] Caused by: [Connection refused: /10.1.70.108:44541]
2018-05-31 11:56:25,915 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.108:44541
2018-05-31 11:56:25,915 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.108:44541] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.108:44541]] Caused by: [Connection refused: /10.1.70.108:44541]
2018-05-31 11:56:25,985 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.108:44541
2018-05-31 11:56:25,985 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.108:44541] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.108:44541]] Caused by: [Connection refused: /10.1.70.108:44541]
2018-05-31 11:56:26,055 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.108:44541
2018-05-31 11:56:26,056 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.108:44541] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.108:44541]] Caused by: [Connection refused: /10.1.70.108:44541]
2018-05-31 11:56:26,125 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.108:44541
2018-05-31 11:56:26,125 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.108:44541] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.108:44541]] Caused by: [Connection refused: /10.1.70.108:44541]
2018-05-31 11:56:26,195 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.108:44541
2018-05-31 11:56:26,196 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.108:44541] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.108:44541]] Caused by: [Connection refused: /10.1.70.108:44541]
2018-05-31 11:57:04,649 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://flink@10.1.70.108:46044/user/dispatcher was granted leadership with fencing token afff7ab6cee26937aa9e977965e14e64
2018-05-31 11:57:04,649 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs.
2018-05-31 11:57:04,652 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://10.1.70.108:8082 was granted leadership with leaderSessionID=d01b6f88-9aa4-4071-addb-5c429c41c25d
2018-05-31 11:57:04,661 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - ResourceManager akka.tcp://flink@10.1.70.108:46044/user/resourcemanager was granted leadership with fencing token a2c50433568c421ecec6b410ffc741d6
2018-05-31 11:57:04,661 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager.
2018-05-31 11:57:04,698 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Recovered SubmittedJobGraph(3f3b45be54a2e5ea987ced475ff9b8bc, null).
2018-05-31 11:57:04,727 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Recovered SubmittedJobGraph(5c545fc3f43d69325fb9966b8dd4c8f3, null).
2018-05-31 11:57:04,752 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Recovered SubmittedJobGraph(56cd7f998a2ca3df192a3213f3f3bf4c, null).
2018-05-31 11:57:04,771 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Recovered SubmittedJobGraph(a4f39270cc6efcf5698d4db2f66d40a7, null).
2018-05-31 11:57:04,847 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Registering TaskManager b482cdd634d6189317994d258c28ce1d under 40653cddbef80f3b572a53a5494668fa at the SlotManager.
2018-05-31 11:57:04,857 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Registering TaskManager 463061f9e9f15ffe42d6895ca8d73cdd under 6846863026e2e2c5723e141243b3efee at the SlotManager.
2018-05-31 11:57:04,858 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Registering TaskManager 23a1e1c027db75819d3776e57e01067d under b59ddbd56b78e6cc1caa1cf353ba4f5a at the SlotManager.
2018-05-31 11:57:05,453 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 .
2018-05-31 11:57:05,462 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job esb raw logs to elastic (3f3b45be54a2e5ea987ced475ff9b8bc).
2018-05-31 11:57:05,468 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=5, delayBetweenRestartAttempts=10000) for esb raw logs to elastic (3f3b45be54a2e5ea987ced475ff9b8bc).
2018-05-31 11:57:05,473 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.slotpool.SlotPool at akka://flink/user/2f4cd5dd-da15-4777-800d-8876b525a65c .
2018-05-31 11:57:05,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart
2018-05-31 11:57:05,512 INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job esb raw logs to elastic (3f3b45be54a2e5ea987ced475ff9b8bc).
2018-05-31 11:57:05,512 INFO org.apache.flink.runtime.jobmaster.JobMaster - Successfully ran initialization on master in 0 ms.
2018-05-31 11:57:05,523 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Initialized in '/checkpoints/3f3b45be54a2e5ea987ced475ff9b8bc'.
2018-05-31 11:57:05,536 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using application-defined state backend: File State Backend (checkpoints: 'hdfs://10.1.70.31:9000/flink/esb_raw/checkpoints', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1)
2018-05-31 11:57:05,537 INFO org.apache.flink.runtime.jobmaster.JobMaster - Configuring application-defined state backend with job/cluster config
2018-05-31 11:57:05,554 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper.
2018-05-31 11:57:05,558 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 0 checkpoints in ZooKeeper.
2018-05-31 11:57:05,558 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to fetch 0 checkpoints from storage.
2018-05-31 11:57:05,560 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/3f3b45be54a2e5ea987ced475ff9b8bc/job_manager_lock'}.
2018-05-31 11:57:05,566 INFO org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager runner for job esb raw logs to elastic (3f3b45be54a2e5ea987ced475ff9b8bc) was granted leadership with session id 809109e1-83e2-40a0-b0ba-17ac0cd6356e at akka.tcp://flink@10.1.70.108:46044/user/jobmanager_0.
2018-05-31 11:57:05,568 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Fatal error occurred in the cluster entrypoint.
org.apache.flink.util.FlinkException: Failed to recover job 5c545fc3f43d69325fb9966b8dd4c8f3.
at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$grantLeadership$22(Dispatcher.java:758)
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:332)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:158)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:84)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager
at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:169)
at org.apache.flink.runtime.dispatcher.Dispatcher$DefaultJobManagerRunnerFactory.createJobManagerRunner(Dispatcher.java:885)
at org.apache.flink.runtime.dispatcher.Dispatcher.createJobManagerRunner(Dispatcher.java:287)
at org.apache.flink.runtime.dispatcher.Dispatcher.runJob(Dispatcher.java:277)
at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$grantLeadership$22(Dispatcher.java:754)
... 20 more
Caused by: java.lang.Exception: Cannot set up the user code libraries: File does not exist: /flink1.5/ha/default/blob/job_5c545fc3f43d69325fb9966b8dd4c8f3/blob_p-5d9f3be555d3b05f90b5e148235d25730eb65b3d-ae486e221962f7b96e36da18fe1c57ca
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:72)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:62)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:150)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1821)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:709)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:381)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:503)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:868)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:814)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1886)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2603)
at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:132)
... 24 more
Caused by: java.io.FileNotFoundException: File does not exist: /flink1.5/ha/default/blob/job_5c545fc3f43d69325fb9966b8dd4c8f3/blob_p-5d9f3be555d3b05f90b5e148235d25730eb65b3d-ae486e221962f7b96e36da18fe1c57ca
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:72)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:62)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:150)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1821)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:709)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:381)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:503)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:868)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:814)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1886)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2603)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:845)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:832)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:821)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:325)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:285)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:270)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1132)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:325)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:322)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:322)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:787)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:119)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:36)
at org.apache.flink.runtime.blob.FileSystemBlobStore.get(FileSystemBlobStore.java:102)
at org.apache.flink.runtime.blob.FileSystemBlobStore.get(FileSystemBlobStore.java:84)
at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:506)
at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:457)
at org.apache.flink.runtime.blob.BlobServer.getFile(BlobServer.java:430)
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:120)
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerJob(BlobLibraryCacheManager.java:91)
at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:129)
... 24 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /flink1.5/ha/default/blob/job_5c545fc3f43d69325fb9966b8dd4c8f3/blob_p-5d9f3be555d3b05f90b5e148235d25730eb65b3d-ae486e221962f7b96e36da18fe1c57ca
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:72)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:62)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:150)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1821)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:709)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:381)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:503)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:868)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:814)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1886)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2603)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
at org.apache.hadoop.ipc.Client.call(Client.java:1345)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy10.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:259)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
at com.sun.proxy.$Proxy11.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:843)
... 45 more
2018-05-31 11:57:05,573 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2018-05-31 11:57:05,574 INFO org.apache.flink.runtime.blob.TransientBlobCache - Shutting down BLOB cache
2018-05-31 11:57:05,573 INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting execution of job esb raw logs to elastic (3f3b45be54a2e5ea987ced475ff9b8bc)
2018-05-31 11:57:05,575 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job esb raw logs to elastic (3f3b45be54a2e5ea987ced475ff9b8bc) switched from state CREATED to RUNNING.
2018-05-31 11:57:05,579 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Sink: Unnamed (1/1) (6823799dbe88c10dd00dd6bd1e71ac2a) switched from CREATED to SCHEDULED.
2018-05-31 11:57:05,588 INFO org.apache.flink.runtime.jobmaster.JobMaster - Connecting to ResourceManager akka.tcp://flink@10.1.70.108:46044/user/resourcemanager(a2c50433568c421ecec6b410ffc741d6)
2018-05-31 11:57:05,591 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPool - Cannot serve slot request, no ResourceManager connected. Adding as pending request 6cc70bdfd880317642f5de42ad417e37
2018-05-31 11:57:05,592 INFO org.apache.flink.runtime.jobmaster.JobMaster - Resolved ResourceManager address, beginning registration
2018-05-31 11:57:05,593 INFO org.apache.flink.runtime.jobmaster.JobMaster - Registration at ResourceManager attempt 1 (timeout=100ms)
2018-05-31 11:57:05,594 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/3f3b45be54a2e5ea987ced475ff9b8bc/job_manager_lock.
2018-05-31 11:57:05,594 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Registering job manager b0ba17ac0cd6356e809109e183e240a0@akka.tcp://flink@10.1.70.108:46044/user/jobmanager_0 for job 3f3b45be54a2e5ea987ced475ff9b8bc.
2018-05-31 11:57:05,598 INFO org.apache.flink.runtime.blob.BlobServer - Stopped BLOB server at 0.0.0.0:40083
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment