Skip to content

Instantly share code, notes, and snippets.

@Ethanlm

Ethanlm/taskmanager.log

Last active February 14, 2019 15:45
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Ethanlm/e6f1b29d27d26813f5f8f40cd2c12643 to your computer and use it in GitHub Desktop.
Save Ethanlm/e6f1b29d27d26813f5f8f40cd2c12643 to your computer and use it in GitHub Desktop.
Download ZIP
TaskManager gets confused when the JobManager restarts (because the JM rpc port changed from 35213 to 34561)
Raw
taskmanager.log
2019-02-14 14:56:54,050 DEBUG org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Leader node has changed.
2019-02-14 14:56:54,050 DEBUG org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - New leader information: Leader=akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:34561/user/jobmanager, session ID=2d2d1c35-0e4b-40d5-9eb6-121704ee93e3.
2019-02-14 14:56:54,059 INFO org.apache.flink.runtime.taskmanager.TaskManager - Trying to register at JobManager akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:34561/user/jobmanager (attempt 1, timeout: 500 milliseconds)
2019-02-14 14:56:54,157 DEBUG org.apache.flink.shaded.akka.org.jboss.netty.handler.ssl.SslHandler - [id: 0x77ac93ae, /10.215.68.243:46796 => openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:34561] HANDSHAKEN: TLS_RSA_WITH_AES_128_CBC_SHA
2019-02-14 14:56:54,276 INFO org.apache.flink.runtime.taskmanager.TaskManager - Successful registration at JobManager (akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:34561/user/jobmanager), starting network stack and library cache.
2019-02-14 14:56:54,276 INFO org.apache.flink.runtime.taskmanager.TaskManager - Determined BLOB server address to be openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:50100. Starting BLOB cache.
2019-02-14 14:56:54,278 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Created BLOB cache storage directory /home/y/var/flink/blobstorage/blobStore-927b523f-f3ff-4ccc-83a0-362e09a3b858
2019-02-14 14:56:54,279 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /home/y/var/flink/blobstorage/blobStore-8492465e-0e94-4792-a346-66e6da299f7a
2019-02-14 14:56:54,572 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - TaskManager was triggered to register at JobManager, but is already registered
2019-02-14 14:56:56,359 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:56:56,360 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:56:56,361 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:56:59,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:57:01,385 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:57:01,388 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:57:01,388 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:57:04,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:57:06,410 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:57:06,410 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:57:06,411 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:57:07,393 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x168a04c6e360059 after 2ms
2019-02-14 14:57:09,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:57:11,431 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:57:11,431 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:57:11,432 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:57:14,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:57:16,451 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:57:16,451 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:57:16,453 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:57:19,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:57:20,740 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x168a04c6e360059 after 2ms
2019-02-14 14:57:21,480 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:57:21,482 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:57:21,483 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:57:24,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:57:26,500 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:57:26,501 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:57:26,502 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:57:29,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:57:31,520 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:57:31,522 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:57:31,523 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:57:34,087 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x168a04c6e360059 after 1ms
2019-02-14 14:57:34,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:57:36,548 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:57:36,550 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:57:36,550 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:57:39,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:57:41,570 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:57:41,570 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:57:41,571 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:57:44,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:57:46,591 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:57:46,592 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:57:46,592 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:57:47,434 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x168a04c6e360059 after 1ms
2019-02-14 14:57:49,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:57:51,619 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:57:51,620 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:57:51,620 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:57:54,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:57:56,638 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:57:56,639 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:57:56,639 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:57:59,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:58:00,781 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x168a04c6e360059 after 1ms
2019-02-14 14:58:01,674 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:58:01,677 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:58:01,686 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:58:04,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:58:06,698 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:58:06,700 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:58:06,700 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:58:09,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:58:11,720 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:58:11,720 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:58:11,720 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:58:14,128 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x168a04c6e360059 after 1ms
2019-02-14 14:58:14,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:58:16,739 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:58:16,740 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:58:16,740 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:58:19,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:58:21,761 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:58:21,762 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:58:21,762 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:58:24,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:58:26,788 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:58:26,790 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:58:26,790 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:58:27,462 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x168a04c6e360059 after 2ms
2019-02-14 14:58:29,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:58:31,810 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:58:31,811 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:58:31,812 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:58:34,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:58:36,830 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:58:36,831 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:58:36,832 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:58:39,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:58:40,797 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x168a04c6e360059 after 1ms
2019-02-14 14:58:41,852 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:58:41,853 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:58:41,853 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:58:44,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:58:46,880 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:58:46,881 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:58:46,881 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:58:49,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:58:51,901 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:58:51,901 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:58:51,901 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:58:54,144 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x168a04c6e360059 after 2ms
2019-02-14 14:58:54,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:58:56,918 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:58:56,919 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:58:56,919 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:58:59,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:59:01,951 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:59:01,956 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:59:01,956 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:59:04,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:59:06,969 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:59:06,970 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:59:06,970 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:59:07,481 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x168a04c6e360059 after 2ms
2019-02-14 14:59:09,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:59:11,989 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:59:11,989 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:59:11,990 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:59:14,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:59:17,010 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:59:17,010 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:59:17,010 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:59:19,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:59:20,822 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x168a04c6e360059 after 1ms
2019-02-14 14:59:22,029 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:59:22,031 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:59:22,031 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:59:24,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:59:27,049 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:59:27,049 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:59:27,051 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:59:29,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:59:32,070 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:59:32,070 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:59:32,070 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:59:34,168 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x168a04c6e360059 after 1ms
2019-02-14 14:59:34,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:59:37,089 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:59:37,089 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:59:37,090 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:59:39,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:59:42,110 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:59:42,110 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:59:42,110 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:59:44,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:59:47,131 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:59:47,131 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:59:47,131 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:59:47,515 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x168a04c6e360059 after 2ms
2019-02-14 14:59:49,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:59:52,150 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213
2019-02-14 14:59:52,150 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213]] Caused by: [Connection refused: openstorm10blue-n1.blue.ygrid.yahoo.com/10.215.68.98:35213]
2019-02-14 14:59:52,150 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - The association error event's root cause is not of type InvalidAssociationException.
2019-02-14 14:59:54,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 14:59:57,168 ERROR akka.remote.Remoting - Association to [akka.ssl.tcp://flink@openstorm10blue-n1.blue.ygrid.yahoo.com:35213] with UID [778171738] irrecoverably failed. Quarantining address.
java.util.concurrent.TimeoutException: Delivery of system messages timed out and they were dropped.
at akka.remote.ReliableDeliverySupervisor$$anonfun$gated$1.applyOrElse(Endpoint.scala:346)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at akka.remote.ReliableDeliverySupervisor.aroundReceive(Endpoint.scala:203)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2019-02-14 14:59:59,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 15:00:00,861 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x168a04c6e360059 after 2ms
2019-02-14 15:00:04,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 15:00:09,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 15:00:14,208 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x168a04c6e360059 after 2ms
2019-02-14 15:00:14,294 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 15:00:19,292 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
2019-02-14 15:00:24,293 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment