Skip to content

Instantly share code, notes, and snippets.

@rfum
Created September 8, 2018 13:10
Show Gist options
  • Save rfum/6891bb390ce031da9e3086a27d1499a5 to your computer and use it in GitHub Desktop.
Save rfum/6891bb390ce031da9e3086a27d1499a5 to your computer and use it in GitHub Desktop.
Flink + Minio Error creating file system
Starting standalonesession as a console application on host flink-job-manager-1.
2018-09-08 11:56:26,025 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2018-09-08 11:56:26,027 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint (Version: 1.6.0, Rev:ff472b4, Date:07.08.2018 @ 13:31:13 UTC)
2018-09-08 11:56:26,027 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: flink
2018-09-08 11:56:26,028 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: <no hadoop dependency found>
2018-09-08 11:56:26,028 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.181-b13
2018-09-08 11:56:26,028 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 981 MiBytes
2018-09-08 11:56:26,028 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /docker-java-home/jre
2018-09-08 11:56:26,028 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - No Hadoop Dependency available
2018-09-08 11:56:26,029 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
2018-09-08 11:56:26,029 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms1024m
2018-09-08 11:56:26,029 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx1024m
2018-09-08 11:56:26,029 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2018-09-08 11:56:26,029 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2018-09-08 11:56:26,029 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments:
2018-09-08 11:56:26,029 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --configDir
2018-09-08 11:56:26,030 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - /opt/flink/conf
2018-09-08 11:56:26,030 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --executionMode
2018-09-08 11:56:26,030 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster
2018-09-08 11:56:26,030 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --host
2018-09-08 11:56:26,030 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster
2018-09-08 11:56:26,030 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: /opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar:::
2018-09-08 11:56:26,030 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2018-09-08 11:56:26,034 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]
2018-09-08 11:56:26,052 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, flink-job-manager-0.flink-job-svc.flink.svc.cluster.local
2018-09-08 11:56:26,053 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m
2018-09-08 11:56:26,054 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m
2018-09-08 11:56:26,054 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 40
2018-09-08 11:56:26,055 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2018-09-08 11:56:26,055 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper
2018-09-08 11:56:26,056 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, custom
2018-09-08 11:56:26,056 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, s3://minio:9000
2018-09-08 11:56:26,057 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181
2018-09-08 11:56:26,058 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081
2018-09-08 11:56:26,060 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: zookeeper.sasl.disable, true
2018-09-08 11:56:26,060 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 6124
2018-09-08 11:56:26,061 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125
2018-09-08 11:56:26,061 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: log.file, /data/flink/job.log
2018-09-08 11:56:26,061 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.access-key, AKIAIOSFODNN7EXAMPLE
2018-09-08 11:56:26,061 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.secret-key, ******
2018-09-08 11:56:26,062 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125
2018-09-08 11:56:26,149 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint.
2018-09-08 11:56:26,150 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem.
2018-09-08 11:56:26,182 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
2018-09-08 11:56:26,189 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install security context.
2018-09-08 11:56:26,197 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2018-09-08 11:56:26,209 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
2018-09-08 11:56:26,210 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services.
2018-09-08 11:56:26,223 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013
2018-09-08 11:56:27,020 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2018-09-08 11:56:27,093 INFO akka.remote.Remoting - Starting remoting
2018-09-08 11:56:27,224 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]
2018-09-08 11:56:27,231 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013
2018-09-08 11:56:28,288 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at s3://minio:9000/custom/blob
2018-09-08 11:56:28,296 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections
2018-09-08 11:56:28,296 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink/custom' as Zookeeper namespace.
2018-09-08 11:56:28,389 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2018-09-08 11:56:28,402 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2018-09-08 11:56:28,402 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=flink-job-manager-1.flink-job-svc.flink.svc.cluster.local
2018-09-08 11:56:28,402 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_181
2018-09-08 11:56:28,402 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2018-09-08 11:56:28,402 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2018-09-08 11:56:28,402 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar:::
2018-09-08 11:56:28,403 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2018-09-08 11:56:28,403 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2018-09-08 11:56:28,403 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2018-09-08 11:56:28,403 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2018-09-08 11:56:28,404 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2018-09-08 11:56:28,404 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-1022-azure
2018-09-08 11:56:28,404 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=flink
2018-09-08 11:56:28,404 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/opt/flink
2018-09-08 11:56:28,404 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/flink
2018-09-08 11:56:28,404 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@5851bd4f
2018-09-08 11:56:28,425 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181
2018-09-08 11:56:28,430 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, initiating session
2018-09-08 11:56:28,442 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, sessionid = 0x165b903c95f0001, negotiated timeout = 40000
2018-09-08 11:56:28,444 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-6408d98a-52ff-4a26-8d82-de0384825640
2018-09-08 11:56:28,449 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2018-09-08 11:56:28,453 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000
2018-09-08 11:56:28,472 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
2018-09-08 11:56:28,478 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/executionGraphStore-0d513fbd-0341-4cd0-b62b-72d21e7a942a, expiration time 3600000, maximum cache size 52428800 bytes.
2018-09-08 11:56:28,511 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-87a55d4d-8256-47eb-8d4a-612093d15f44
2018-09-08 11:56:28,528 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'
2018-09-08 11:56:28,529 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-bd38d844-bc95-490e-b1b2-83325257d8d2/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.
2018-09-08 11:56:28,531 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-bd38d844-bc95-490e-b1b2-83325257d8d2/flink-web-upload for file uploads.
2018-09-08 11:56:28,535 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint.
2018-09-08 11:56:29,047 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - Log file environment variable 'log.file' is not set.
2018-09-08 11:56:29,047 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'Key: 'web.log.path' , default: null (deprecated keys: [jobmanager.web.log.path])'.
2018-09-08 11:56:29,285 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081
2018-09-08 11:56:29,286 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}.
2018-09-08 11:56:29,325 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend listening at http://flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081.
2018-09-08 11:56:29,343 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager .
2018-09-08 11:56:29,399 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
2018-09-08 11:56:29,419 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}.
2018-09-08 11:56:29,421 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2018-09-08 11:56:29,430 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}.
2018-09-08 11:56:29,432 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock.
2018-09-08 12:58:52,708 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:36411] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2018-09-08 12:59:34,016 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/dispatcher was granted leadership with fencing token 62abf6d0-8509-4ea9-9cb5-1a70b37e7eb5
2018-09-08 12:59:34,016 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs.
2018-09-08 12:59:34,021 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081 was granted leadership with leaderSessionID=25d1bd81-5ab0-4f54-9908-241a31905109
2018-09-08 12:59:34,075 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - ResourceManager akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager was granted leadership with fencing token 88ea0b9394bde28a8629de01c7b94903
2018-09-08 12:59:34,077 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager.
2018-09-08 11:58:21,842 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2018-09-08 11:58:21,844 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint (Version: 1.6.0, Rev:ff472b4, Date:07.08.2018 @ 13:31:13 UTC)
2018-09-08 11:58:21,844 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: flink
2018-09-08 11:58:21,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: <no hadoop dependency found>
2018-09-08 11:58:21,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.181-b13
2018-09-08 11:58:21,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 981 MiBytes
2018-09-08 11:58:21,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /docker-java-home/jre
2018-09-08 11:58:21,846 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - No Hadoop Dependency available
2018-09-08 11:58:21,846 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
2018-09-08 11:58:21,846 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms1024m
2018-09-08 11:58:21,846 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx1024m
2018-09-08 11:58:21,846 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2018-09-08 11:58:21,847 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2018-09-08 11:58:21,847 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments:
2018-09-08 11:58:21,847 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --configDir
2018-09-08 11:58:21,847 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - /opt/flink/conf
2018-09-08 11:58:21,847 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --executionMode
2018-09-08 11:58:21,847 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster
2018-09-08 11:58:21,848 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --host
2018-09-08 11:58:21,848 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster
2018-09-08 11:58:21,848 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: /opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar:::
2018-09-08 11:58:21,848 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2018-09-08 11:58:21,850 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]
2018-09-08 11:58:21,872 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, flink-job-manager-0.flink-job-svc.flink.svc.cluster.local
2018-09-08 11:58:21,876 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m
2018-09-08 11:58:21,876 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m
2018-09-08 11:58:21,876 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 40
2018-09-08 11:58:21,876 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2018-09-08 11:58:21,877 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper
2018-09-08 11:58:21,877 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, custom
2018-09-08 11:58:21,878 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, s3://minio:9000
2018-09-08 11:58:21,878 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181
2018-09-08 11:58:21,879 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081
2018-09-08 11:58:21,880 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: zookeeper.sasl.disable, true
2018-09-08 11:58:21,881 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 6124
2018-09-08 11:58:21,881 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125
2018-09-08 11:58:21,881 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: log.file, /data/flink/job.log
2018-09-08 11:58:21,881 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.access-key, AKIAIOSFODNN7EXAMPLE
2018-09-08 11:58:21,882 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.secret-key, ******
2018-09-08 11:58:21,882 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125
2018-09-08 11:58:22,003 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint.
2018-09-08 11:58:22,006 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem.
2018-09-08 11:58:22,034 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
2018-09-08 11:58:22,043 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install security context.
2018-09-08 11:58:22,056 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2018-09-08 11:58:22,070 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
2018-09-08 11:58:22,071 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services.
2018-09-08 11:58:22,086 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:34357
2018-09-08 11:58:22,839 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2018-09-08 11:58:22,932 INFO akka.remote.Remoting - Starting remoting
2018-09-08 11:58:23,091 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:34357]
2018-09-08 11:58:23,098 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:34357
2018-09-08 11:58:24,303 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at s3://minio:9000/custom/blob
2018-09-08 11:58:24,309 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections
2018-09-08 11:58:24,310 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink/custom' as Zookeeper namespace.
2018-09-08 11:58:24,400 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2018-09-08 11:58:24,416 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2018-09-08 11:58:24,417 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=flink-job-manager-3.flink-job-svc.flink.svc.cluster.local
2018-09-08 11:58:24,417 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_181
2018-09-08 11:58:24,417 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2018-09-08 11:58:24,417 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2018-09-08 11:58:24,418 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar:::
2018-09-08 11:58:24,418 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2018-09-08 11:58:24,418 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2018-09-08 11:58:24,418 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2018-09-08 11:58:24,418 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2018-09-08 11:58:24,418 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2018-09-08 11:58:24,419 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-1022-azure
2018-09-08 11:58:24,419 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=flink
2018-09-08 11:58:24,419 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/opt/flink
2018-09-08 11:58:24,419 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/flink
2018-09-08 11:58:24,420 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@5851bd4f
2018-09-08 11:58:24,442 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.0.76:2181
2018-09-08 11:58:24,461 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.0.76:2181, initiating session
2018-09-08 11:58:24,484 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-1c94e424-8f4d-470d-bcb8-c122e8b3e56c
2018-09-08 11:58:24,486 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.0.76:2181, sessionid = 0x365b903c2fe0001, negotiated timeout = 40000
2018-09-08 11:58:24,492 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000
2018-09-08 11:58:24,500 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2018-09-08 11:58:24,520 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
2018-09-08 11:58:24,527 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/executionGraphStore-cae866a9-4109-49c0-9e81-b5be77c48ab9, expiration time 3600000, maximum cache size 52428800 bytes.
2018-09-08 11:58:24,589 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-c62109a6-032d-4541-acde-937375922162
2018-09-08 11:58:24,638 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'
2018-09-08 11:58:24,641 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-c7a37d10-072f-46c8-923b-6401836c8d5d/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.
2018-09-08 11:58:24,652 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-c7a37d10-072f-46c8-923b-6401836c8d5d/flink-web-upload for file uploads.
2018-09-08 11:58:24,658 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint.
2018-09-08 11:58:25,194 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - Log file environment variable 'log.file' is not set.
2018-09-08 11:58:25,195 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'Key: 'web.log.path' , default: null (deprecated keys: [jobmanager.web.log.path])'.
2018-09-08 11:58:25,388 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081
2018-09-08 11:58:25,388 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}.
2018-09-08 11:58:25,431 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend listening at http://flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081.
2018-09-08 11:58:25,453 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager .
2018-09-08 11:58:25,542 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
2018-09-08 11:58:25,571 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}.
2018-09-08 11:58:25,573 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2018-09-08 11:58:25,575 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}.
2018-09-08 11:58:25,577 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock.
2018-09-08 12:58:52,702 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:36411] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2018-09-08 12:59:34,175 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local: Name or service not known]
2018-09-08 12:59:34,248 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local]
2018-09-08 12:59:34,314 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local]
2018-09-08 12:59:34,386 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local]
2018-09-08 12:59:34,454 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local]
2018-09-08 12:59:34,525 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local]
2018-09-08 12:59:34,595 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local]
2018-09-08 12:59:34,665 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local]
2018-09-08 12:59:34,735 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local]
2018-09-08 12:59:34,812 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local]
2018-09-08 12:59:34,893 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local]
Starting taskexecutor as a console application on host flink-task-manager-0.
2018-09-08 12:53:19,496 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - --------------------------------------------------------------------------------
2018-09-08 12:53:19,497 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting TaskManager (Version: 1.6.0, Rev:ff472b4, Date:07.08.2018 @ 13:31:13 UTC)
2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - OS current user: flink
2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Current Hadoop/Kerberos user: <no hadoop dependency found>
2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.181-b13
2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum heap size: 922 MiBytes
2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JAVA_HOME: /docker-java-home/jre
2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - No Hadoop Dependency available
2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM Options:
2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:+UseG1GC
2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xms922M
2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xmx922M
2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:MaxDirectMemorySize=8388607T
2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Program Arguments:
2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - --configDir
2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - /opt/flink/conf
2018-09-08 12:53:19,501 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Classpath: /opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar:::
2018-09-08 12:53:19,501 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - --------------------------------------------------------------------------------
2018-09-08 12:53:19,503 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Registered UNIX signal handlers for [TERM, HUP, INT]
2018-09-08 12:53:19,510 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum number of open file descriptors is 1048576.
2018-09-08 12:53:19,524 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, flink-job-manager-0.flink-job-svc.flink.svc.cluster.local
2018-09-08 12:53:19,524 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2018-09-08 12:53:19,525 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m
2018-09-08 12:53:19,525 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m
2018-09-08 12:53:19,525 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 40
2018-09-08 12:53:19,526 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2018-09-08 12:53:19,526 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper
2018-09-08 12:53:19,526 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, custom
2018-09-08 12:53:19,527 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, s3://minio:9000
2018-09-08 12:53:19,528 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181
2018-09-08 12:53:19,529 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081
2018-09-08 12:53:19,530 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: zookeeper.sasl.disable, true
2018-09-08 12:53:19,531 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: log.file, /data/flink/task.log
2018-09-08 12:53:19,531 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 6124
2018-09-08 12:53:19,532 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125
2018-09-08 12:53:19,532 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.access-key, AKIAIOSFODNN7EXAMPLE
2018-09-08 12:53:19,532 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.secret-key, ******
2018-09-08 12:53:19,533 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125
2018-09-08 12:53:19,556 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
2018-09-08 12:53:19,575 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2018-09-08 12:53:19,603 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
2018-09-08 12:53:21,263 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at s3://minio:9000/custom/blob
2018-09-08 12:53:21,269 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections
2018-09-08 12:53:21,270 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink/custom' as Zookeeper namespace.
2018-09-08 12:53:21,381 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2018-09-08 12:53:21,405 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2018-09-08 12:53:21,405 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=flink-task-manager-0
2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_181
2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar:::
2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-1022-azure
2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=flink
2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/opt/flink
2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/flink
2018-09-08 12:53:21,408 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@2631f68c
2018-09-08 12:53:21,465 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181
2018-09-08 12:53:21,489 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, initiating session
2018-09-08 12:53:21,493 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2018-09-08 12:53:21,506 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, sessionid = 0x165b903c95f0003, negotiated timeout = 40000
2018-09-08 12:53:21,519 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2018-09-08 12:53:21,557 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - Trying to select the network interface and address to use by connecting to the leading JobManager.
2018-09-08 12:53:21,557 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - TaskManager will try to connect for 10000 milliseconds before falling back to heuristics
2018-09-08 12:53:21,633 INFO org.apache.flink.runtime.net.ConnectionUtils - Retrieved new target address flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.164:36411.
2018-09-08 12:53:21,638 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Stopping ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2018-09-08 12:53:21,639 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - TaskManager will use hostname/address 'flink-task-manager-0' (10.244.3.165) for communication.
2018-09-08 12:53:21,643 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Starting AkkaRpcService at flink-task-manager-0:0.
2018-09-08 12:53:22,281 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2018-09-08 12:53:22,371 INFO akka.remote.Remoting - Starting remoting
2018-09-08 12:53:22,616 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@flink-task-manager-0:44689]
2018-09-08 12:53:22,692 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
2018-09-08 12:53:22,699 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Created BLOB cache storage directory /tmp/blobStore-741f05fd-7a5f-4b20-ab11-66098ddd7bb5
2018-09-08 12:53:22,705 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-51498ac0-f068-4522-bfde-dc6047d851dd
2018-09-08 12:53:22,713 INFO org.apache.flink.runtime.io.network.netty.NettyConfig - NettyConfig [server address: flink-task-manager-0/10.244.3.165, server port: 0, ssl enabled: false, memory segment size (bytes): 32768, transport type: NIO, number of server threads: 40 (manual), number of client threads: 40 (manual), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)]
2018-09-08 12:53:22,811 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file directory '/tmp': total 29 GB, usable 14 GB (48.28% usable)
2018-09-08 12:53:23,045 INFO org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 102 MB for network buffer pool (number of memory segments: 3278, bytes per segment: 32768).
2018-09-08 12:53:23,173 INFO org.apache.flink.runtime.query.QueryableStateUtils - Could not load Queryable State Client Proxy. Probable reason: flink-queryable-state-runtime is not in the classpath. To enable Queryable State, please move the flink-queryable-state-runtime jar from the opt to the lib folder.
2018-09-08 12:53:23,174 INFO org.apache.flink.runtime.query.QueryableStateUtils - Could not load Queryable State Server. Probable reason: flink-queryable-state-runtime is not in the classpath. To enable Queryable State, please move the flink-queryable-state-runtime jar from the opt to the lib folder.
2018-09-08 12:53:23,175 INFO org.apache.flink.runtime.io.network.NetworkEnvironment - Starting the network environment and its components.
2018-09-08 12:53:23,249 INFO org.apache.flink.runtime.io.network.netty.NettyClient - Successful initialization (took 73 ms).
2018-09-08 12:53:23,368 INFO org.apache.flink.runtime.io.network.netty.NettyServer - Successful initialization (took 118 ms). Listening on SocketAddress /10.244.3.165:40913.
2018-09-08 12:53:23,369 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Limiting managed memory to 0.7 of the currently free heap space (634 MB), memory will be allocated lazily.
2018-09-08 12:53:23,375 INFO org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager uses directory /tmp/flink-io-908bca06-9e34-4ebc-882f-055d76bf3454 for spill files.
2018-09-08 12:53:23,443 INFO org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration - Messages have a max timeout of 10000 ms
2018-09-08 12:53:23,454 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at akka://flink/user/taskmanager_0 .
2018-09-08 12:53:23,472 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2018-09-08 12:53:23,473 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Start job leader service.
2018-09-08 12:53:23,475 INFO org.apache.flink.runtime.filecache.FileCache - User file cache uses directory /tmp/flink-dist-cache-5e7e5b20-b71a-4e31-9766-da70e668619f
2018-09-08 12:53:23,489 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to ResourceManager akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:36411/user/resourcemanager(a5a439df914aa590bdd27cbd63084651).
2018-09-08 12:53:23,736 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Resolved ResourceManager address, beginning registration
2018-09-08 12:53:23,736 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at ResourceManager attempt 1 (timeout=100ms)
2018-09-08 12:53:23,895 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at ResourceManager attempt 2 (timeout=200ms)
2018-09-08 12:53:23,937 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Successful registration at resource manager akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:36411/user/resourcemanager under registration id d223747355b4842abc73660ffa585ccb.
2018-09-08 12:58:52,701 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:36411] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2018-09-08 12:59:34,076 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Close ResourceManager connection a4355e1fb750ab2dc11bedbaddd35c38.
2018-09-08 12:59:34,102 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:36411] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:36411]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local: Name or service not known]
2018-09-08 12:59:34,122 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to ResourceManager akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager(88ea0b9394bde28a8629de01c7b94903).
2018-09-08 12:59:34,128 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local]
2018-09-08 12:59:34,129 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 12:59:44,155 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 12:59:44,155 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local: Name or service not known]
2018-09-08 12:59:54,186 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local: Name or service not known]
2018-09-08 12:59:54,186 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:00:04,234 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:00:04,234 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local: Name or service not known]
2018-09-08 13:00:14,262 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:00:14,262 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [flink-job-manager-0.flink-job-svc.flink.svc.cluster.local: Name or service not known]
2018-09-08 13:00:24,280 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:00:24,281 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:00:24,282 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:00:34,299 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:00:34,301 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:00:34,301 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:00:44,319 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:00:44,319 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:00:44,320 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:00:54,338 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:00:54,339 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:00:54,340 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:01:04,358 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:01:04,359 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:01:04,359 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:01:14,378 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:01:14,378 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:01:14,380 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:01:24,399 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:01:24,400 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:01:24,402 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:01:34,419 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:01:34,420 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:01:34,421 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:01:44,442 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:01:44,444 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:01:44,445 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:01:54,460 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:01:54,461 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:01:54,462 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:02:04,477 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:02:04,478 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:02:04,478 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:02:14,499 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:02:14,501 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:02:14,501 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:02:24,519 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:02:24,520 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:02:24,520 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:02:34,538 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:02:34,539 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:02:34,540 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:02:44,559 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:02:44,560 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:02:44,561 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:02:54,578 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:02:54,579 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:02:54,579 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:03:04,598 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:03:04,599 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:03:04,601 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:03:14,618 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:03:14,620 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:03:14,620 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:03:24,639 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:03:24,639 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:03:24,640 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:03:34,660 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:03:34,661 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:03:34,661 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:03:44,678 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:03:44,679 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:03:44,679 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
2018-09-08 13:03:54,700 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013
2018-09-08 13:03:54,701 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013]] Caused by: [Connection refused: flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013]
2018-09-08 13:03:54,707 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:44013/user/resourcemanager..
Starting standalonesession as a console application on host flink-job-manager-0.
2018-09-08 11:55:37,829 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2018-09-08 11:55:37,830 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint (Version: 1.6.0, Rev:ff472b4, Date:07.08.2018 @ 13:31:13 UTC)
2018-09-08 11:55:37,830 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: flink
2018-09-08 11:55:37,831 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: <no hadoop dependency found>
2018-09-08 11:55:37,831 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.181-b13
2018-09-08 11:55:37,831 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 981 MiBytes
2018-09-08 11:55:37,831 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /docker-java-home/jre
2018-09-08 11:55:37,832 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - No Hadoop Dependency available
2018-09-08 11:55:37,832 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
2018-09-08 11:55:37,832 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms1024m
2018-09-08 11:55:37,832 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx1024m
2018-09-08 11:55:37,832 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2018-09-08 11:55:37,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2018-09-08 11:55:37,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments:
2018-09-08 11:55:37,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --configDir
2018-09-08 11:55:37,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - /opt/flink/conf
2018-09-08 11:55:37,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --executionMode
2018-09-08 11:55:37,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster
2018-09-08 11:55:37,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --host
2018-09-08 11:55:37,834 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster
2018-09-08 11:55:37,834 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: /opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar:::
2018-09-08 11:55:37,834 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2018-09-08 11:55:37,836 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]
2018-09-08 11:55:37,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, flink-job-manager-0.flink-job-svc.flink.svc.cluster.local
2018-09-08 11:55:37,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m
2018-09-08 11:55:37,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m
2018-09-08 11:55:37,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 40
2018-09-08 11:55:37,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2018-09-08 11:55:37,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper
2018-09-08 11:55:37,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, custom
2018-09-08 11:55:37,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, s3://minio:9000
2018-09-08 11:55:37,857 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181
2018-09-08 11:55:37,858 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081
2018-09-08 11:55:37,859 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: zookeeper.sasl.disable, true
2018-09-08 11:55:37,859 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 6124
2018-09-08 11:55:37,860 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125
2018-09-08 11:55:37,860 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: log.file, /data/flink/job.log
2018-09-08 11:55:37,860 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.access-key, AKIAIOSFODNN7EXAMPLE
2018-09-08 11:55:37,860 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.secret-key, ******
2018-09-08 11:55:37,861 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125
2018-09-08 11:55:37,975 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint.
2018-09-08 11:55:37,975 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem.
2018-09-08 11:55:38,002 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
2018-09-08 11:55:38,013 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install security context.
2018-09-08 11:55:38,022 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2018-09-08 11:55:38,038 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
2018-09-08 11:55:38,038 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services.
2018-09-08 11:55:38,061 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:36411
2018-09-08 11:55:38,873 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2018-09-08 11:55:38,969 INFO akka.remote.Remoting - Starting remoting
2018-09-08 11:55:39,117 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:36411]
2018-09-08 11:55:39,130 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:36411
2018-09-08 11:55:40,597 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at s3://minio:9000/custom/blob
2018-09-08 11:55:40,603 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections
2018-09-08 11:55:40,604 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink/custom' as Zookeeper namespace.
2018-09-08 11:55:40,771 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2018-09-08 11:55:40,800 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2018-09-08 11:55:40,800 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=flink-job-manager-0.flink-job-svc.flink.svc.cluster.local
2018-09-08 11:55:40,800 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_181
2018-09-08 11:55:40,800 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2018-09-08 11:55:40,800 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2018-09-08 11:55:40,801 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar:::
2018-09-08 11:55:40,801 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2018-09-08 11:55:40,801 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2018-09-08 11:55:40,803 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2018-09-08 11:55:40,803 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2018-09-08 11:55:40,804 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2018-09-08 11:55:40,804 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-1022-azure
2018-09-08 11:55:40,804 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=flink
2018-09-08 11:55:40,804 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/opt/flink
2018-09-08 11:55:40,804 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/flink
2018-09-08 11:55:40,805 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@29caf222
2018-09-08 11:55:40,875 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181
2018-09-08 11:55:40,889 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, initiating session
2018-09-08 11:55:40,890 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-2a386297-ce71-42d9-a968-8c66117cea0a
2018-09-08 11:55:40,903 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, sessionid = 0x165b903c95f0000, negotiated timeout = 40000
2018-09-08 11:55:40,913 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2018-09-08 11:55:40,917 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000
2018-09-08 11:55:40,949 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
2018-09-08 11:55:40,955 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/executionGraphStore-4ef201cf-7028-4a71-9041-83de535b3147, expiration time 3600000, maximum cache size 52428800 bytes.
2018-09-08 11:55:41,001 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-b9468b26-b300-4c2e-ac54-dc3acee1e610
2018-09-08 11:55:41,020 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'
2018-09-08 11:55:41,032 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-92eb8b33-ac65-4c24-a42b-5c64fb4da5a2/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.
2018-09-08 11:55:41,034 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-92eb8b33-ac65-4c24-a42b-5c64fb4da5a2/flink-web-upload for file uploads.
2018-09-08 11:55:41,040 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint.
2018-09-08 11:55:41,461 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - Log file environment variable 'log.file' is not set.
2018-09-08 11:55:41,462 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'Key: 'web.log.path' , default: null (deprecated keys: [jobmanager.web.log.path])'.
2018-09-08 11:55:41,689 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081
2018-09-08 11:55:41,689 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}.
2018-09-08 11:55:41,741 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend listening at http://flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081.
2018-09-08 11:55:41,770 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager .
2018-09-08 11:55:41,783 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081 was granted leadership with leaderSessionID=0d817ea6-85a8-4af3-ab02-0cae2b2490dd
2018-09-08 11:55:41,836 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
2018-09-08 11:55:41,873 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}.
2018-09-08 11:55:41,880 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2018-09-08 11:55:41,888 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}.
2018-09-08 11:55:41,890 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock.
2018-09-08 11:55:41,902 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - ResourceManager akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:36411/user/resourcemanager was granted leadership with fencing token a5a439df914aa590bdd27cbd63084651
2018-09-08 11:55:41,903 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager.
2018-09-08 11:55:41,908 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:36411/user/dispatcher was granted leadership with fencing token bc6db51a-f6d7-4b5e-a201-8b5fccc37b4a
2018-09-08 11:55:41,908 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs.
2018-09-08 12:53:23,924 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Replacing old registration of TaskExecutor 5737ffe2496c7432429419864fbb76a0.
2018-09-08 12:53:23,924 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Unregister TaskManager a0ddec1a885cee59adf6504db55f73fe from the SlotManager.
2018-09-08 12:53:23,925 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - The target with resource ID 5737ffe2496c7432429419864fbb76a0 is already been monitored.
2018-09-08 12:53:23,978 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Registering TaskManager 5737ffe2496c7432429419864fbb76a0 under d223747355b4842abc73660ffa585ccb at the SlotManager.
2018-09-08 12:54:10,259 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Registering TaskManager b0f6f22da68a2e2e997b219f10fe5828 under f1f06580d44e554ad6375d5aff799c26 at the SlotManager.
2018-09-08 12:55:00,634 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Registering TaskManager 286dbce39f7cfd3178979012f5015873 under 9814c8f1bc76eb99c57c796884621491 at the SlotManager.
Starting taskexecutor as a console application on host flink-task-manager-0.
2018-09-08 12:53:19,496 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - --------------------------------------------------------------------------------
2018-09-08 12:53:19,497 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting TaskManager (Version: 1.6.0, Rev:ff472b4, Date:07.08.2018 @ 13:31:13 UTC)
2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - OS current user: flink
2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Current Hadoop/Kerberos user: <no hadoop dependency found>
2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.181-b13
2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum heap size: 922 MiBytes
2018-09-08 12:53:19,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JAVA_HOME: /docker-java-home/jre
2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - No Hadoop Dependency available
2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM Options:
2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:+UseG1GC
2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xms922M
2018-09-08 12:53:19,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xmx922M
2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:MaxDirectMemorySize=8388607T
2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Program Arguments:
2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - --configDir
2018-09-08 12:53:19,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - /opt/flink/conf
2018-09-08 12:53:19,501 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Classpath: /opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar:::
2018-09-08 12:53:19,501 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - --------------------------------------------------------------------------------
2018-09-08 12:53:19,503 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Registered UNIX signal handlers for [TERM, HUP, INT]
2018-09-08 12:53:19,510 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum number of open file descriptors is 1048576.
2018-09-08 12:53:19,524 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, flink-job-manager-0.flink-job-svc.flink.svc.cluster.local
2018-09-08 12:53:19,524 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2018-09-08 12:53:19,525 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m
2018-09-08 12:53:19,525 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m
2018-09-08 12:53:19,525 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 40
2018-09-08 12:53:19,526 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2018-09-08 12:53:19,526 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper
2018-09-08 12:53:19,526 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, custom
2018-09-08 12:53:19,527 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, s3://minio:9000
2018-09-08 12:53:19,528 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181
2018-09-08 12:53:19,529 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081
2018-09-08 12:53:19,530 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: zookeeper.sasl.disable, true
2018-09-08 12:53:19,531 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: log.file, /data/flink/task.log
2018-09-08 12:53:19,531 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 6124
2018-09-08 12:53:19,532 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125
2018-09-08 12:53:19,532 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.access-key, AKIAIOSFODNN7EXAMPLE
2018-09-08 12:53:19,532 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.secret-key, ******
2018-09-08 12:53:19,533 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125
2018-09-08 12:53:19,556 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
2018-09-08 12:53:19,575 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2018-09-08 12:53:19,603 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
2018-09-08 12:53:21,263 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at s3://minio:9000/custom/blob
2018-09-08 12:53:21,269 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections
2018-09-08 12:53:21,270 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink/custom' as Zookeeper namespace.
2018-09-08 12:53:21,381 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2018-09-08 12:53:21,405 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2018-09-08 12:53:21,405 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=flink-task-manager-0
2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_181
2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt/flink/lib/flink-python_2.11-1.6.0.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.6.0.jar:/opt/flink/lib/flink-s3-fs-presto-1.6.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.6.0.jar:::
2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2018-09-08 12:53:21,406 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-1022-azure
2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=flink
2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/opt/flink
2018-09-08 12:53:21,407 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/flink
2018-09-08 12:53:21,408 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-1.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181,flink-zk-zookeeper-2.flink-zk-zookeeper-headless.flink.svc.cluster.local:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@2631f68c
2018-09-08 12:53:21,465 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181
2018-09-08 12:53:21,489 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, initiating session
2018-09-08 12:53:21,493 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2018-09-08 12:53:21,506 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server flink-zk-zookeeper-0.flink-zk-zookeeper-headless.flink.svc.cluster.local/10.244.2.89:2181, sessionid = 0x165b903c95f0003, negotiated timeout = 40000
2018-09-08 12:53:21,519 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2018-09-08 12:53:21,557 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - Trying to select the network interface and address to use by connecting to the leading JobManager.
2018-09-08 12:53:21,557 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - TaskManager will try to connect for 10000 milliseconds before falling back to heuristics
2018-09-08 12:53:21,633 INFO org.apache.flink.runtime.net.ConnectionUtils - Retrieved new target address flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.164:36411.
2018-09-08 12:53:21,638 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Stopping ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2018-09-08 12:53:21,639 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - TaskManager will use hostname/address 'flink-task-manager-0' (10.244.3.165) for communication.
2018-09-08 12:53:21,643 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Starting AkkaRpcService at flink-task-manager-0:0.
2018-09-08 12:53:22,281 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2018-09-08 12:53:22,371 INFO akka.remote.Remoting - Starting remoting
2018-09-08 12:53:22,616 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@flink-task-manager-0:44689]
2018-09-08 12:53:22,692 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
2018-09-08 12:53:22,699 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Created BLOB cache storage directory /tmp/blobStore-741f05fd-7a5f-4b20-ab11-66098ddd7bb5
2018-09-08 12:53:22,705 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-51498ac0-f068-4522-bfde-dc6047d851dd
2018-09-08 12:53:22,713 INFO org.apache.flink.runtime.io.network.netty.NettyConfig - NettyConfig [server address: flink-task-manager-0/10.244.3.165, server port: 0, ssl enabled: false, memory segment size (bytes): 32768, transport type: NIO, number of server threads: 40 (manual), number of client threads: 40 (manual), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)]
2018-09-08 12:53:22,811 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file directory '/tmp': total 29 GB, usable 14 GB (48.28% usable)
2018-09-08 12:53:23,045 INFO org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 102 MB for network buffer pool (number of memory segments: 3278, bytes per segment: 32768).
2018-09-08 12:53:23,173 INFO org.apache.flink.runtime.query.QueryableStateUtils - Could not load Queryable State Client Proxy. Probable reason: flink-queryable-state-runtime is not in the classpath. To enable Queryable State, please move the flink-queryable-state-runtime jar from the opt to the lib folder.
2018-09-08 12:53:23,174 INFO org.apache.flink.runtime.query.QueryableStateUtils - Could not load Queryable State Server. Probable reason: flink-queryable-state-runtime is not in the classpath. To enable Queryable State, please move the flink-queryable-state-runtime jar from the opt to the lib folder.
2018-09-08 12:53:23,175 INFO org.apache.flink.runtime.io.network.NetworkEnvironment - Starting the network environment and its components.
2018-09-08 12:53:23,249 INFO org.apache.flink.runtime.io.network.netty.NettyClient - Successful initialization (took 73 ms).
2018-09-08 12:53:23,368 INFO org.apache.flink.runtime.io.network.netty.NettyServer - Successful initialization (took 118 ms). Listening on SocketAddress /10.244.3.165:40913.
2018-09-08 12:53:23,369 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Limiting managed memory to 0.7 of the currently free heap space (634 MB), memory will be allocated lazily.
2018-09-08 12:53:23,375 INFO org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager uses directory /tmp/flink-io-908bca06-9e34-4ebc-882f-055d76bf3454 for spill files.
2018-09-08 12:53:23,443 INFO org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration - Messages have a max timeout of 10000 ms
2018-09-08 12:53:23,454 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at akka://flink/user/taskmanager_0 .
2018-09-08 12:53:23,472 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2018-09-08 12:53:23,473 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Start job leader service.
2018-09-08 12:53:23,475 INFO org.apache.flink.runtime.filecache.FileCache - User file cache uses directory /tmp/flink-dist-cache-5e7e5b20-b71a-4e31-9766-da70e668619f
2018-09-08 12:53:23,489 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to ResourceManager akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:36411/user/resourcemanager(a5a439df914aa590bdd27cbd63084651).
2018-09-08 12:53:23,736 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Resolved ResourceManager address, beginning registration
2018-09-08 12:53:23,736 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at ResourceManager attempt 1 (timeout=100ms)
2018-09-08 12:53:23,895 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at ResourceManager attempt 2 (timeout=200ms)
2018-09-08 12:53:23,937 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Successful registration at resource manager akka.tcp://flink@flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:36411/user/resourcemanager under registration id d223747355b4842abc73660ffa585ccb.
FROM flink:latest
WORKDIR /opt/flink
COPY flink-conf.yaml conf
COPY masters conf
RUN cp opt/flink-s3-fs-presto-1.6.0.jar lib && cp opt/flink-s3-fs-hadoop-1.6.0.jar lib
flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081
flink-job-manager-1.flink-job-svc.flink.svc.cluster.local:8081
flink-job-manager-2.flink-job-svc.flink.svc.cluster.local:8081
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: flink-job-manager
namespace: flink
spec:
selector:
matchLabels:
app: flink
role: job # has to match .spec.template.metadata.labels
serviceName: "flink-job-svc" # "flink-job-svc-headless"
replicas: 4 # by default is 1
template:
metadata:
labels:
app: flink # has to match .spec.selector.matchLabels
role: job
spec:
terminationGracePeriodSeconds: 10
restartPolicy: Always
imagePullSecrets:
- name: customn-cr-secret
containers:
- name: flink
image: customcr.azurecr.io/flink-job:v1
args:
- jobmanager
ports:
- containerPort: 6123
name: rpc
- containerPort: 6124
name: blob
- containerPort: 6125
name: query
- containerPort: 8081
name: ui
volumeMounts:
- name: checkpoint
mountPath: /data/flink/checkpoints
readinessProbe:
exec:
command:
- ls
- /data/flink/checkpoints
initialDelaySeconds: 5
periodSeconds: 5
imagePullPolicy: Always
volumeClaimTemplates:
- metadata:
name: checkpoint
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "custom-ssd-gen"
resources:
requests:
storage: 20Gi
NAME READY STATUS RESTARTS AGE
flink-job-manager-0 1/1 Running 0 1h
flink-job-manager-1 1/1 Running 0 1h
flink-job-manager-2 1/1 Running 0 1h
flink-job-manager-3 1/1 Running 0 1h
flink-task-manager-0 1/1 Running 0 5m
flink-task-manager-1 1/1 Running 0 5m
flink-task-manager-2 1/1 Running 0 4m
flink-zk-zookeeper-0 1/1 Running 1 2d
flink-zk-zookeeper-1 1/1 Running 1 2d
flink-zk-zookeeper-2 1/1 Running 1 2d
minio-55c776944c-jl9x5 1/1 Running 1 2d
apiVersion: v1
kind: Service
metadata:
name: flink-job-svc
namespace: flink
spec:
clusterIP: "None"
ports:
- name: rpc
port: 6123
- name: blob
port: 6124
- name: query
port: 6125
- name: ui
port: 8081
selector:
app: flink
role: job
FROM flink:latest
WORKDIR /opt/flink
COPY flink-conf.yaml conf
COPY masters conf
RUN cp opt/flink-s3-fs-presto-1.6.0.jar lib && cp opt/flink-s3-fs-hadoop-1.6.0.jar lib
flink-job-manager-0.flink-job-svc.flink.svc.cluster.local:8081
flink-job-manager-1.flink-job-svc.flink.svc.cluster.local:8081
flink-job-manager-2.flink-job-svc.flink.svc.cluster.local:8081
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: flink-task-manager
namespace: flink
spec:
serviceName: ""
selector:
matchLabels:
app: flink
role: task
updateStrategy:
type: RollingUpdate
replicas: 3
template:
metadata:
labels:
app: flink
role: task
spec:
terminationGracePeriodSeconds: 10
imagePullSecrets:
- name: custom-cr-secret
containers:
- name: flink
image: customcr.azurecr.io/flink-task:v1
args:
- taskmanager
ports:
- containerPort: 6121
name: data
- containerPort: 6122
name: rpc
- containerPort: 6125
name: query
volumeMounts:
- name: checkpoint
mountPath: /data/flink/checkpoints
readinessProbe:
exec:
command:
- ls
- /data/flink/checkpoints
initialDelaySeconds: 5
periodSeconds: 5
imagePullPolicy: Always
volumeClaimTemplates:
- metadata:
name: checkpoint
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "custom-ssd-gen"
resources:
requests:
storage: 20Gi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment