Skip to content

Instantly share code, notes, and snippets.

@Xeli

Xeli/gist:0321031655e47006f00d38fc4bc08e16 Secret

Last active July 17, 2019 17:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Xeli/0321031655e47006f00d38fc4bc08e16 to your computer and use it in GitHub Desktop.
Save Xeli/0321031655e47006f00d38fc4bc08e16 to your computer and use it in GitHub Desktop.
Download ZIP
Flink jobmanager zk ha exception
Raw
gistfile1.txt
Starting standalonesession as a console application on host flink-jobmanager-0.
2019-07-17 16:44:29,636 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2019-07-17 16:44:29,637 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint (Version: 1.8.0, Rev:4caec0d, Date:03.04.2019 @ 13:25:54 PDT)
2019-07-17 16:44:29,638 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: flink
2019-07-17 16:44:29,936 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-07-17 16:44:30,226 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: flink
2019-07-17 16:44:30,226 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.212-b01
2019-07-17 16:44:30,226 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 1342 MiBytes
2019-07-17 16:44:30,227 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /docker-java-home/jre
2019-07-17 16:44:30,232 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Hadoop version: 2.4.1
2019-07-17 16:44:30,232 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
2019-07-17 16:44:30,232 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms1400m
2019-07-17 16:44:30,232 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx1400m
2019-07-17 16:44:30,232 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2019-07-17 16:44:30,233 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2019-07-17 16:44:30,233 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments:
2019-07-17 16:44:30,233 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --configDir
2019-07-17 16:44:30,233 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - /opt/flink/conf
2019-07-17 16:44:30,233 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --executionMode
2019-07-17 16:44:30,233 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster
2019-07-17 16:44:30,233 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: /opt/flink/lib/flink-docker-1.0-SNAPSHOT.jar:/opt/flink/lib/flink-shaded-hadoop2-uber-2.4.1-1.8.0-avro-1.9.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/flink/lib/flink-dist_2.12-1.8.0.jar:/opt/flink/lib::
2019-07-17 16:44:30,233 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2019-07-17 16:44:30,235 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]
2019-07-17 16:44:30,272 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper
2019-07-17 16:44:30,273 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, hunch-zookeeper-0.hunch-zookeeper-hs.hunch.svc.cluster.local:2181,hunch-zookeeper-1.hunch-zookeeper-hs.hunch.svc.cluster.local:2181,hunch-zookeeper-2.hunch-zookeeper-hs.hunch.svc.cluster.local:2181
2019-07-17 16:44:30,273 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, gs://some-project-f97-flink-state/recovery
2019-07-17 16:44:30,273 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, hunch
2019-07-17 16:44:30,274 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.jobmanager.port, 50077
2019-07-17 16:44:30,274 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, flink-jobmanager
2019-07-17 16:44:30,274 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1400m
2019-07-17 16:44:30,275 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 5000m
2019-07-17 16:44:30,275 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2019-07-17 16:44:30,275 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 2
2019-07-17 16:44:30,275 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.ask.timeout, 60s
2019-07-17 16:44:30,275 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 6124
2019-07-17 16:44:30,276 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend, rocksdb
2019-07-17 16:44:30,276 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.checkpoints.dir, gs://some-project-flink-state/checkpoints
2019-07-17 16:44:30,276 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.savepoints.dir, gs://some-project-flink-state/savepoints
2019-07-17 16:44:30,276 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.incremental, true
2019-07-17 16:44:30,276 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.local-recovery, true
2019-07-17 16:44:30,276 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.rocksdb.metrics.total-sst-files-size, true
2019-07-17 16:44:30,276 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.rocksdb.metrics.cur-size-all-mem-tables, true
2019-07-17 16:44:30,277 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.rocksdb.metrics.estimate-live-data-size, true
2019-07-17 16:44:30,277 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.rocksdb.metrics.estimate-num-keys, true
2019-07-17 16:44:30,277 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.rocksdb.metrics.num-running-compactions, true
2019-07-17 16:44:30,277 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.rocksdb.metrics.num-running-flushes, true
2019-07-17 16:44:30,277 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.rocksdb.timer-service.factory, ROCKSDB
2019-07-17 16:44:30,278 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporter.prom.class, org.apache.flink.metrics.prometheus.PrometheusReporter
2019-07-17 16:44:30,278 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.scope.jm, <host>.jobmanager
2019-07-17 16:44:30,278 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.scope.tm, <host>.taskmanager.<tm_id>
2019-07-17 16:44:30,278 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.scope.jm.job, <host>.jobmanager.<job_name>
2019-07-17 16:44:30,278 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.scope.tm.job, <host>.taskmanager.<tm_id>.<job_name>
2019-07-17 16:44:30,278 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.scope.task, <host>.taskmanager.<tm_id>.<job_name>
2019-07-17 16:44:30,279 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: zookeeper.sasl.disable, true
2019-07-17 16:44:30,279 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.java.opts.taskmanager, -XX:+UseG1GC -XX:MaxDirectMemorySize=2G -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2
2019-07-17 16:44:30,310 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint.
2019-07-17 16:44:30,310 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem.
2019-07-17 16:44:30,326 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install security context.
2019-07-17 16:44:30,406 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to flink (auth:SIMPLE)
2019-07-17 16:44:30,425 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services.
2019-07-17 16:44:31,026 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at flink-jobmanager:50077
2019-07-17 16:44:31,910 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2019-07-17 16:44:32,075 INFO akka.remote.Remoting - Starting remoting
2019-07-17 16:44:32,277 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@flink-jobmanager:50077]
2019-07-17 16:44:32,309 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://flink@flink-jobmanager:50077
2019-07-17 16:44:32,364 INFO com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase - GHFS version: hadoop2-1.9.5
2019-07-17 16:44:32,946 WARN com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase - No working directory configured, using default: 'gs://some-project-flink-state/'
2019-07-17 16:44:32,950 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at gs://some-project-flink-state/recovery/hunch/blob
2019-07-17 16:44:33,878 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections
2019-07-17 16:44:33,879 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink/hunch' as Zookeeper namespace.
2019-07-17 16:44:33,974 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2019-07-17 16:44:33,983 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2019-07-17 16:44:33,983 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=flink-jobmanager-0.flink-jobmanager-service.hunch.svc.cluster.local
2019-07-17 16:44:33,984 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_212
2019-07-17 16:44:33,984 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2019-07-17 16:44:33,984 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2019-07-17 16:44:33,984 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt/flink/lib/flink-docker-1.0-SNAPSHOT.jar:/opt/flink/lib/flink-shaded-hadoop2-uber-2.4.1-1.8.0-avro-1.9.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/flink/lib/flink-dist_2.12-1.8.0.jar:/opt/flink/lib::
2019-07-17 16:44:33,984 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2019-07-17 16:44:33,984 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2019-07-17 16:44:33,984 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2019-07-17 16:44:33,984 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2019-07-17 16:44:33,984 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2019-07-17 16:44:33,984 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.14.127+
2019-07-17 16:44:33,984 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=flink
2019-07-17 16:44:33,984 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/opt/flink
2019-07-17 16:44:33,984 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/flink
2019-07-17 16:44:33,985 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=hunch-zookeeper-0.hunch-zookeeper-hs.hunch.svc.cluster.local:2181,hunch-zookeeper-1.hunch-zookeeper-hs.hunch.svc.cluster.local:2181,hunch-zookeeper-2.hunch-zookeeper-hs.hunch.svc.cluster.local:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@2b491fee
2019-07-17 16:44:34,012 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server hunch-zookeeper-2.hunch-zookeeper-hs.hunch.svc.cluster.local/10.26.91.31:2181
2019-07-17 16:44:34,018 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to hunch-zookeeper-2.hunch-zookeeper-hs.hunch.svc.cluster.local/10.26.91.31:2181, initiating session
2019-07-17 16:44:34,023 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-403831f8-bf62-443e-b8c6-f4f8c0a7f620
2019-07-17 16:44:34,027 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000
2019-07-17 16:44:34,067 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server hunch-zookeeper-2.hunch-zookeeper-hs.hunch.svc.cluster.local/10.26.91.31:2181, sessionid = 0x36c00aec3150018, negotiated timeout = 40000
2019-07-17 16:44:34,078 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2019-07-17 16:44:34,222 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - Configuring prom with {class=org.apache.flink.metrics.prometheus.PrometheusReporter}.
2019-07-17 16:44:34,231 INFO org.apache.flink.metrics.prometheus.PrometheusReporter - Started PrometheusReporter HTTP server on port 9249.
2019-07-17 16:44:34,232 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - Reporting metrics for reporter prom of type org.apache.flink.metrics.prometheus.PrometheusReporter.
2019-07-17 16:44:34,234 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at flink-jobmanager:0
2019-07-17 16:44:34,301 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2019-07-17 16:44:34,314 INFO akka.remote.Remoting - Starting remoting
2019-07-17 16:44:34,367 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink-metrics@flink-jobmanager:34813]
2019-07-17 16:44:34,369 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink-metrics@flink-jobmanager:34813
2019-07-17 16:44:34,376 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/executionGraphStore-bbb2f569-e893-4d23-9f93-2007ffc0d7f5, expiration time 3600000, maximum cache size 52428800 bytes.
2019-07-17 16:44:34,404 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-2212a77b-8825-46c5-8714-7d4586cc9fcd
2019-07-17 16:44:34,445 INFO org.apache.flink.configuration.Configuration - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address'
2019-07-17 16:44:34,447 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-714d22d5-e45a-40d3-a1fa-44c1d13c9883/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.
2019-07-17 16:44:34,448 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-714d22d5-e45a-40d3-a1fa-44c1d13c9883/flink-web-upload for file uploads.
2019-07-17 16:44:34,450 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint.
2019-07-17 16:44:34,795 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - Log file environment variable 'log.file' is not set.
2019-07-17 16:44:34,796 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'Key: 'web.log.path' , default: null (fallback keys: [{key=jobmanager.web.log.path, isDeprecated=true}])'.
2019-07-17 16:44:35,034 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at flink-jobmanager:8081
2019-07-17 16:44:35,035 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}.
2019-07-17 16:44:35,095 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend listening at http://flink-jobmanager:8081.
2019-07-17 16:44:35,121 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://flink-jobmanager:8081 was granted leadership with leaderSessionID=f49f6aa8-4e38-4ee0-8099-981ec8a5859f
2019-07-17 16:44:35,282 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager .
2019-07-17 16:44:35,333 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
2019-07-17 16:44:35,380 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2019-07-17 16:44:35,382 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock.
2019-07-17 16:44:35,382 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}.
2019-07-17 16:44:35,389 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}.
2019-07-17 16:44:35,399 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - ResourceManager akka.tcp://flink@flink-jobmanager:50077/user/resourcemanager was granted leadership with fencing token 9b2a2c0968b1e44514868f130c334ab3
2019-07-17 16:44:35,401 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager.
2019-07-17 16:44:35,403 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://flink@flink-jobmanager:50077/user/dispatcher was granted leadership with fencing token ea40dba3-806b-429c-84f0-5cc48454d928
2019-07-17 16:44:35,410 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs.
2019-07-17 16:44:36,184 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Recovered SubmittedJobGraph(e6ad857af7f09b56594e95fe273e9eff).
2019-07-17 16:44:36,574 INFO org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - Recovered SubmittedJobGraph(1dccee15d84e1d2cededf89758ac2482).
2019-07-17 16:44:36,872 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Registering TaskManager with ResourceID eb3c2fca8ddfb4fef7d0074e02118ef0 (akka.tcp://flink@1.2.3.4:46183/user/taskmanager_0) at ResourceManager
**i've removed the logs with registering taskmanagers because they show ips...**
2019-07-17 16:44:39,500 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 .
2019-07-17 16:44:39,513 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job Job (1dccee15d84e1d2cededf89758ac2482).
2019-07-17 16:44:39,525 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, delayBetweenRestartAttempts=5000) for Job (1dccee15d84e1d2cededf89758ac2482).
2019-07-17 16:44:39,538 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Fatal error occurred in the cluster entrypoint.
org.apache.flink.runtime.dispatcher.DispatcherException: Failed to take leadership with session id ea40dba3-806b-429c-84f0-5cc48454d928.
at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$null$31(Dispatcher.java:888)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at org.apache.flink.runtime.concurrent.FutureUtils$WaitingConjunctFuture.handleCompletedFuture(FutureUtils.java:635)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:739)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:392)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:185)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:147)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
at akka.actor.Actor.aroundReceive(Actor.scala:502)
at akka.actor.Actor.aroundReceive$(Actor.scala:500)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.RuntimeException: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager
at org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:36)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
... 4 more
Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager
at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:152)
at org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:76)
at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:351)
at org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:34)
... 7 more
Caused by: java.lang.Exception: Cannot set up the user code libraries: Item not found: some-project-flink-state/recovery/hunch/blob/job_e6ad857af7f09b56594e95fe273e9eff/blob_p-486d68fa98fa05665f341d79302c40566b81034e-306d493f5aa810b5f4f7d8d63f5b18b5. If you enabled STRICT generation consistency, it is possible that the live version is still available but the intended generation is deleted.
at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:131)
... 10 more
Caused by: java.io.FileNotFoundException: Item not found: some-project-flink-state/recovery/hunch/blob/job_e6ad857af7f09b56594e95fe273e9eff/blob_p-486d68fa98fa05665f341d79302c40566b81034e-306d493f5aa810b5f4f7d8d63f5b18b5. If you enabled STRICT generation consistency, it is possible that the live version is still available but the intended generation is deleted.
at com.google.cloud.hadoop.gcsio.GoogleCloudStorageExceptions.getFileNotFoundException(GoogleCloudStorageExceptions.java:41)
at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.open(GoogleCloudStorageImpl.java:659)
at com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.open(GoogleCloudStorageFileSystem.java:323)
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream.<init>(GoogleHadoopFSInputStream.java:136)
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.open(GoogleHadoopFileSystemBase.java:1102)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:120)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:37)
at org.apache.flink.runtime.blob.FileSystemBlobStore.get(FileSystemBlobStore.java:102)
at org.apache.flink.runtime.blob.FileSystemBlobStore.get(FileSystemBlobStore.java:84)
at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:493)
at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:444)
at org.apache.flink.runtime.blob.BlobServer.getFile(BlobServer.java:417)
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:120)
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerJob(BlobLibraryCacheManager.java:91)
at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:128)
... 10 more
2019-07-17 16:44:39,570 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart
2019-07-17 16:44:39,571 INFO org.apache.flink.runtime.blob.TransientBlobCache - Shutting down BLOB cache
2019-07-17 16:44:39,586 INFO org.apache.flink.runtime.blob.BlobServer - Stopped BLOB server at 0.0.0.0:6124
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment