Skip to content

Instantly share code, notes, and snippets.

@debraj-manna

debraj-manna/gist:a38addc37a322cb242fc66fab1f9fee7 Secret

Last active February 24, 2021 16:43
Show Gist options
  • Save debraj-manna/a38addc37a322cb242fc66fab1f9fee7 to your computer and use it in GitHub Desktop.
Save debraj-manna/a38addc37a322cb242fc66fab1f9fee7 to your computer and use it in GitHub Desktop.
Download ZIP
Flink Debug Logs for Single Node Yarn Cluster
Raw
gistfile1.txt
ubuntu@vrni-platform:~/build-target/flink$ ./bin/flink run-application -t yarn-application ./examples/streaming/TopSpeedWindowing.jar
Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/ubuntu/build-target/flink/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2021-02-24 16:19:35,352 DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory [] - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
2021-02-24 16:19:35,365 DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory [] - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
2021-02-24 16:19:35,366 DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory [] - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
2021-02-24 16:19:35,367 DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory [] - field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since startup])
2021-02-24 16:19:35,367 DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory [] - field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since last successful login])
2021-02-24 16:19:35,370 DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl [] - UgiMetrics, User and group related metrics
2021-02-24 16:19:35,497 DEBUG org.apache.hadoop.security.SecurityUtil [] - Setting hadoop.security.token.service.use_ip to true
2021-02-24 16:19:35,509 DEBUG org.apache.hadoop.util.Shell [] - Failed to detect a valid hadoop home directory
java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
at org.apache.hadoop.util.Shell.checkHadoopHomeInner(Shell.java:469) ~[hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:440) ~[hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:517) [hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:78) [hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1651) [hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.security.SecurityUtil.setConfigurationInternal(SecurityUtil.java:102) [hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:86) [hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:326) [hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:314) [hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1973) [hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:743) [hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:693) [hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:604) [hadoop-common-3.0.0.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_272]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_272]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_272]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_272]
at org.apache.flink.runtime.util.EnvironmentInformation.getHadoopUser(EnvironmentInformation.java:214) [flink-dist_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.runtime.util.EnvironmentInformation.logEnvironmentInfo(EnvironmentInformation.java:431) [flink-dist_2.12-1.12.1.jar:1.12.1]
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1118) [flink-dist_2.12-1.12.1.jar:1.12.1]
2021-02-24 16:19:35,533 DEBUG org.apache.hadoop.util.Shell [] - setsid exited with exit code 0
2021-02-24 16:19:35,559 DEBUG org.apache.hadoop.security.Groups [] - Creating new Groups object
2021-02-24 16:19:35,571 DEBUG org.apache.hadoop.util.PerformanceAdvisory [] - Falling back to shell based
2021-02-24 16:19:35,576 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback [] - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
2021-02-24 16:19:35,704 DEBUG org.apache.hadoop.security.Groups [] - Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
2021-02-24 16:19:35,711 DEBUG org.apache.hadoop.security.UserGroupInformation [] - hadoop login
2021-02-24 16:19:35,712 DEBUG org.apache.hadoop.security.UserGroupInformation [] - hadoop login commit
2021-02-24 16:19:35,717 DEBUG org.apache.hadoop.security.UserGroupInformation [] - using local user:UnixPrincipal: ubuntu
2021-02-24 16:19:35,717 DEBUG org.apache.hadoop.security.UserGroupInformation [] - Using user: "UnixPrincipal: ubuntu" with name ubuntu
2021-02-24 16:19:35,718 DEBUG org.apache.hadoop.security.UserGroupInformation [] - User entry: "ubuntu"
2021-02-24 16:19:35,718 DEBUG org.apache.hadoop.security.UserGroupInformation [] - UGI loginUser:ubuntu (auth:SIMPLE)
2021-02-24 16:19:36,008 DEBUG org.apache.hadoop.security.UserGroupInformation [] - hadoop login
2021-02-24 16:19:36,009 DEBUG org.apache.hadoop.security.UserGroupInformation [] - hadoop login commit
2021-02-24 16:19:36,009 DEBUG org.apache.hadoop.security.UserGroupInformation [] - using local user:UnixPrincipal: ubuntu
2021-02-24 16:19:36,009 DEBUG org.apache.hadoop.security.UserGroupInformation [] - Using user: "UnixPrincipal: ubuntu" with name ubuntu
2021-02-24 16:19:36,009 DEBUG org.apache.hadoop.security.UserGroupInformation [] - User entry: "ubuntu"
2021-02-24 16:19:36,013 DEBUG org.apache.hadoop.security.UserGroupInformation [] - UGI loginUser:ubuntu (auth:SIMPLE)
2021-02-24 16:19:36,039 DEBUG org.apache.hadoop.security.UserGroupInformation [] - PrivilegedAction as:ubuntu (auth:SIMPLE) from:org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
2021-02-24 16:19:36,068 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/home/ubuntu/build-target/flink/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2021-02-24 16:19:36,092 DEBUG org.apache.hadoop.service.AbstractService [] - Service: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state INITED
2021-02-24 16:19:36,163 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at /0.0.0.0:8032
2021-02-24 16:19:36,165 DEBUG org.apache.hadoop.security.UserGroupInformation [] - PrivilegedAction as:ubuntu (auth:SIMPLE) from:org.apache.hadoop.yarn.client.RMProxy.getProxy(RMProxy.java:147)
2021-02-24 16:19:36,167 DEBUG org.apache.hadoop.yarn.ipc.YarnRPC [] - Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
2021-02-24 16:19:36,169 DEBUG org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC [] - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ApplicationClientProtocol
2021-02-24 16:19:36,258 DEBUG org.apache.hadoop.ipc.Server [] - rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc.ProtobufRpcEngine$RpcProtobufRequest, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@386f0da3
2021-02-24 16:19:36,268 DEBUG org.apache.hadoop.ipc.Client [] - ipc.client.bind.wildcard.addr set to true. Will bind client sockets to wildcard address.
2021-02-24 16:19:36,270 DEBUG org.apache.hadoop.ipc.Client [] - getting client out of cache: org.apache.hadoop.ipc.Client@7ca20101
2021-02-24 16:19:36,638 DEBUG org.apache.hadoop.service.AbstractService [] - Service org.apache.hadoop.yarn.client.api.impl.YarnClientImpl is started
2021-02-24 16:19:36,652 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2021-02-24 16:19:36,813 DEBUG org.apache.hadoop.ipc.Client [] - The ping interval is 60000 ms.
2021-02-24 16:19:36,819 DEBUG org.apache.hadoop.ipc.Client [] - Connecting to /0.0.0.0:8032
2021-02-24 16:19:36,845 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu: starting, having connections 1
2021-02-24 16:19:36,848 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu sending #0 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getClusterNodes
2021-02-24 16:19:36,859 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu got value #0
2021-02-24 16:19:36,860 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine [] - Call: getClusterNodes took 94ms
2021-02-24 16:19:36,934 INFO org.apache.hadoop.conf.Configuration [] - resource-types.xml not found
2021-02-24 16:19:36,936 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Unable to find 'resource-types.xml'.
2021-02-24 16:19:36,947 DEBUG org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE
2021-02-24 16:19:36,948 DEBUG org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Adding resource type - name = vcores, units = , type = COUNTABLE
2021-02-24 16:19:36,948 DEBUG org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Mandatory Resource 'yarn.resource-types.memory-mb.minimum-allocation' is not configured in resource-types config file. Setting allocation specified using 'yarn.scheduler.minimum-allocation-mb'
2021-02-24 16:19:36,948 DEBUG org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Mandatory Resource 'yarn.resource-types.memory-mb.maximum-allocation' is not configured in resource-types config file. Setting allocation specified using 'yarn.scheduler.maximum-allocation-mb'
2021-02-24 16:19:36,948 DEBUG org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Mandatory Resource 'yarn.resource-types.vcores.minimum-allocation' is not configured in resource-types config file. Setting allocation specified using 'yarn.scheduler.minimum-allocation-vcores'
2021-02-24 16:19:36,948 DEBUG org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Mandatory Resource 'yarn.resource-types.vcores.maximum-allocation' is not configured in resource-types config file. Setting allocation specified using 'yarn.scheduler.maximum-allocation-vcores'
2021-02-24 16:19:36,961 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu sending #1 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getQueueInfo
2021-02-24 16:19:36,965 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu got value #1
2021-02-24 16:19:36,966 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine [] - Call: getQueueInfo took 6ms
2021-02-24 16:19:36,982 DEBUG org.apache.flink.yarn.YarnClusterDescriptor [] - The YARN cluster does not have any queues configured
2021-02-24 16:19:37,004 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu sending #2 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getNewApplication
2021-02-24 16:19:37,006 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu got value #2
2021-02-24 16:19:37,007 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine [] - Call: getNewApplication took 3ms
2021-02-24 16:19:37,012 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu sending #3 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getClusterNodes
2021-02-24 16:19:37,014 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu got value #3
2021-02-24 16:19:37,014 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine [] - Call: getClusterNodes took 2ms
2021-02-24 16:19:37,015 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - The configured JobManager memory is 1600 MB. YARN will allocate 2048 MB to make up an integer multiple of its minimum allocation memory (512 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 448 MB may not be used by Flink.
2021-02-24 16:19:37,016 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - The configured TaskManager memory is 1728 MB. YARN will allocate 2048 MB to make up an integer multiple of its minimum allocation memory (512 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 320 MB may not be used by Flink.
2021-02-24 16:19:37,016 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster specification: ClusterSpecification{masterMemoryMB=1600, taskManagerMemoryMB=1728, slotsPerTaskManager=1}
2021-02-24 16:19:37,054 DEBUG org.apache.hadoop.fs.FileSystem [] - Loading filesystems
2021-02-24 16:19:37,068 DEBUG org.apache.hadoop.fs.FileSystem [] - file:// = class org.apache.hadoop.fs.LocalFileSystem from /usr/lib/hadoop/hadoop-common-3.0.0.jar
2021-02-24 16:19:37,074 DEBUG org.apache.hadoop.fs.FileSystem [] - viewfs:// = class org.apache.hadoop.fs.viewfs.ViewFileSystem from /usr/lib/hadoop/hadoop-common-3.0.0.jar
2021-02-24 16:19:37,080 DEBUG org.apache.hadoop.fs.FileSystem [] - ftp:// = class org.apache.hadoop.fs.ftp.FTPFileSystem from /usr/lib/hadoop/hadoop-common-3.0.0.jar
2021-02-24 16:19:37,085 DEBUG org.apache.hadoop.fs.FileSystem [] - har:// = class org.apache.hadoop.fs.HarFileSystem from /usr/lib/hadoop/hadoop-common-3.0.0.jar
2021-02-24 16:19:37,088 DEBUG org.apache.hadoop.fs.FileSystem [] - http:// = class org.apache.hadoop.fs.http.HttpFileSystem from /usr/lib/hadoop/hadoop-common-3.0.0.jar
2021-02-24 16:19:37,089 DEBUG org.apache.hadoop.fs.FileSystem [] - https:// = class org.apache.hadoop.fs.http.HttpsFileSystem from /usr/lib/hadoop/hadoop-common-3.0.0.jar
2021-02-24 16:19:37,094 DEBUG org.apache.hadoop.fs.FileSystem [] - s3n:// = class org.apache.hadoop.fs.s3native.NativeS3FileSystem from /usr/lib/hadoop/hadoop-aws-3.0.0.jar
2021-02-24 16:19:37,104 DEBUG org.apache.hadoop.fs.FileSystem [] - hdfs:// = class org.apache.hadoop.hdfs.DistributedFileSystem from /usr/lib/hadoop-hdfs/hadoop-hdfs-client-3.0.0.jar
2021-02-24 16:19:37,118 DEBUG org.apache.hadoop.fs.FileSystem [] - webhdfs:// = class org.apache.hadoop.hdfs.web.WebHdfsFileSystem from /usr/lib/hadoop-hdfs/hadoop-hdfs-client-3.0.0.jar
2021-02-24 16:19:37,120 DEBUG org.apache.hadoop.fs.FileSystem [] - swebhdfs:// = class org.apache.hadoop.hdfs.web.SWebHdfsFileSystem from /usr/lib/hadoop-hdfs/hadoop-hdfs-client-3.0.0.jar
2021-02-24 16:19:37,121 DEBUG org.apache.hadoop.fs.FileSystem [] - Looking for FS supporting file
2021-02-24 16:19:37,121 DEBUG org.apache.hadoop.fs.FileSystem [] - looking for configuration option fs.file.impl
2021-02-24 16:19:37,121 DEBUG org.apache.hadoop.fs.FileSystem [] - Looking in service filesystems for implementation class
2021-02-24 16:19:37,121 DEBUG org.apache.hadoop.fs.FileSystem [] - FS for file is class org.apache.hadoop.fs.LocalFileSystem
2021-02-24 16:19:37,153 WARN org.apache.flink.yarn.YarnClusterDescriptor [] - The file system scheme is 'file'. This indicates that the specified Hadoop configuration path is wrong and the system is using the default Hadoop configuration values.The Flink YARN client needs to store its files in a distributed file system
2021-02-24 16:19:37,204 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/lib/log4j-slf4j-impl-2.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/lib/log4j-slf4j-impl-2.12.1.jar with replication factor 1
2021-02-24 16:19:37,232 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/lib/log4j-slf4j-impl-2.12.1.jar
2021-02-24 16:19:37,270 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/lib/log4j-api-2.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/lib/log4j-api-2.12.1.jar with replication factor 1
2021-02-24 16:19:37,279 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/lib/log4j-api-2.12.1.jar
2021-02-24 16:19:37,280 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/lib/log4j-1.2-api-2.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/lib/log4j-1.2-api-2.12.1.jar with replication factor 1
2021-02-24 16:19:37,287 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/lib/log4j-1.2-api-2.12.1.jar
2021-02-24 16:19:37,289 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/lib/flink-table-blink_2.12-1.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/lib/flink-table-blink_2.12-1.12.1.jar with replication factor 1
2021-02-24 16:19:37,427 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/lib/flink-table-blink_2.12-1.12.1.jar
2021-02-24 16:19:37,428 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/lib/flink-json-1.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/lib/flink-json-1.12.1.jar with replication factor 1
2021-02-24 16:19:37,435 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/lib/flink-json-1.12.1.jar
2021-02-24 16:19:37,437 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/lib/flink-table_2.12-1.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/lib/flink-table_2.12-1.12.1.jar with replication factor 1
2021-02-24 16:19:37,562 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/lib/flink-table_2.12-1.12.1.jar
2021-02-24 16:19:37,564 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/lib/flink-shaded-zookeeper-3.4.14.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/lib/flink-shaded-zookeeper-3.4.14.jar with replication factor 1
2021-02-24 16:19:37,598 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/lib/flink-shaded-zookeeper-3.4.14.jar
2021-02-24 16:19:37,600 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/lib/log4j-core-2.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/lib/log4j-core-2.12.1.jar with replication factor 1
2021-02-24 16:19:37,611 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/lib/log4j-core-2.12.1.jar
2021-02-24 16:19:37,612 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/lib/flink-csv-1.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/lib/flink-csv-1.12.1.jar with replication factor 1
2021-02-24 16:19:37,622 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/lib/flink-csv-1.12.1.jar
2021-02-24 16:19:37,623 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/conf/log4j.properties to file:/home/ubuntu/.flink/application_1614159836384_0045/log4j.properties with replication factor 1
2021-02-24 16:19:37,629 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/log4j.properties
2021-02-24 16:19:37,637 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/plugins/metrics-prometheus/flink-metrics-prometheus-1.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/metrics-prometheus/flink-metrics-prometheus-1.12.1.jar with replication factor 1
2021-02-24 16:19:37,651 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/metrics-prometheus/flink-metrics-prometheus-1.12.1.jar
2021-02-24 16:19:37,652 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/plugins/README.txt to file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/README.txt with replication factor 1
2021-02-24 16:19:37,657 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/README.txt
2021-02-24 16:19:37,658 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/plugins/external-resource-gpu/gpu-discovery-common.sh to file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/external-resource-gpu/gpu-discovery-common.sh with replication factor 1
2021-02-24 16:19:37,670 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/external-resource-gpu/gpu-discovery-common.sh
2021-02-24 16:19:37,671 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/plugins/external-resource-gpu/flink-external-resource-gpu-1.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/external-resource-gpu/flink-external-resource-gpu-1.12.1.jar with replication factor 1
2021-02-24 16:19:37,676 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/external-resource-gpu/flink-external-resource-gpu-1.12.1.jar
2021-02-24 16:19:37,677 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/plugins/external-resource-gpu/nvidia-gpu-discovery.sh to file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/external-resource-gpu/nvidia-gpu-discovery.sh with replication factor 1
2021-02-24 16:19:37,683 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/external-resource-gpu/nvidia-gpu-discovery.sh
2021-02-24 16:19:37,684 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/plugins/metrics-graphite/flink-metrics-graphite-1.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/metrics-graphite/flink-metrics-graphite-1.12.1.jar with replication factor 1
2021-02-24 16:19:37,696 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/metrics-graphite/flink-metrics-graphite-1.12.1.jar
2021-02-24 16:19:37,698 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/plugins/metrics-influx/flink-metrics-influxdb-1.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/metrics-influx/flink-metrics-influxdb-1.12.1.jar with replication factor 1
2021-02-24 16:19:37,712 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/metrics-influx/flink-metrics-influxdb-1.12.1.jar
2021-02-24 16:19:37,713 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/plugins/metrics-datadog/flink-metrics-datadog-1.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/metrics-datadog/flink-metrics-datadog-1.12.1.jar with replication factor 1
2021-02-24 16:19:37,727 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/metrics-datadog/flink-metrics-datadog-1.12.1.jar
2021-02-24 16:19:37,728 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/plugins/metrics-jmx/flink-metrics-jmx-1.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/metrics-jmx/flink-metrics-jmx-1.12.1.jar with replication factor 1
2021-02-24 16:19:37,738 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/metrics-jmx/flink-metrics-jmx-1.12.1.jar
2021-02-24 16:19:37,739 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/plugins/metrics-slf4j/flink-metrics-slf4j-1.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/metrics-slf4j/flink-metrics-slf4j-1.12.1.jar with replication factor 1
2021-02-24 16:19:37,748 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/metrics-slf4j/flink-metrics-slf4j-1.12.1.jar
2021-02-24 16:19:37,749 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/plugins/metrics-statsd/flink-metrics-statsd-1.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/metrics-statsd/flink-metrics-statsd-1.12.1.jar with replication factor 1
2021-02-24 16:19:37,761 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/plugins/metrics-statsd/flink-metrics-statsd-1.12.1.jar
2021-02-24 16:19:37,762 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/examples/streaming/TopSpeedWindowing.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/TopSpeedWindowing.jar with replication factor 1
2021-02-24 16:19:37,768 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183577000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/TopSpeedWindowing.jar
2021-02-24 16:19:37,769 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from file:/home/ubuntu/build-target/flink/lib/flink-dist_2.12-1.12.1.jar to file:/home/ubuntu/.flink/application_1614159836384_0045/flink-dist_2.12-1.12.1.jar with replication factor 1
2021-02-24 16:19:38,100 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183578000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/flink-dist_2.12-1.12.1.jar
2021-02-24 16:19:38,115 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Copying from /tmp/application_1614159836384_0045-flink-conf.yaml3004677402421957468.tmp to file:/home/ubuntu/.flink/application_1614159836384_0045/application_1614159836384_0045-flink-conf.yaml3004677402421957468.tmp with replication factor 1
2021-02-24 16:19:38,123 DEBUG org.apache.flink.yarn.YarnApplicationFileUploader [] - Got modification time 1614183578000 from remote path file:/home/ubuntu/.flink/application_1614159836384_0045/application_1614159836384_0045-flink-conf.yaml3004677402421957468.tmp
2021-02-24 16:19:38,241 DEBUG org.apache.flink.yarn.YarnClusterDescriptor [] - Application Master start command: $JAVA_HOME/bin/java -Xmx1073741824 -Xms1073741824 -XX:MaxMetaspaceSize=268435456 -Dlog.file="<LOG_DIR>/jobmanager.log" -Dlog4j.configuration=file:log4j.properties -Dlog4j.configurationFile=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnApplicationClusterEntryPoint -D jobmanager.memory.off-heap.size=134217728b -D jobmanager.memory.jvm-overhead.min=201326592b -D jobmanager.memory.jvm-metaspace.size=268435456b -D jobmanager.memory.heap.size=1073741824b -D jobmanager.memory.jvm-overhead.max=201326592b 1> <LOG_DIR>/jobmanager.out 2> <LOG_DIR>/jobmanager.err
2021-02-24 16:19:38,256 DEBUG org.apache.flink.yarn.YarnClusterDescriptor$ApplicationSubmissionContextReflector [] - org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext supports method setApplicationTags.
2021-02-24 16:19:38,256 DEBUG org.apache.flink.yarn.YarnClusterDescriptor$ApplicationSubmissionContextReflector [] - org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext supports method setAttemptFailuresValidityInterval.
2021-02-24 16:19:38,256 DEBUG org.apache.flink.yarn.YarnClusterDescriptor$ApplicationSubmissionContextReflector [] - org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext supports method setKeepContainersAcrossApplicationAttempts.
2021-02-24 16:19:38,256 DEBUG org.apache.flink.yarn.YarnClusterDescriptor$ApplicationSubmissionContextReflector [] - org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext supports method setNodeLabelExpression.
2021-02-24 16:19:38,257 DEBUG org.apache.flink.yarn.YarnClusterDescriptor$ApplicationSubmissionContextReflector [] - Calling method setApplicationTags of org.apache.hadoop.yarn.api.records.impl.pb.ApplicationSubmissionContextPBImpl.
2021-02-24 16:19:38,257 DEBUG org.apache.hadoop.service.AbstractService [] - Service: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state INITED
2021-02-24 16:19:38,257 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting application master application_1614159836384_0045
2021-02-24 16:19:38,282 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu sending #4 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.submitApplication
2021-02-24 16:19:38,289 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu got value #4
2021-02-24 16:19:38,290 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine [] - Call: submitApplication took 11ms
2021-02-24 16:19:38,301 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu sending #5 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport
2021-02-24 16:19:38,306 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu got value #5
2021-02-24 16:19:38,306 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine [] - Call: getApplicationReport took 5ms
2021-02-24 16:19:38,347 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted application application_1614159836384_0045
2021-02-24 16:19:38,347 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for the cluster to be allocated
2021-02-24 16:19:38,348 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu sending #6 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport
2021-02-24 16:19:38,350 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu got value #6
2021-02-24 16:19:38,351 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine [] - Call: getApplicationReport took 3ms
2021-02-24 16:19:38,351 DEBUG org.apache.flink.yarn.YarnClusterDescriptor [] - Application State: ACCEPTED
2021-02-24 16:19:38,352 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deploying cluster, current state ACCEPTED
2021-02-24 16:19:38,604 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu sending #7 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport
2021-02-24 16:19:38,606 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu got value #7
2021-02-24 16:19:38,607 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine [] - Call: getApplicationReport took 4ms
2021-02-24 16:19:38,607 DEBUG org.apache.flink.yarn.YarnClusterDescriptor [] - Application State: ACCEPTED
2021-02-24 16:19:38,858 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu sending #8 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport
2021-02-24 16:19:38,861 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu got value #8
2021-02-24 16:19:38,862 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine [] - Call: getApplicationReport took 4ms
2021-02-24 16:19:38,863 DEBUG org.apache.flink.yarn.YarnClusterDescriptor [] - Application State: ACCEPTED
2021-02-24 16:19:39,114 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu sending #9 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport
2021-02-24 16:19:39,117 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu got value #9
2021-02-24 16:19:39,117 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine [] - Call: getApplicationReport took 3ms
2021-02-24 16:19:39,118 DEBUG org.apache.flink.yarn.YarnClusterDescriptor [] - Application State: ACCEPTED
2021-02-24 16:19:39,369 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu sending #10 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport
2021-02-24 16:19:39,371 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu got value #10
2021-02-24 16:19:39,372 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine [] - Call: getApplicationReport took 3ms
2021-02-24 16:19:39,372 DEBUG org.apache.flink.yarn.YarnClusterDescriptor [] - Application State: ACCEPTED
2021-02-24 16:19:39,623 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu sending #11 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport
2021-02-24 16:19:39,625 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu got value #11
2021-02-24 16:19:39,625 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine [] - Call: getApplicationReport took 3ms
2021-02-24 16:19:39,626 DEBUG org.apache.flink.yarn.YarnClusterDescriptor [] - Application State: FAILED
2021-02-24 16:19:39,627 DEBUG org.apache.hadoop.service.AbstractService [] - Service: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state STOPPED
2021-02-24 16:19:39,627 DEBUG org.apache.hadoop.ipc.Client [] - stopping client from cache: org.apache.hadoop.ipc.Client@7ca20101
2021-02-24 16:19:39,627 DEBUG org.apache.hadoop.ipc.Client [] - removing client from cache: org.apache.hadoop.ipc.Client@7ca20101
2021-02-24 16:19:39,627 DEBUG org.apache.hadoop.ipc.Client [] - stopping actual client because no more references remain: org.apache.hadoop.ipc.Client@7ca20101
2021-02-24 16:19:39,627 DEBUG org.apache.hadoop.ipc.Client [] - Stopping client
2021-02-24 16:19:39,628 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu: closed
2021-02-24 16:19:39,628 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu: stopped, remaining connections 0
------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn Application Cluster
at org.apache.flink.yarn.YarnClusterDescriptor.deployApplicationCluster(YarnClusterDescriptor.java:465)
at org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:67)
at org.apache.flink.client.cli.CliFrontend.runApplication(CliFrontend.java:213)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1061)
at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1136)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1136)
Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment.
Diagnostics from YARN: Application application_1614159836384_0045 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1614159836384_0045_000001 exited with exitCode: -1000
Failing this attempt.Diagnostics: [2021-02-24 16:19:39.409]File file:/home/ubuntu/.flink/application_1614159836384_0045/flink-dist_2.12-1.12.1.jar does not exist
java.io.FileNotFoundException: File file:/home/ubuntu/.flink/application_1614159836384_0045/flink-dist_2.12-1.12.1.jar does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
at org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:269)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:67)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:414)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:411)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:411)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:242)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:235)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:223)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
For more detailed output, check the application tracking page: http://localhost:8088/cluster/app/application_1614159836384_0045 Then click on links to logs of each attempt.
. Failing the application.
If log aggregation is enabled on your cluster, use this command to further investigate the issue:
yarn logs -applicationId application_1614159836384_0045
at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1200)
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:592)
at org.apache.flink.yarn.YarnClusterDescriptor.deployApplicationCluster(YarnClusterDescriptor.java:458)
... 9 more
2021-02-24 16:19:39,732 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cancelling deployment from Deployment Failure Hook
2021-02-24 16:19:39,734 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at /0.0.0.0:8032
2021-02-24 16:19:39,734 DEBUG org.apache.hadoop.security.UserGroupInformation [] - PrivilegedAction as:ubuntu (auth:SIMPLE) from:org.apache.hadoop.yarn.client.RMProxy.getProxy(RMProxy.java:147)
2021-02-24 16:19:39,735 DEBUG org.apache.hadoop.yarn.ipc.YarnRPC [] - Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
2021-02-24 16:19:39,735 DEBUG org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC [] - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ApplicationClientProtocol
2021-02-24 16:19:39,735 DEBUG org.apache.hadoop.util.ShutdownHookManager [] - Completed shutdown in 0.002 seconds; Timeouts: 0
2021-02-24 16:19:39,735 DEBUG org.apache.hadoop.ipc.Client [] - ipc.client.bind.wildcard.addr set to true. Will bind client sockets to wildcard address.
2021-02-24 16:19:39,736 DEBUG org.apache.hadoop.ipc.Client [] - getting client out of cache: org.apache.hadoop.ipc.Client@20d4145c
2021-02-24 16:19:39,736 DEBUG org.apache.hadoop.service.AbstractService [] - Service org.apache.hadoop.yarn.client.api.impl.YarnClientImpl is started
2021-02-24 16:19:39,737 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Killing YARN application
2021-02-24 16:19:39,741 DEBUG org.apache.hadoop.ipc.Client [] - The ping interval is 60000 ms.
2021-02-24 16:19:39,741 DEBUG org.apache.hadoop.ipc.Client [] - Connecting to /0.0.0.0:8032
2021-02-24 16:19:39,743 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu: starting, having connections 1
2021-02-24 16:19:39,744 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu sending #12 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.forceKillApplication
2021-02-24 16:19:39,747 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu got value #12
2021-02-24 16:19:39,747 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine [] - Call: forceKillApplication took 7ms
2021-02-24 16:19:39,750 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Killed application application_1614159836384_0045
2021-02-24 16:19:39,751 DEBUG org.apache.hadoop.service.AbstractService [] - Service: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state STOPPED
2021-02-24 16:19:39,751 DEBUG org.apache.hadoop.ipc.Client [] - stopping client from cache: org.apache.hadoop.ipc.Client@20d4145c
2021-02-24 16:19:39,751 DEBUG org.apache.hadoop.ipc.Client [] - removing client from cache: org.apache.hadoop.ipc.Client@20d4145c
2021-02-24 16:19:39,751 DEBUG org.apache.hadoop.ipc.Client [] - stopping actual client because no more references remain: org.apache.hadoop.ipc.Client@20d4145c
2021-02-24 16:19:39,751 DEBUG org.apache.hadoop.ipc.Client [] - Stopping client
2021-02-24 16:19:39,752 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu: closed
2021-02-24 16:19:39,752 DEBUG org.apache.hadoop.ipc.Client [] - IPC Client (661047965) connection to /0.0.0.0:8032 from ubuntu: stopped, remaining connections 0
2021-02-24 16:19:39,754 DEBUG org.apache.hadoop.util.ShutdownHookManager [] - ShutdownHookManger completed shutdown.
2021-02-24 16:19:39,853 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deleting files in file:/home/ubuntu/.flink/application_1614159836384_0045.
2021-02-24 16:19:39,854 DEBUG org.apache.hadoop.fs.FileSystem [] - Looking for FS supporting file
2021-02-24 16:19:39,854 DEBUG org.apache.hadoop.fs.FileSystem [] - looking for configuration option fs.file.impl
2021-02-24 16:19:39,854 DEBUG org.apache.hadoop.fs.FileSystem [] - Looking in service filesystems for implementation class
2021-02-24 16:19:39,854 DEBUG org.apache.hadoop.fs.FileSystem [] - FS for file is class org.apache.hadoop.fs.LocalFileSystem
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment