Skip to content

Instantly share code, notes, and snippets.

@headmyshoulder
Created August 13, 2015 20:31
Show Gist options
  • Save headmyshoulder/48279f06c1850c62c28c to your computer and use it in GitHub Desktop.
Save headmyshoulder/48279f06c1850c62c28c to your computer and use it in GitHub Desktop.
$ ./bin/yarn-session.sh -n 4 -jm 1024 -tm 4096
20:30:55,403 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at /0.0.0.0:8032
20:30:55,566 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20:30:55,911 WARN org.apache.flink.yarn.FlinkYarnClient - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set.The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
20:30:55,911 INFO org.apache.flink.yarn.FlinkYarnClient - Using values:
20:30:55,913 INFO org.apache.flink.yarn.FlinkYarnClient - TaskManager count = 4
20:30:55,913 INFO org.apache.flink.yarn.FlinkYarnClient - JobManager memory = 1024
20:30:55,913 INFO org.apache.flink.yarn.FlinkYarnClient - TaskManager memory = 4096
20:30:56,238 WARN org.apache.flink.yarn.FlinkYarnClient - The file system scheme is 'file'. This indicates that the specified Hadoop configuration path is wrong and the sytem is using the default Hadoop configuration values.The Flink YARN client needs to store its files in a distributed file system
20:30:56,247 INFO org.apache.flink.yarn.Utils - Copying from file:/home/hadoop/flink-0.9.0/lib/flink-dist-0.9.0.jar to file:/home/hadoop/.flink/application_1439480667820_0003/flink-dist-0.9.0.jar
20:30:56,879 INFO org.apache.flink.yarn.Utils - Copying from /home/hadoop/flink-0.9.0/conf/flink-conf.yaml to file:/home/hadoop/.flink/application_1439480667820_0003/flink-conf.yaml
20:30:56,885 INFO org.apache.flink.yarn.Utils - Copying from file:/home/hadoop/flink-0.9.0/lib/flink-python-0.9.0.jar to file:/home/hadoop/.flink/application_1439480667820_0003/flink-python-0.9.0.jar
20:30:56,912 INFO org.apache.flink.yarn.Utils - Copying from file:/home/hadoop/flink-0.9.0/conf/logback.xml to file:/home/hadoop/.flink/application_1439480667820_0003/logback.xml
20:30:56,918 INFO org.apache.flink.yarn.Utils - Copying from file:/home/hadoop/flink-0.9.0/conf/log4j.properties to file:/home/hadoop/.flink/application_1439480667820_0003/log4j.properties
20:30:56,930 INFO org.apache.flink.yarn.FlinkYarnClient - Submitting application master application_1439480667820_0003
20:30:56,957 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1439480667820_0003
20:30:56,957 INFO org.apache.flink.yarn.FlinkYarnClient - Waiting for the cluster to be allocated
20:30:56,959 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
20:30:57,961 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
Error while deploying YARN cluster: The YARN application unexpectedly switched to state FAILED during deployment.
Diagnostics from YARN: Application application_1439480667820_0003 failed 1 times due to AM Container for appattempt_1439480667820_0003_000001 exited with exitCode: -1000
For more detailed output, check application tracking page:http://ip-172-31-12-138.eu-central-1.compute.internal:20888/proxy/application_1439480667820_0003/Then, click on links to logs of each attempt.
Diagnostics: File file:/home/hadoop/.flink/application_1439480667820_0003/flink-conf.yaml does not exist
java.io.FileNotFoundException: File file:/home/hadoop/.flink/application_1439480667820_0003/flink-conf.yaml does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:539)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:752)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:529)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:419)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Failing this attempt. Failing the application.
If log aggregation is enabled on your cluster, use this command to further invesitage the issue:
yarn logs -applicationId application_1439480667820_0003
org.apache.flink.yarn.FlinkYarnClient$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment.
Diagnostics from YARN: Application application_1439480667820_0003 failed 1 times due to AM Container for appattempt_1439480667820_0003_000001 exited with exitCode: -1000
For more detailed output, check application tracking page:http://ip-172-31-12-138.eu-central-1.compute.internal:20888/proxy/application_1439480667820_0003/Then, click on links to logs of each attempt.
Diagnostics: File file:/home/hadoop/.flink/application_1439480667820_0003/flink-conf.yaml does not exist
java.io.FileNotFoundException: File file:/home/hadoop/.flink/application_1439480667820_0003/flink-conf.yaml does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:539)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:752)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:529)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:419)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Failing this attempt. Failing the application.
If log aggregation is enabled on your cluster, use this command to further invesitage the issue:
yarn logs -applicationId application_1439480667820_0003
at org.apache.flink.yarn.FlinkYarnClient.deployInternal(FlinkYarnClient.java:621)
at org.apache.flink.yarn.FlinkYarnClient.deploy(FlinkYarnClient.java:334)
at org.apache.flink.client.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:396)
at org.apache.flink.client.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:338)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment