Skip to content

Instantly share code, notes, and snippets.

@hsaputra

hsaputra/gist:b47a91597001ae43c4eb

Created April 16, 2015 21:46
Show Gist options
  • Save hsaputra/b47a91597001ae43c4eb to your computer and use it in GitHub Desktop.
Save hsaputra/b47a91597001ae43c4eb to your computer and use it in GitHub Desktop.
Download ZIP
Apache Flink YARN Session Flow
Raw
gistfile1.txt
flink-0.9-SNAPSHOT/bin ./yarn-session.sh -n 4 -tm 1024 -jm 1024 -s 5
14:26:49,187 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14:26:49,346 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at /0.0.0.0:8032
14:26:49,370 INFO org.apache.flink.yarn.FlinkYarnClient - Using values:
14:26:49,371 INFO org.apache.flink.yarn.FlinkYarnClient - TaskManager count = 4
14:26:49,372 INFO org.apache.flink.yarn.FlinkYarnClient - JobManager memory = 1024
14:26:49,372 INFO org.apache.flink.yarn.FlinkYarnClient - TaskManager memory = 1024
14:26:49,715 INFO org.apache.flink.yarn.Utils - Copying from file:/Users/henrysaputra/flink-installs/flink-0.9-SNAPSHOT/lib/flink-dist-0.9-SNAPSHOT.jar to hdfs://localhost:9000/user/henrysaputra/.flink/application_1429213513983_0002/flink-dist-0.9-SNAPSHOT.jar
14:26:50,146 INFO org.apache.flink.yarn.Utils - Copying from /Users/henrysaputra/flink-installs/flink-0.9-SNAPSHOT/conf/flink-conf.yaml to hdfs://localhost:9000/user/henrysaputra/.flink/application_1429213513983_0002/flink-conf.yaml
14:26:50,159 INFO org.apache.flink.yarn.Utils - Copying from file:/Users/henrysaputra/flink-installs/flink-0.9-SNAPSHOT/conf/logback.xml to hdfs://localhost:9000/user/henrysaputra/.flink/application_1429213513983_0002/logback.xml
14:26:50,171 INFO org.apache.flink.yarn.Utils - Copying from file:/Users/henrysaputra/flink-installs/flink-0.9-SNAPSHOT/conf/log4j.properties to hdfs://localhost:9000/user/henrysaputra/.flink/application_1429213513983_0002/log4j.properties
14:26:50,190 INFO org.apache.flink.yarn.FlinkYarnClient - Submitting application master application_1429213513983_0002
14:26:50,209 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1429213513983_0002 to ResourceManager at /0.0.0.0:8032
14:26:50,209 INFO org.apache.flink.yarn.FlinkYarnClient - Waiting for the cluster to be allocated
14:26:50,211 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
14:26:51,216 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
14:26:52,219 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
14:26:53,221 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
14:26:54,225 INFO org.apache.flink.yarn.FlinkYarnClient - YARN application has been deployed successfully.
14:26:54,405 INFO org.apache.flink.yarn.FlinkYarnCluster - Start actor system.
14:26:54,411 INFO org.apache.flink.runtime.net.NetUtils - Determined /10.150.22.250 as the machine's own IP address
14:26:54,678 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
14:26:54,709 INFO Remoting - Starting remoting
14:26:54,805 INFO Remoting - Remoting started; listening on addresses :[akka.tcp://flink@10.150.22.250:64012]
14:26:54,809 INFO org.apache.flink.yarn.FlinkYarnCluster - Start application client.
Flink JobManager is now running on 10.150.22.250:64004
JobManager Web Interface: http://kingwulf.local:8088/proxy/application_1429213513983_0002/
New messages from the YARN cluster:
[Thu Apr 16 14:26:56 PDT 2015] Launching container 0 (container_1429213513983_0002_01_000002 on host 10.150.22.250).
[Thu Apr 16 14:26:57 PDT 2015] Launching container 1 (container_1429213513983_0002_01_000003 on host 10.150.22.250).
Number of connected TaskManagers changed to 1. Slots available: 5
New messages from the YARN cluster:
[Thu Apr 16 14:26:58 PDT 2015] Launching container 2 (container_1429213513983_0002_01_000004 on host 10.150.22.250).
[Thu Apr 16 14:26:59 PDT 2015] Launching container 3 (container_1429213513983_0002_01_000005 on host 10.150.22.250).
Number of connected TaskManagers changed to 4. Slots available: 20
^C14:44:42,593 INFO org.apache.flink.yarn.FlinkYarnCluster - Shutting down FlinkYarnCluster from the client shutdown hook
14:44:42,593 INFO org.apache.flink.yarn.FlinkYarnCluster - Sending shutdown request to the Application Master
14:44:42,595 INFO org.apache.flink.yarn.ApplicationClient - Sending StopYarnSession request to ApplicationMaster.
14:44:42,723 INFO org.apache.flink.client.FlinkYarnSessionCli - Stopping interactive command line interface, YARN cluster has been stopped.
14:44:42,738 INFO org.apache.flink.yarn.ApplicationClient - Remote JobManager has been stopped successfully. Stopping local application client
14:44:42,741 INFO org.apache.flink.yarn.ApplicationClient - Stopped Application client.
14:44:42,777 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon.
14:44:42,778 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports.
14:44:42,792 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting shut down.
14:44:42,811 INFO org.apache.flink.yarn.FlinkYarnCluster - Deleting files in hdfs://localhost:9000/user/henrysaputra/.flink/application_1429213513983_0002
14:44:42,813 INFO org.apache.flink.yarn.FlinkYarnCluster - Application application_1429213513983_0002 finished with state FINISHED and final state SUCCEEDED at 1429220682605
14:44:43,545 INFO org.apache.flink.yarn.FlinkYarnCluster - YARN Client is shutting down
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment