Skip to content

Instantly share code, notes, and snippets.

@rmetzger

rmetzger/gist:bcb57a01aa15d9a64cc3

Created August 30, 2015 12:17
Show Gist options
  • Save rmetzger/bcb57a01aa15d9a64cc3 to your computer and use it in GitHub Desktop.
Save rmetzger/bcb57a01aa15d9a64cc3 to your computer and use it in GitHub Desktop.
Download ZIP
Run Apache Flink on a Kerberos secured YARN cluster on CDH 5.3.0
Raw
gistfile1.txt
[cloudera@quickstart build-target]$ klist
Ticket cache: FILE:/tmp/krb5cc_501
Default principal: hdfs@CLOUDERA
Valid starting Expires Service principal
08/30/15 05:15:22 08/31/15 05:15:22 krbtgt/CLOUDERA@CLOUDERA
renew until 09/06/15 05:15:22
[cloudera@quickstart build-target]$ export HADOOP_CONF_DIR=/usr/lib/hadoop/etc/hadoop/
[cloudera@quickstart build-target]$ ./bin/yarn-session.sh -n 1
05:15:37,483 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
05:15:37,509 INFO org.apache.flink.yarn.FlinkYarnClient - Using values:
05:15:37,510 INFO org.apache.flink.yarn.FlinkYarnClient - TaskManager count = 1
05:15:37,510 INFO org.apache.flink.yarn.FlinkYarnClient - JobManager memory = 1024
05:15:37,510 INFO org.apache.flink.yarn.FlinkYarnClient - TaskManager memory = 1024
05:15:38,135 INFO org.apache.flink.yarn.Utils - Copying from file:/home/cloudera/flink/flink-dist/target/flink-0.10-SNAPSHOT-bin/flink-0.10-SNAPSHOT/lib/flink-dist-0.10-SNAPSHOT.jar to hdfs://quickstart.cloudera:8020/user/hdfs/.flink/application_1440768826963_0005/flink-dist-0.10-SNAPSHOT.jar
05:15:38,981 INFO org.apache.flink.yarn.Utils - Copying from /home/cloudera/flink/flink-dist/target/flink-0.10-SNAPSHOT-bin/flink-0.10-SNAPSHOT/conf/flink-conf.yaml to hdfs://quickstart.cloudera:8020/user/hdfs/.flink/application_1440768826963_0005/flink-conf.yaml
05:15:38,996 INFO org.apache.flink.yarn.Utils - Copying from file:/home/cloudera/flink/flink-dist/target/flink-0.10-SNAPSHOT-bin/flink-0.10-SNAPSHOT/lib/flink-python-0.10-SNAPSHOT.jar to hdfs://quickstart.cloudera:8020/user/hdfs/.flink/application_1440768826963_0005/flink-python-0.10-SNAPSHOT.jar
05:15:39,414 INFO org.apache.flink.yarn.Utils - Copying from file:/home/cloudera/flink/flink-dist/target/flink-0.10-SNAPSHOT-bin/flink-0.10-SNAPSHOT/conf/logback.xml to hdfs://quickstart.cloudera:8020/user/hdfs/.flink/application_1440768826963_0005/logback.xml
05:15:39,428 INFO org.apache.flink.yarn.Utils - Copying from file:/home/cloudera/flink/flink-dist/target/flink-0.10-SNAPSHOT-bin/flink-0.10-SNAPSHOT/conf/log4j.properties to hdfs://quickstart.cloudera:8020/user/hdfs/.flink/application_1440768826963_0005/log4j.properties
05:15:39,470 INFO org.apache.hadoop.hdfs.DFSClient - Created HDFS_DELEGATION_TOKEN token 27 for hdfs on 127.0.0.1:8020
05:15:39,479 INFO org.apache.hadoop.mapreduce.security.TokenCache - Got dt for hdfs://quickstart.cloudera:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 127.0.0.1:8020, Ident: (HDFS_DELEGATION_TOKEN token 27 for hdfs)
05:15:39,481 INFO org.apache.flink.yarn.FlinkYarnClient - Submitting application master application_1440768826963_0005
05:15:40,510 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1440768826963_0005 to ResourceManager at quickstart.cloudera/127.0.0.1:8032
05:15:40,510 INFO org.apache.flink.yarn.FlinkYarnClient - Waiting for the cluster to be allocated
05:15:40,511 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
05:15:41,513 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
05:15:42,515 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
05:15:43,517 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
05:15:44,519 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
05:15:45,521 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
05:15:46,523 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
05:15:47,526 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
05:15:48,528 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
05:15:49,530 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
05:15:50,532 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
05:15:51,534 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
05:15:52,536 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
05:15:53,543 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
05:15:54,546 INFO org.apache.flink.yarn.FlinkYarnClient - Deploying cluster, current state ACCEPTED
05:15:55,548 INFO org.apache.flink.yarn.FlinkYarnClient - YARN application has been deployed successfully.
05:15:55,745 INFO org.apache.flink.yarn.FlinkYarnCluster - Start actor system.
05:15:55,750 INFO org.apache.flink.runtime.net.NetUtils - Determined /127.0.0.1 as the machine's own IP address
05:15:56,095 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
05:15:56,153 INFO Remoting - Starting remoting
05:15:56,310 INFO Remoting - Remoting started; listening on addresses :[akka.tcp://flink@127.0.0.1:40678]
05:15:56,315 INFO org.apache.flink.yarn.FlinkYarnCluster - Start application client.
Flink JobManager is now running on quickstart.cloudera:39956
JobManager Web Interface: http://quickstart.cloudera:8088/proxy/application_1440768826963_0005/
Number of connected TaskManagers changed to 1. Slots available: 1
^C05:16:05,394 INFO org.apache.flink.yarn.FlinkYarnCluster - Shutting down FlinkYarnCluster from the client shutdown hook
05:16:05,395 INFO org.apache.flink.yarn.FlinkYarnCluster - Sending shutdown request to the Application Master
05:16:05,397 INFO org.apache.flink.yarn.ApplicationClient - Sending StopYarnSession request to ApplicationMaster.
05:16:05,406 INFO org.apache.flink.client.FlinkYarnSessionCli - Stopping interactive command line interface, YARN cluster has been stopped.
05:16:05,557 INFO org.apache.flink.yarn.ApplicationClient - Remote JobManager has been stopped successfully. Stopping local application client
05:16:05,564 INFO org.apache.flink.yarn.ApplicationClient - Stopped Application client.
05:16:05,587 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon.
05:16:05,587 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports.
05:16:05,667 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting shut down.
05:16:05,690 INFO org.apache.flink.yarn.FlinkYarnCluster - Deleting files in hdfs://quickstart.cloudera:8020/user/hdfs/.flink/application_1440768826963_0005
05:16:05,693 INFO org.apache.flink.yarn.FlinkYarnCluster - Application application_1440768826963_0005 finished with state FINISHED and final state FAILED at 1440936965412
05:16:06,404 INFO org.apache.flink.yarn.FlinkYarnCluster - YARN Client is shutting down
[cloudera@quickstart build-target]$
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment