Skip to content

Instantly share code, notes, and snippets.

@haridsv
Created June 6, 2016 13:15
Show Gist options
  • Save haridsv/13cd96381dde74296c68226121e7de04 to your computer and use it in GitHub Desktop.
Save haridsv/13cd96381dde74296c68226121e7de04 to your computer and use it in GitHub Desktop.
Cause of "Failed to renew token: Kind: HDFS_DELEGATION_TOKEN"

One of the reasons that this failure occurs is because of the incorrect client setting of yarn.resourcemanager.cluster-id that ends up not matching the one in the cluster. This often results in a cryptic error that looks like this:

Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:test-local-EMPTY, Ident: (HDFS_DELEGATION_TOKEN token 40242 for hari)

Unfortunately, the original IOException seems to get completely lost, neither logged on the server nor gets seen on the client. When I put an extra trace log line to print the exception, I got this:

2016-06-06 12:57:57,394 ERROR [tionTokenRenewer #50] security.DelegationTokenRenewer - Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:test-local-EMPTY, Ident: (HDFS_DELEGATION_TOKEN token 40663 for sfdc)
java.io.IOException: Unable to map logical nameservice URI 'hdfs://test-local-EMPTY' to a NameNode. Local configuration does not have a failover proxy provider configured.
        at org.apache.hadoop.hdfs.DFSClient$Renewer.getNNProxy(DFSClient.java:1177)
        at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:1141)
        at org.apache.hadoop.security.token.Token.renew(Token.java:377)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:582)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:579)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1706)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:577)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:463)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:79)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:881)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:858)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2016-06-06 12:57:57,395 WARN  [tionTokenRenewer #50] security.DelegationTokenRenewer - Unable to add the application to the delegation token renewer.
java.io.IOException: Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:test-local-EMPTY, Ident: (HDFS_DELEGATION_TOKEN token 40663 for sfdc)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:466)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:79)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:881)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:858)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unable to map logical nameservice URI 'hdfs://test-local-EMPTY' to a NameNode. Local configuration does not have a failover proxy provider configured.
        at org.apache.hadoop.hdfs.DFSClient$Renewer.getNNProxy(DFSClient.java:1177)
        at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:1141)
        at org.apache.hadoop.security.token.Token.renew(Token.java:377)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:582)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:579)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1706)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:577)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:463)
        ... 6 more
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment