One of the reasons that this failure occurs is because of the incorrect client setting of yarn.resourcemanager.cluster-id that ends up not matching the one in the cluster. This often results in a cryptic error that looks like this:
Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:test-local-EMPTY, Ident: (HDFS_DELEGATION_TOKEN token 40242 for hari)
Unfortunately, the original IOException seems to get completely lost, neither logged on the server nor gets seen on the client. When I put an extra trace log line to print the exception, I got this:
2016-06-06 12:57:57,394 ERROR [tionTokenRenewer #50] security.DelegationTokenRenewer - Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:test-local-EMPTY, Ident: (HDFS_DELEGATION_TOKEN token 40663 for sfdc)
java.io.IOException: Unable to map logical nameservice URI 'hdfs://test-local-EMPTY' to a NameNode. Local configuration does not have a failover proxy provider configured.
at org.apache.hadoop.hdfs.DFSClient$Renewer.getNNProxy(DFSClient.java:1177)
at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:1141)
at org.apache.hadoop.security.token.Token.renew(Token.java:377)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:582)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:579)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1706)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:577)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:463)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:79)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:881)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:858)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-06-06 12:57:57,395 WARN [tionTokenRenewer #50] security.DelegationTokenRenewer - Unable to add the application to the delegation token renewer.
java.io.IOException: Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:test-local-EMPTY, Ident: (HDFS_DELEGATION_TOKEN token 40663 for sfdc)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:466)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:79)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:881)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:858)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unable to map logical nameservice URI 'hdfs://test-local-EMPTY' to a NameNode. Local configuration does not have a failover proxy provider configured.
at org.apache.hadoop.hdfs.DFSClient$Renewer.getNNProxy(DFSClient.java:1177)
at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:1141)
at org.apache.hadoop.security.token.Token.renew(Token.java:377)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:582)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:579)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1706)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:577)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:463)
... 6 more