Hadoop: Java exception "GSS initiate failed"

In a Hadoop/Revolution R Enterprise environment, a mapreduce job may fail with a kerberos-related Java exception:
Setting options: 
15/02/03 16:59:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
DEBUG: allArgs = [-Dmapred.reduce.tasks=1, /user/RevoShare/a377683/EE594B467CDE4C988C3C3F7AF563DABB/.input, /user/RevoShare/a377683/EE594B467CDE4C988C3C3F7AF563DABB/IRO.iro, /user/RevoShare/a377683/AirlineDemoSmall/*, dojo3m20002.rtp1.hadoop.fmr.com, 8020, /usr/bin/Revoscript] 
15/02/03 16:59:09 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 242420 for a377683 on ha-hdfs:nameservice1 
15/02/03 16:59:09 INFO security.TokenCache: Got dt for hdfs://nameservice1; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice1, Ident: (HDFS_DELEGATION_TOKEN token 242420 for a377683)

......

15/02/03 16:59:25 INFO mapreduce.Job: map 0% reduce 0% 
15/02/03 16:59:38 INFO mapreduce.Job: Task Id : attempt_1422647534016_1760_m_000000_0, Status : FAILED 
Error: java.io.IOException: R VM exited with exit code: 1, output: 
starting script 
[1] TRUE 
$fileName 
[1] NA 
$start 
[1] 0 
$length 
[1] 0 
15/02/03 16:59:37 WARN security.UserGroupInformation: PriviledgedActionException as:a377683 (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 
15/02/03 16:59:37 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 
15/02/03 16:59:37 WARN security.UserGroupInformation: PriviledgedActionException as:a377683 (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 
hdfsOpenFile(/user/RevoShare/a377683/EE594B467CDE4C988C3C3F7AF563DABB/.input): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: 
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "<hostname>"; destination host is: "<hostname>":8020; 
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) 
at org.apache.hadoop.ipc.Client.call(Client.java:1415) 
at org.apache.hadoop.ipc.Client.call(Client.java:1364) 
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) 
at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source) 
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:246) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:606) 
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) 
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) 
at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source) 
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1179) 
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1169) 
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1159) 
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270) 
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:237) 
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:230) 
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1457) 
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:301) 
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:297) 
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) 
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:297) 
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:679) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:415) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) 
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:642) 
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725) 
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) 
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) 
at org.apache.hadoop.ipc.Client.call(Client.java:1382)
First, confirm that the user submitting the job does have a current and valid kerberos ticket but it is possible to see this failure even with a confirmed valid ticket.

If the cluster is configured for name node high availability, it is necessary to be specific in the RxHadoopMR() and RxHdfsFileSystem() calls in the script to specify the name node service though specifying the active name node hostname will also work. In the case of a kerberized and HA name node environment, the HDFS_DELEGATION_TOKEN must match the nameservice reference. In the error above, see:
INFO security.TokenCache: Got dt for hdfs://nameservice1; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice1, Ident: (HDFS_DELEGATION_TOKEN token 242420 for a377683
In this case, the nameNode/hostName arguments in the RxHadoopMR() and RxHdfsFileSystem() calls should be set to "nameservice1".
Note This is a "FAST PUBLISH" article created directly from within the Microsoft support organization. The information contained herein is provided as-is in response to emerging issues. As a result of the speed in making it available, the materials may include typographical errors and may be revised at any time without notice. See Terms of Use for other considerations.
Vlastnosti

ID článku: 3104129 - Poslední kontrola: 10/31/2015 16:28:00 - Revize: 1.0

Revolution Analytics

  • KB3104129
Váš názor