Internal Error: Cannot reset hdfs internal params while connected to an hdfs file system

Issue found on Cloudera CDH4, but applicable to any supported Hadoop version.

When running a Hadoop script in "local" context  results are produced, but when running in hadoop context it shows following error:

"Internal Error: Cannot reset hdfs internal params while connected to an hdfs file system."

Possible causes:

1.-  You are either not setting the 'nameNode' option correctly in the RxHdfsFileSystem() command or you are using the wrong port number for HDFS in that same function.

If you are running your code form an EdgeNode, please make sure that the setting for 'hostName' is the actual name of the nameNode and NOT the hostname of the edgenode you are running from.

Also, please check Cloudera Manager and verify that the HDFS service is using port '8020', the default port. If it is running on a different port, you will need to explicitly set that in the following call in your Hadoop test script:

(For example)

myNameNode <- "test1.acme.com"  

myPort <- 1700 

hdfsFS <- RxHdfsFileSystem(hostName=myNameNode, port=myPort)

2.-  In case of specifying an edge node you need to specify the same hostname and port settings in both RxHadoopMR() and RxHdfsFileSystem()

3.-  Make sure that you have copied the RevoScaleR jar file from the directory in which you launched the Revolution installer (Revolution folder), 

scaleR-hadoop-0.1-SNAPSHOT.jar into the Cloudera Hadoop lib directory, which typically is:

/opt/cloudera/parcels/CDH/lib/hadoop/lib  (for parcels)  or

/usr/lib/hadoop/lib/ 

This file needs to be copied into this folder on ALL of the nodes of your Hadoop cluster.

After modifying any of these parameters, run again the script that initially showed the error.

If the error persists, contact Technical Support, for a deeper troubleshooting.
Note This is a "FAST PUBLISH" article created directly from within the Microsoft support organization. The information contained herein is provided as-is in response to emerging issues. As a result of the speed in making it available, the materials may include typographical errors and may be revised at any time without notice. See Terms of Use for other considerations.
Properties

Article ID: 3104168 - Last Review: 11/01/2015 05:21:00 - Revision: 1.0

Revolution Analytics

  • KB3104168
Feedback