MapR 4.0.1: Revolution R job fails with segfault

On a MapR 4.0.1 cluster, Revolution R mapreduce jobs may fail with the error:
2015-02-13 08:00:37,657 FATAL [IPC Server handler 2 on 52243] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1423734868663_0289_m_000000_0 - exited : java.io.IOException: R VM exited with exit code: 1, output: 
starting script 
[1] TRUE 
$fileName 
[1] NA 
$start 
[1] 0 
$length 
[1] 0 
/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/library/RevoScaleR/rxLibs/x64/libExaCore.so.2(_Z21CriticalSignalHandleri+0x1e)[0x7fe02d3d423e] 
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64/jre/lib/amd64/server/libjvm.so(+0x7e2628)[0x7fe0282f8628] 
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64/jre/lib/amd64/server/libjvm.so(JVM_handle_linux_signal+0x95)[0x7fe0282fea55] 
/lib64/libpthread.so.0[0x30cec0f710] 
/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/library/RevoScaleR/rxLibs/x64/libExaCore.so.2(_ZNK10CxAnalysis19HaveWriteDataSourceEv+0x0)[0x7fe02acbc280] 
/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/library/RevoScaleR/rxLibs/x64/libExaCore.so.2(_ZN10CxAnalysis18GetWriteDataSourceEv+0x9)[0x7fe02acbcb49] 
/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/library/RevoScaleR/rxLibs/x64/libRxLink.so.2(_Z19RxComputeHadoopTaskR11RListObjectR11RxAllocator+0x3b3)[0x7fe02e2a5d23] 
/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/library/RevoScaleR/rxLibs/x64/libRxLink.so.2(_Z10RxUtilBaseP7SEXPREC+0x6db)[0x7fe02e2b2fab] 
/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/library/RevoScaleR/rxLibs/x64/libRxLink.so.2(RxUtil+0x34)[0x7fe02e2b3f14] 
/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/lib/libR.so(+0x9f36b)[0x7fe04083536b] 
/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/lib/libR.so(Rf_eval+0x871)[0x7fe04086be91] 
/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/lib/libR.so(Rf_ReplIteration+0x212)[0x7fe0408946a2] 
/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/lib/libR.so(+0xfea59)[0x7fe040894a59] 
/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/lib/libR.so(run_Rmainloop+0x44)[0x7fe040894f64] 
/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/bin/exec/R(main+0x1b)[0x40084b] 
/lib64/libc.so.6(__libc_start_main+0xfd)[0x30ce41ed5d] 
/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/bin/exec/R[0x400739] 
Segmentation fault.

at com.revolutionanalytics.scaleR.hadoop.Rscript.waitFor(Rscript.java:103) 
at com.revolutionanalytics.scaleR.hadoop.Rscript.run(Rscript.java:59) 
at com.revolutionanalytics.scaleR.hadoop.RDelegatingMapper.run(RDelegatingMapper.java:73) 
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:773) 
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:345) 
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:415) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1469) 
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
On MapR 4.0.1 platforms, this is due to the Revolution R Enterprise environment configuration using the wrong libhdfs.so library.

To work around the problem, confirm the following post-installation configuration steps have been taken for Revolution R 7.3 on MapR 4.0.1 clusters:

On the "client" node, create the native filesystem shared directory for each user:

1. sudo mkdir -p /var/RevoShare/mapr 
3. chmod 777 /var/RevoShare/mapr 
4. add . /usr/lib64/Revo-7.3/scripts/RevoHadoopEnvVars.site to /home/mapr/.bash_profile

On EVERY worker node, make the following changes:

# cp scaleR-hadoop-0.1-SNAPSHOT.jar /opt/mapr/hadoop/hadoop-2.4.1/share/hadoop/common/lib/ # ln -s /opt/mapr/hadoop/hadoop-0.20.2/c++/Linux-amd64-64/lib/libhdfs.so /usr/lib64 # mv /opt/mapr/hadoop/hadoop-2.4.1/lib/native/libhdfs.so /opt/mapr/hadoop/hadoop-2.4.1/lib/native/libhdfs.so.save


The installation of Revolution R automatically creates a link to scaleR-hadoop-0.1-SNAPSHOT.jar in $HADOOP_HOME/lib. However, this is the incorrect directory (a bug has been reported) and the file should be linked or copied in /opt/mapr/hadoop/hadoop-2.4.1/share/hadoop/common/lib/.

Linking /opt/mapr/hadoop/hadoop-0.20.2/c++/Linux-amd64-64/lib/libhdfs.so in /usr/lib64 will ensure that the proper libhdfs.so is found by Revolution R. The third step is necessary to move the problem libhdfs.so out of the way.
Note This is a "FAST PUBLISH" article created directly from within the Microsoft support organization. The information contained herein is provided as-is in response to emerging issues. As a result of the speed in making it available, the materials may include typographical errors and may be revised at any time without notice. See Terms of Use for other considerations.
Properties

Article ID: 3104128 - Last Review: 10/31/2015 16:04:00 - Revision: 1.0

Revolution Analytics

  • KB3104128
Feedback