1) RHadoop安裝
下載RHadoop相關的2個程序包:
https://github.com/RevolutionAnalytics/RHadoop/wiki/Downloads
rmr-3.3.1
rhdfs-1.0.8
安裝依賴庫:
安裝上面的兩個庫之前,我們需要先安裝這兩個庫的依賴庫。
首先是rJava,由於我們已經配置好JDK1.8的環境,運行R CMD javareconf命令,R的程序從系統變量中會讀取Java配置。然後打開R程序,通過install.packages的方式,安裝rJava。
然後,我們還要安裝其他的幾個依賴庫,reshape2,Rcpp,iterators,itertools,digest,RJSONIO,functional,通過install.packages都直接安裝。
命令行執行
[Command005]:
R CMD javareconf
[Command006]:
R
啓動R程序
[Command007]:
install.packages(“rJava”)
[Command008]:
install.packages(“reshape2”)
[Command009]:
install.packages(“Rcpp”)
[Command010]:
install.packages(“iterators”)
[Command011]:
install.packages(“itertools”)
[Command012]:
install.packages(“digest”)
[Command013]:
install.packages(“RJSONIO”)
[Command014]:
install.packages(“functional”)
[Command015]:
install.packages(“caTools”)
配置rhdfs庫:
hadoop-streaming-<version>.jar下載地址:
http://www.java2s.com/Code/Jar/h/Downloadhadoopstreaming210betajar.htm
然後在hadoop目錄下建contrib文件夾,並在它文件夾下建立streaming文件夾,最後把hadoop-streaming-<version>.jar放在/contrib/streaming/
文件夾下。
例如我用的路徑:
/home/hyxy/project/hadoop-2.7.3/contrib/streaming/hadoop-streaming-2.1.0-beta.jar
把HADOOP_CMD和HADOOP_STREAMING設置到環境變量
[Command016]:
vi ~/.bashrc
exportHADOOP_CMD=/home/hyxy/project/hadoop-2.7.3/bin/hadoop
exportHADOOP_STREAMING=/home/hyxy/project/hadoop-2.7.3/contrib/streaming/ hadoop-streaming-2.1.0-beta.jar
[Command017]:
source ~/.bashrc
修改/etc/Hadoop/yarn-site.xml
[Code001]:
<property>
<name>yarn.scheduler.maximum-allocation-mb </name>
<value>9216</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>4000</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>4.1</value>
</property>
安裝rmr庫和rhdfs庫:
[Command 018]:
R CMD INSTALL rmr2_3.3.1.tar.gz
[Command018]:
R CMD INSTALL rhdfs_1.0.8.tar.gz