win10下设置虚拟机配置hadoop-yarn单机伪分布环境

win10下设置虚拟机配置hadoop-yarn单机伪分布环境

本文以CentOS6.5为主，虚拟机以VirtualBox，hadoop版本为2.6.2：
一、设置ssh及网络
1. 配置ssh免密码登录

ssh-keygen -t rsa
cat id_rsa.pub >> authorized_keys

2.修改host主机配置
linux通过这个文件知道某个ip对应于某个主机名，如比方说google的ip是10.23.56.238，那么可以在这个文件的最后加上一行：
10.23.56.238 google.com

vi /etc/hosts

二、配置hadoop
1.下载并安装Hadoop

#mkdir -p /opt/yarn
#cd /opt/yarn
#tar xvzf hadoop-2.5.2.tar.gz

2.设置JAVA_HOME
本文以内置的openJDK为例，

#echo "export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre" > /etc/profile.d/java.sh
#source /etc/profile.d/java.sh

3.创建用户和用户组

#groupadd hadoop
#useradd -g hadoop yarn
#useradd -g hadoop hdfs
#useradd -g hadoop mapred

4.创建数据和日志目录
Hadoop需要不同权限的数据和日志目录，

#mkdir -p /var/data/hadoop/hdfs/nn
#mkdir -p /var/data/hadoop/hdfs/snn
#mkdir -p /var/data/hadoop/hdfs/dn
#chown hdfs:hadoop /var/data/hadoop/hdfs -R
#mkdir -p /var/log/hadoop/yarn
#chown yarn:hadoop /var/log/hadoop/yarn -R

进入YARN安装目录

#cd /opt/yarn/hadoop-2.5.2
#mkdir logs
#chmod g+w logs
chown yarn:hadoop . -R

5.配置core-site.xml

<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>
    <property>
        <name>hadoop.http.staticuser.user</name>
        <value>hdfs</value>
    </property>
</configuration>

6.配置hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property> 
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/var/data/hadoop/hdfs/nn</value>
    </property> 
    <property>
        <name>fs.checkpoint.dir</name>
        <value>file:/var/data/hadoop/hdfs/snn</value>
    </property> 
    <property>
        <name>fs.checkpoint.edits.dir</name>
        <value>file:/var/data/hadoop/hdfs/snn</value>
    </property> 
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/var/data/hadoop/hdfs/dn</value>
    </property>
</configuration>

7.配置mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

8配置yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>  
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
</configuration>

9.调整Java堆大小
安装Hadoop时将使用环境变量来决定每个Hadoop进程的堆大小。etc/hadoop/*-env.sh。
编辑文件etc/hadoop/hadoop-env.sh

export HADOOP_HEAPSIZE="500"
export HADOOP_NAMENODE_INIT_HEAPSIZE="500"

然后编辑mapred-env.sh

JAVA_HEAP_MAX=-Xmx500m
YARN_HEAPSIZE=500

10.格式化HDFS
切换到bin目录

#su -hdfs
#cd /opt/yarn/hadoop-2.5.2/bin
#./hdfs namenode -format

11.启动HDFS服务

#cd ../sbin
#./hadoop-daemon.sh start namenode
starting namenode,logging to /opt/yarn/hadoop-2.5.2/logs/hadoop-hdfs-namenode-limulus.out

#./hadoop-daemon.sh start secondarynamenode
starting namenode,logging to /opt/yarn/hadoop-2.5.2/logs/hadoop-hdfs-secondarynamenode-limulus.out

#./hadoop-daemon.sh start datanode
starting datanode,logging to /opt/yarn/hadoop-2.5.2/logs/hadoop-hdfs-datanode-limulus.out

停止hadoop服务

#./hadoop-daemon.sh stop datanode

12.启动YARN服务

$exit
logout
#su - yarn
$cd /opt/yarn/hadoop-2.5.2/sbin
$./yarn-daemon.sh start resourcemanager
starting resourcemanager,logging to /opt/yarn/hadoop-2.5.2/logs/hadoop-hdfs-resourcemanager-limulus.out
$./yarn-daemon.sh start nodemanager
starting nodemanager,logging to /opt/yarn/hadoop-2.5.2/logs/hadoop-hdfs-nodemanager-limulus.out

停止服务

#./yarn-daemon.sh stop nodemanager

13.通过Web接口验证正在运行的服务
namenode

firefox http://localhost:50070

ResourceManager

firefox http://localhost:8088

三、运行MapReduce示例程序

#su hdfs
$cd /opt/yarn/hadoop-2.5.2/bin
$export YARN_EXAMPLES=/opt/yarn/hadoop-2.5.2/share/hadoop/mapreduce
$./yarn jar $YARN_EXAMPLES/hadoop-mapreduce-examples-2.5.2.jar pi 16 1000
……
Estimated value of Pi is 3.142500000000000000000

所有例子的详细清单
./yarn $YARN_EXAMPLES/hadoop-mapreduce-examples-2.5.2.jar