Centos6.X虛擬機配置Hadoop2.6的步驟

1. 通過xmanager的Xftp上傳hadoop-2.6.4.tar.gz文件到/opt目錄

2. 解壓縮hadoop-2.6.4.tar.gz 文件
   tar -zxf hadoop-2.6.4.tar.gz -C /usr/local
   

解壓後即可，看到/usr/local/hadoop-2.6.4文件夾

1. 配置Hadoop

進入目錄：
cd /usr/local/hadoop-2.6.4/etc/hadoop/
依次修改下面的文件：
4.1 core-site.xml（在底部Configuration標籤內添加以下Configuration標籤內的內容，其他文件也是）
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/log/hadoop/tmp</value>
</property>
</configuration>

4.2 hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.7.0_80

4.3 hdfs-site.xml

<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///data/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///data/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:50090</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>

4.4 mapred-site.xml
複製cp mapred-site.xml.template mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<!-- jobhistory properties -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>

4.5 yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/data/hadoop/yarn/local</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/data/tmp/logs</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://master:19888/jobhistory/logs/</value>;
<description>URL for job history server</description>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>512</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>4096</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>2048</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>1</value>
</property>
</configuration>

4.6 yarn-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_80

4.7 slaves
刪除localhost，添加：
slave1
slave2
slave3

4.8 設置IP映射
編輯各節點/etc/hosts
添加：
192.168.128.130 master master.centos.com
192.168.128.131 slave1 slave1.centos.com
192.168.128.132 slave2 slave2.centos.com
192.168.128.133 slave3 slave3.centos.com

4.9 克隆虛擬機，如果在配置前已提前克隆，則需拷貝hadoop安裝文件到集羣slave節點
scp -r /usr/local/hadoop-2.6.4 slave1:/usr/local
scp -r /usr/local/hadoop-2.6.4 slave2:/usr/local
scp -r /usr/local/hadoop-2.6.4 slave3:/usr/local

5.配置SSH無密碼登錄
（1）使用ssh-keygen產生公鑰與私鑰對。
輸入命令“ssh-keygen -t rsa”，接着按三次Enter鍵

[root@master ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
a6:13:5a:7b:54:eb:77:58:bd:56:ef:d0:64:90:66:d4 [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
| .. |
| . .E|
| . = |
| . . o o |
| o S . . =|
| o * . o ++|
| . + . . o ooo|
| o . ..o |
| .|
+-----------------+

生成私有密鑰id_rsa和公有密鑰id_rsa.pub兩個文件。
（2）用ssh-copy-id將公鑰複製到遠程機器中
ssh-copy-id -i /root/.ssh/id_rsa.pub master//依次輸入yes,123456（root用戶的密碼）
ssh-copy-id -i /root/.ssh/id_rsa.pub slave1
ssh-copy-id -i /root/.ssh/id_rsa.pub slave2
ssh-copy-id -i /root/.ssh/id_rsa.pub slave3
（3）驗證是否設置無密碼登錄
依次輸入
ssh slave1
ssh slave2
ssh slave3

6.配置時間同步服務
（1）安裝NTP服務。在各節點：
yum -y install ntp
（2）設置master節點爲NTP服務主節點，
使用命令“vim /etc/ntp.conf”打開/etc/ntp.conf文件，註釋掉以server開頭的行，並添加:
restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
server 127.127.1.0
fudge 127.127.1.0 stratum 10

（3）在所有slave節點中配置NTP，同樣修改/etc/ntp.conf文件，註釋掉server開頭的行，並添加:
server master

（4）執行命令“service iptables stop & chkconfig iptables off”永久性關閉防火牆，主節點和從節點都要關閉。
（5）啓動NTP服務。
① 在master節點執行命令“service ntpd start & chkconfig ntpd on”
② 在slave1、slave2、slave3上執行命令“ntpdate master”即可同步時間
③ 在slave1、slave2、slave3上分別執行“service ntpd start & chkconfig ntpd on”即可啓動並永久啓動NTP服務。

7.在/etc/profile添加JAVA_HOME和Hadoop路徑
export HADOOP_HOME=/usr/local/hadoop-2.6.4
export JAVA_HOME=/usr/java/jdk1.7.0_80
export PATH=$HADOOP_HOME/bin:$PATH:$JAVA_HOME/bin

source /etc/profile使修改生效

1. 格式化NameNode
   進入目錄
   cd /usr/local/hadoop-2.6.4/bin
   
   

執行格式化
./hdfs namenode -format

9.啓動集羣
進入目錄
cd /usr/local/hadoop-2.6.4/sbin
執行啓動：
./start-dfs.sh
./start-yarn.sh
./mr-jobhistory-daemon.sh start historyserver

使用jps，查看進程
[root@master sbin]# jps
1765 NameNode
1929 SecondaryNameNode
2378 JobHistoryServer
2412 Jps
2077 ResourceManager

[root@slave1 ~]# jps
1844 Jps
1612 DataNode
1711 NodeManager

1. 關閉防火牆（在所有節點執行）：
   service iptables stop
   chkconfig iptables off
   
   

11.在Windows下C:\Windows\System32\drivers\etc\hosts添加IP映射
192.168.128.130 master master.centos.com
192.168.128.131 slave1 slave1.centos.com
192.168.128.132 slave2 slave2.centos.com
192.168.128.133 slave3 slave3.centos.com

1. 瀏覽器查看：

http://master:50070
http://master:8088