Hadoop集羣搭建（二）-- 集羣搭建（部分）

一、基礎信息

1.集羣節點地址分配

主機名	ip
master	192.168.100.200
slave1	192.168.100.201
slave2	192.168.100.202

2.軟件使用版本

軟件	版本
jdk	jdk1.8.0_172
hadoop	hadoop-2.7.6

3.軟件安裝路徑

文件名	路徑
jdk1.8.0_172	/opt/app/Java/jdk1.8.0_172
hadoop-2.7.6	/opt/app/Hadoop/hadoop-2.7.6

二、基本軟件安裝

最小化安裝需要手動安裝基本的操作軟件

==可以直接在shell窗口進行全部會話操作==

1.關閉防火牆

#關閉防火牆
[root@master ~]# systemctl stop firewalld
#禁止開機啓動防火牆
[root@master ~]# systemctl disable firewalld

擴展

#查看防火牆狀態
[root@master ~]# systemctl status firewalld
#開啓防火牆
[root@master ~]# systemctl start firewalld
#設置開機啓動防火牆
[root@master ~]# systemctl enable firewalld
#重啓防火牆
[root@master ~]# systemctl restart firewalld

2.openssh安裝

遠程上傳文件

[root@master ~]# yum -y install openssh-clients

3.ntpdate安裝

同步時間

#安裝ntpdate
[root@master ~]# yum -y install ntp ntpdate
#同步網絡時間
[root@master ~]# ntpdate cn.pool.ntp.org
#系統時間寫入硬件時間
[root@master ~]# hwclock --systohc

4.lrzsz安裝

文件上傳（rz）下載（sz），在Xshell中通過rz調出上傳文件的窗口，sz文件名下載某一個文件

[root@master ~]# yum -y install lrzsz
#上傳
[root@master ~]# rz

5.vim安裝

系統自帶vi編輯器，要使用vim編輯器需要下載

[root@master ~]# yum -y install vim

三、配置SSH免密登錄

1.修改hosts文件

[root@master ~]# vim /etc/hosts

#刪除或註釋
#127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1       localhost localhost.localdomain localhost6 localhost6.localdomain6

#添加地址主機映射
192.168.100.200 master
192.168.100.201 slave1
192.168.100.202 slave2

2.給每個節點生成.ssh

#生成.ssh文件夾，然後選擇no或者直接Ctrl+C退出
[root@master ~]# ssh master
#查看是否生成.ssh文件夾
[root@master ~]# ll -a
總用量 36
dr-xr-x---.  4 root root  188 7月  28 14:38 .
dr-xr-xr-x. 17 root root  224 7月  27 18:59 ..
-rw-------.  1 root root 1339 7月  27 19:01 anaconda-ks.cfg
-rw-------.  1 root root 2045 7月  27 23:24 .bash_history
-rw-r--r--.  1 root root   18 12月 29 2013 .bash_logout
-rw-r--r--.  1 root root  176 12月 29 2013 .bash_profile
-rw-r--r--.  1 root root  176 12月 29 2013 .bashrc
-rw-r--r--.  1 root root  100 12月 29 2013 .cshrc
drwxr-xr-x.  2 root root   40 7月  27 20:47 .oracle_jre_usage
drwx------.  2 root root   80 7月  27 20:23 .ssh
-rw-r--r--.  1 root root  129 12月 29 2013 .tcshrc
-rw-------.  1 root root 4857 7月  28 14:38 .viminfo

3.給每個節點生成公鑰私鑰

#進入.ssh目錄進行操作
[root@master ~]# cd ./.ssh
#生成公鑰和私鑰
[root@master .ssh]# ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
#回車
Enter file in which to save the key (/root/.ssh/id_rsa): 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:sl6sgAPPdHe2Ky4ocjkzUd0Sw3tKWcUoPc3CacCSMkI root@master
The key's randomart image is:
+---[RSA 2048]----+
|.E   +.+ B.      |
|. o o * X +      |
| . o o X o       |
|    . * o        |
|. ....o+S        |
| =.o ..* .       |
|  =+. . +        |
|o B..o.o .       |
|.o + o+..        |
+----[SHA256]-----+
#查看生成的公鑰和私鑰
[root@master .ssh]# ls
id_rsa  id_rsa.pub

4.選擇某一個節點操作

#將.ssh文件下的公鑰id_rsa.pub內容複製到authorized_keys文件中
[root@master .ssh]# cp id_rsa.pub authorized_keys 
#查看
[root@master .ssh]# ls
authorized_keys  id_rsa  id_rsa.pub

5.操作其它節點

#將其它節點的公鑰id_rsa.pub也複製到此authorized_keys文件中

#slave1
[root@slave1 .ssh]# cat ~/.ssh/id_rsa.pub | ssh root@master 'cat >> ~/.ssh/authorized_keys'

The authenticity of host 'master (192.168.100.200)' can't be established.
ECDSA key fingerprint is SHA256:O2tZj1XfJ6upjXbjTa2YATYJgXVjYcMwVa3APH5+6qg.
ECDSA key fingerprint is MD5:66:3c:28:31:6d:a9:d7:60:1f:be:7d:2d:63:89:36:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,192.168.100.200' (ECDSA) to the list of known hosts.
root@master's password: 

#slave2
[root@slave2 .ssh]# cat ~/.ssh/id_rsa.pub | ssh root@master 'cat >> ~/.ssh/authorized_keys'

The authenticity of host 'master (192.168.100.200)' can't be established.
ECDSA key fingerprint is SHA256:O2tZj1XfJ6upjXbjTa2YATYJgXVjYcMwVa3APH5+6qg.
ECDSA key fingerprint is MD5:66:3c:28:31:6d:a9:d7:60:1f:be:7d:2d:63:89:36:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,192.168.100.200' (ECDSA) to the list of known hosts.
root@master's password: 

6.authorized_keys文件信息

#查看authorized_keys文件
[root@master .ssh]# more authorized_keys 

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNg57gsLDFUfBJEHF/KRTMCRObUPEaeqvZu9fR6c2Z
hBuaJcGiz9A5Z1eQpyEEw3udFRCjrKk7TehnJ1PajnjaT26OXJr2FebtUF7l1mPGVHExgYPj6xwFivG8
Zta8JECzrxY/CywrTasFq3N3gTD7x/9tW3Xrz0jsd6EeKUJdeDGiY8d7jw0IWsGTjmjjq+BGve4newAa
s9QnTAtp2APga1MSDSOLVYGLGkUOYa9ohAePQi6puC7RnOT5d/bVrqcoH6rBT/t9dGQOggeqyY3WYPTT
CfpN6yCbvMMi56mOtg/XEcEfdbJYsvSHHLGbao4gEGWeDaf3FT6J/OrW2UF1 root@master

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDWB8cAg7WQrvZIYMrISnDSKemiQb9U7mcakz1AF1+A
5FENrZ1yRBJ0cHlfhKGLzGqzXqi8l6pwtk3V7B3YGSGNNIxhPYLvstVspWrnUStonBy1OLOjDvVL3dLk
+UtERR4HaqsTJk7Px1B4Bu6qpXW19NJjVryObPzeuE7e/Kfl2IfnFn47HKgNNVrPVwmjXPzndUSbVYcB
nwv3C+iB6otPQYG6koW8TaI/hctXxPGJ91bhaW4MgM7iSigOiXoZA3TX07QD1RZnAi2WK2P7Ds+qToP2
NgmezuQFdmR82RcsNmYQFxphwrFYNba4BP9cDwYYbg4P6KDXiBik/hB6O5+d root@slave2

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYIcB2/DaCl+971eENOBOVZIw74FxemGFPFgVunw
ZZL1RQF9bybzRbUpU3fZvY6DcaRlq8KtD9ClN0ol1bUBJVuWsT8rS95kw5GQyhWwui5b+JFs80qFC8yM
AYzlYUx5Ok2ofYkz88IeUblLmdigOQNCpQg/hG7Y0GGEC0bNaVmScvqWiE3+O3YGqtD4aXu5XY9IblRt
RMNXupGCwwYI57VgXP6gFFWWePcaEcyZWGT/XNfrumz8cABxq3J/prfSB5i6Gipo+13Q+hMGNzzzGhnp
tXhDoAQKiCgsPNNVkLPxiJliQvtIEaplB31r5oTW2WyWMSphY/kc/50y3Oq3 root@slave1

7.將此authorized_keys文件共享給其它節點

[root@master .ssh]# scp ~/.ssh/authorized_keys root@slave1:~/.ssh/
The authenticity of host 'slave1 (192.168.100.201)' can't be established.
ECDSA key fingerprint is SHA256:8YTgPFbLIp322KNxWivNLz5LxmytmsF/pBt/FADTHvk.
ECDSA key fingerprint is MD5:72:5c:57:e0:75:75:83:4f:8d:90:ea:52:64:f1:fa:7b.
Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'slave1,192.168.100.201' (ECDSA) to the list of known hosts.
root@slave1's password: 
authorized_keys                               100% 1179   434.4KB/s   00:00    

[root@master .ssh]# scp ~/.ssh/authorized_keys root@slave2:~/.ssh/
The authenticity of host 'slave2 (192.168.100.202)' can't be established.
ECDSA key fingerprint is SHA256:futhvXj6JkULXb9xcO54CZuA60xcWvKaY3NZ5SAu4Vk.
ECDSA key fingerprint is MD5:1f:40:66:17:5a:4e:b0:5d:ba:8c:78:53:be:75:1f:03.
Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'slave2,192.168.100.202' (ECDSA) to the list of known hosts.
root@slave2's password: 
authorized_keys                               100% 1179   381.6KB/s   00:00    

8.在每個節點之間進行ssh免密登錄操作，包括自己與自己

[root@master .ssh]# ssh master
The authenticity of host 'master (192.168.100.200)' can't be established.
ECDSA key fingerprint is SHA256:O2tZj1XfJ6upjXbjTa2YATYJgXVjYcMwVa3APH5+6qg.
ECDSA key fingerprint is MD5:66:3c:28:31:6d:a9:d7:60:1f:be:7d:2d:63:89:36:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,192.168.100.200' (ECDSA) to the list of known hosts.
Last login: Sat Jul 28 14:38:11 2018 from master

[root@master ~]# ssh slave1
Last login: Sat Jul 28 08:53:58 2018 from 192.168.100.1

[root@slave1 ~]# ssh slave1
The authenticity of host 'slave1 (192.168.100.201)' can't be established.
ECDSA key fingerprint is SHA256:8YTgPFbLIp322KNxWivNLz5LxmytmsF/pBt/FADTHvk.
ECDSA key fingerprint is MD5:72:5c:57:e0:75:75:83:4f:8d:90:ea:52:64:f1:fa:7b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave1,192.168.100.201' (ECDSA) to the list of known hosts.
Last login: Sat Jul 28 15:40:24 2018 from master

[root@slave1 ~]# ssh slave2
The authenticity of host 'slave2 (192.168.100.202)' can't be established.
ECDSA key fingerprint is SHA256:futhvXj6JkULXb9xcO54CZuA60xcWvKaY3NZ5SAu4Vk.
ECDSA key fingerprint is MD5:1f:40:66:17:5a:4e:b0:5d:ba:8c:78:53:be:75:1f:03.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave2,192.168.100.202' (ECDSA) to the list of known hosts.
Last login: Sat Jul 28 08:53:59 2018 from 192.168.100.1

[root@slave2 ~]# ssh slave2
The authenticity of host 'slave2 (192.168.100.202)' can't be established.
ECDSA key fingerprint is SHA256:futhvXj6JkULXb9xcO54CZuA60xcWvKaY3NZ5SAu4Vk.
ECDSA key fingerprint is MD5:1f:40:66:17:5a:4e:b0:5d:ba:8c:78:53:be:75:1f:03.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave2,192.168.100.202' (ECDSA) to the list of known hosts.
Last login: Sat Jul 28 15:41:00 2018 from slave1

[root@slave2 ~]# ssh slave1
The authenticity of host 'slave1 (192.168.100.201)' can't be established.
ECDSA key fingerprint is SHA256:8YTgPFbLIp322KNxWivNLz5LxmytmsF/pBt/FADTHvk.
ECDSA key fingerprint is MD5:72:5c:57:e0:75:75:83:4f:8d:90:ea:52:64:f1:fa:7b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave1,192.168.100.201' (ECDSA) to the list of known hosts.
Last login: Sat Jul 28 15:40:38 2018 from slave1

[root@slave1 ~]# ssh master
Last login: Sat Jul 28 15:40:01 2018 from master

[root@master ~]# ssh slave2
Last login: Sat Jul 28 15:41:11 2018 from slave2

[root@slave2 ~]# ssh master
Last login: Sat Jul 28 15:41:57 2018 from slave1

四、安裝配置java

選某一個節點操作，然後複製到其它節點

#創建指定目錄
[root@master ~]# mkdir ‐p /opt/app/Java

1.安裝jdk

#進入到Java目錄
[root@master ~]# cd /opt/app/Java
#rz上傳jdk壓縮包
[root@master Java]# rz
#查看
[root@master Java]# ls
jdk-8u172-linux-x64.tar.gz
#解壓到當前目錄
[root@master Java]# tar ‐zxvf jdk-8u172-linux-x64.tar.gz
#查看
[root@master Java]# ls
jdk1.8.0_172  jdk-8u172-linux-x64.tar.gz

2.配置jdk環境變量

#進入系統配置文件
[root@master Java]# vim /etc/profile

#java環境變量
export JAVA_HOME=/opt/app/Java/jdk1.8.0_172
export JRE_HOME=/opt/app/Java/jdk1.8.0_172/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib/rt.jar
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

#使設置立即生效
[root@master Java]# . /etc/profile
# 或者 source /etc/profile

#檢測配置是否成功
[root@master Java]# java -version
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)

[root@master Java]# javac -version
javac 1.8.0_172

五、安裝配置hadoop

#創建指定目錄
[root@master ~]# mkdir ‐p /opt/app/Hadoop

1.安裝hadoop

#進入到Hadoop目錄
[root@master ~]# cd /opt/app/Hadoop
#rz上傳jdk壓縮包
[root@master Hadoop]# rz
#查看
[root@master Hadoop]# ls
hadoop-2.7.6.tar.gz
#解壓到當前目錄
[root@master Hadoop]# tar ‐zxvf hadoop-2.7.6.tar.gz
#查看
[root@master Hadoop]# ls
hadoop-2.7.6  hadoop-2.7.6.tar.gz

2.創建備用目錄

#進入hadoop‐2.7.6目錄
[root@master Hadoop]# cd hadoop‐2.7.6
#創建臨時文件目錄
[root@master hadoop‐2.7.6]# mkdir tmp
#創建日誌信息目錄
[root@master hadoop‐2.7.6]# mkdir logs
#創建hdfs名稱空間
[root@master hadoop‐2.7.6]# mkdir ‐p hdfs/name
#創建hdfs數據空間
[root@master hadoop‐2.7.6]# mkdir ‐p hdfs/data

#查看
[root@master hadoop-2.7.6]# ls
bin  hdfs     lib      LICENSE.txt  NOTICE.txt  sbin   tmp
etc  include  libexec  logs         README.txt  share

3.修改配置文件

編輯slaves文件

#該文件記錄的是集羣裏的數據節點
[root@master hadoop-2.7.6]# vi etc/hadoop/slaves

#刪除原來內容，添加以下內容
slave1
slave2

編輯hadoop-env.sh文件

[root@master hadoop-2.7.6]# vim etc/hadoop/hadoop-env.sh 

#修改JAVA_HOME爲絕對路徑
export JAVA_HOME=/opt/app/Java/jdk1.8.0_172

編輯yarn-env.sh文件

[root@master hadoop-2.7.6]# vim etc/hadoop/yarn-env.sh 

#修改JAVA_HOME爲絕對路徑
export JAVA_HOME=/opt/app/Java/jdk1.8.0_172

編輯core-site.xml文件

[root@master hadoop-2.7.6]# vim etc/hadoop/core-site.xml 

<configuration>
    <property>
            <name>fs.defaultFS</name>
            <value>hdfs://master:9000</value>
    </property>
    <property>
            <name>hadoop.tmp.dir</name>
            <value>file:/opt/app/Hadoop/hadoop-2.7.6/tmp</value>
    </property>
    <property>
            <name>io.file.buffer.size</name>
            <value>131702</value>
    </property>
</configuration>

編輯hdfs-site.xml文件

[root@master hadoop-2.7.6]# vim etc/hadoop/hdfs-site.xml

<configuration>
    <property>
            <name>dfs.namenode.name.dir</name>
            <value>file:/opt/app/Hadoop/hadoop-2.7.6/hdfs/name</value>
    </property>
    <property>
            <name>dfs.datanode.data.dir</name>
            <value>file:/opt/app/Hadoop/hadoop-2.7.6/hdfs/data</value>
    </property>
    <property>
            <name>dfs.replication</name>
            <value>2</value>
    </property>
    <property>
            <name>dfs.namenode.secondary.https-address</name>
            <value>master:50070</value>
    </property>
    <property>
            <name>dfs.webhdfs.enabled</name>
            <value>true</value>
    </property>
</configuration>

編輯yarn-site.xml文件

[root@master hadoop-2.7.6]# vim etc/hadoop/yarn-site.xml 

<configuration>
    <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
    </property>
    <property>
            <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
            <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
            <name>yarn.resourcemanager.address</name>
            <value>master:8032</value>
    </property>
    <property>
            <name>yarn.resourcemanager.scheduler.address</name>
            <value>master:8030</value>
    </property>
    <property>
            <name>yarn.resourcemanager.resource-tracker.address</name>
            <value>master:8031</value>
    </property>
    <property>
            <name>yarn.resourcemanager.admin.address</name>
            <value>master:8033</value>
    </property>
    <property>
            <name>yarn.resourcemanager.webapp.address</name>
            <value>master:8088</value>
    </property>
    <property>
            <name>yarn.nodemanager.resource.memory-mb</name>
            <value>1024</value>
    </property>
</configuration>

編輯mapred-site.xml文件

[root@master hadoop-2.7.6]# vim etc/hadoop/mapred-site.xml

<configuration>
    <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
    </property>
    <property>
            <name>mapreduce.jobhistory.address</name>
            <value>master:10020</value>
    </property>
    <property>
            <name>mapreduce.jobhistory.webapp.address</name>
            <value>master:19888</value>
    </property>
</configuration>

4.配置hadoop環境變量

[root@master hadoop-2.7.6]# vim /etc/profile

#hadoop環境變量
export HADOOP_HOME=/opt/app/Hadoop/hadoop-2.7.6
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
export YARN_LOG_DIR=$HADOOP_LOG_DIR
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

#使設置立即生效
[root@master hadoop-2.7.6]# . /etc/profile

六、文件遠程拷貝

[root@master hadoop‐2.7.6]# scp ‐r /opt/app root@slave1:/opt/
[root@master hadoop‐2.7.6]# scp ‐r /opt/app root@slave2:/opt/

[root@master hadoop‐2.7.6]# scp ‐r /etc/profile root@slave1:/etc/
[root@master hadoop‐2.7.6]# scp ‐r /etc/profile root@slave2:/etc/

七、格式化Hadoop

[root@master hadoop‐2.7.6]# bin/hdfs namenode ‐format

八、啓動Hadoop集羣

不建議直接使用start‐all.sh命令

#先啓動HDFS
[root@master hadoop‐2.7.6]# sbin/start‐dfs.sh

#然後啓動YARN
[root@master hadoop‐2.7.6]# sbin/start‐yarn.sh

九、檢測集羣是否搭建成功

[root@master hadoop-2.7.6]# jps
16277 ResourceManager
16346 Jps
16124 SecondaryNameNode
15933 NameNode
[root@slave1 hadoop-2.7.6]# jps
11105 Jps
10904 DataNode
11004 NodeManager
[root@slave2 hadoop-2.7.6]# jps
11107 Jps
10906 DataNode
11006 NodeManager

Web登錄訪問：下面的圖是再次搭建的時候截的，僅僅登錄地址不一樣而已
192.168.100.200:50070

192.168.100.200:8088