CentOS 安裝 Hadoop

環境需求

jdk 1.8

安裝資源

https://hadoop.apache.org/releases.html

步驟

1 環境準備

1. ssh免密登錄

   ssh-keygen -t rsa # 一路回車
   cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys	
   ssh localhost
   

2. ssh免密登錄失敗解決方法

   chmod 700 ~/.ssh
   chmod 600 ~/.ssh/authorized_keys
   

3. 如果還是失敗，用root用戶查看日誌，大部分是因爲文件或目錄權限問題

   tail -20 /var/log/secure | grep "bad ownership"
   

2 安裝Hadoop

1. 創建Hadoop程序文件夾

   mkdir /DATA/hadoop
   

2. 下載並解壓

   wget https://downloads.apache.org/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
   tar -zxvf hadoop-3.2.1.tar.gz
   

3. 配置環境變量
   vim ~/.bashrc
   添加如下兩行

   export HADOOP_HOME=/DATA/hadoop/hadoop-3.2.1
   export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
   

   使環境變量生效
   source ~/.bashrc

4. Hadoop配置

   vim $HADOOP_HOME/etc/hadoop/core-site.xml

   <configuration>
      <property>
          <name>fs.defaultFS</name>
          <value>hdfs://localhost:9000</value>
      </property>
   </configuration>
   

   vim $HADOOP_HOME/etc/hadoop/hdfs-site.xml

   <configuration>
       <property>
           <name>dfs.replication</name>
           <value>1</value>
       </property>
   </configuration>
   

5. 格式化HDFS（只需一次）

   hdfs namenode -format
   

3 啓動Hadoop

1. 啓動Hadoop

   start-dfs.sh
   

2. 查看NamNode:
   http://localhost:9870

3. 停止Hadoop

   stop-dfs.sh
   

4 配置YARN

1. vim $HADOOP_HOME/etc/hadoop/mapred-site.xml

    <configuration>
         <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
         </property>
     </configuration>
   

2. vim $HADOOP_HOME/etc/hadoop/yarn-site.xml

    <configuration>
        <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>
    </configuration>
   

5 啓動YARN

1. 啓動YARN

   start-yarn.sh
   

2. 查看NamNode:
   http://localhost:8088

3. 停止Hadoop

   stop-yarn.sh