CentOS 安装 Hadoop

环境需求

jdk 1.8

安装资源

https://hadoop.apache.org/releases.html

步骤

1 环境准备

1. ssh免密登录

   ssh-keygen -t rsa # 一路回车
   cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys	
   ssh localhost
   

2. ssh免密登录失败解决方法

   chmod 700 ~/.ssh
   chmod 600 ~/.ssh/authorized_keys
   

3. 如果还是失败，用root用户查看日志，大部分是因为文件或目录权限问题

   tail -20 /var/log/secure | grep "bad ownership"
   

2 安装Hadoop

1. 创建Hadoop程序文件夹

   mkdir /DATA/hadoop
   

2. 下载并解压

   wget https://downloads.apache.org/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
   tar -zxvf hadoop-3.2.1.tar.gz
   

3. 配置环境变量
   vim ~/.bashrc
   添加如下两行

   export HADOOP_HOME=/DATA/hadoop/hadoop-3.2.1
   export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
   

   使环境变量生效
   source ~/.bashrc

4. Hadoop配置

   vim $HADOOP_HOME/etc/hadoop/core-site.xml

   <configuration>
      <property>
          <name>fs.defaultFS</name>
          <value>hdfs://localhost:9000</value>
      </property>
   </configuration>
   

   vim $HADOOP_HOME/etc/hadoop/hdfs-site.xml

   <configuration>
       <property>
           <name>dfs.replication</name>
           <value>1</value>
       </property>
   </configuration>
   

5. 格式化HDFS（只需一次）

   hdfs namenode -format
   

3 启动Hadoop

1. 启动Hadoop

   start-dfs.sh
   

2. 查看NamNode:
   http://localhost:9870

3. 停止Hadoop

   stop-dfs.sh
   

4 配置YARN

1. vim $HADOOP_HOME/etc/hadoop/mapred-site.xml

    <configuration>
         <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
         </property>
     </configuration>
   

2. vim $HADOOP_HOME/etc/hadoop/yarn-site.xml

    <configuration>
        <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>
    </configuration>
   

5 启动YARN

1. 启动YARN

   start-yarn.sh
   

2. 查看NamNode:
   http://localhost:8088

3. 停止Hadoop

   stop-yarn.sh