記一次Hadoop配置操作記錄

最近一週都是再看大數據方面的知識,計劃着這段時間來學習大數據的知識,然後在搭建Hadoop等環境的時候,也是用了一些時間吧，感覺還是很有必要來記錄一下這方面的配置知識。

---

先安裝一太,安裝好JDK的環境,然後其他的幾臺直接克隆就可以了,就不用一個一個的安裝,要是一個一個的安裝還是比較要時間的。

export JAVA_HOME=/opt/jdk1.8.0_181

export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib

 這裏的/opt/jdk1.8.0_181 就根據你安裝JDK的路徑了。 然後克隆出幾臺服務出來。

 

配置免密登陸

ssh-keygen -t rsa

在~/.ssh 目錄下生成二個文件,id_rsa爲私鑰,id_rsa.pub爲公鑰。

修改/etc/hosts中IP和Hostname的映射

192.168.119.141 master.yang.com master
192.168.119.142 slave1.yang.com slave1
192.168.119.143 slave2.yang.com slave2

導入公鑰到認證文件中 : ssh-copy-id -i /root/.ssh/id_rsa.pub master

 

分別執行 :

ssh-copy-id -i /root/.ssh/id_rsa.pub slave1

ssh-copy-id -i /root/.ssh/id_rsa.pub slave2

這裏是根據你克隆了幾臺服務器來操作,比如我就只有二個,所以我這裏自然也就只有二個.

 

測試下看下免密登陸的配置是否成功

ssh master \ ssh slave1 \ ssh slave2

如果你沒有要密碼就可以登陸成功的話,那麼就說明你是配置成功的。

 

 Hadoop配置

上傳Hadoop hadoop-2.7.3.tar.gz 我這裏上傳的是2.7.3;對文件進行解壓

 tar -zxvf hadoop-2.7.3.tar.gz -C /usr/local

 接下來就是具體修改配置文件信息了

 

 都是在/usr/local/hadoop-2.7.3/etc/hadoop 目錄下

hadoop-env.sh

 這個文件主要是修改JDK的路徑

 export JAVA_HOME=/opt/jdk1.8.0_181

 

yarn-env.sh 也是修改JDK的路徑

這個文件也是修改JDK的指向

export JAVA_HOME=/opt/jdk1.8.0_181

 

salves如果存在就修改,不存在就自己創建一個 touch salves

master
slave1
slave2

 

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://master:8020</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/log/hadoop/tmp</value>
  </property>
</configuration>

 

 hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
       <name>dfs.namenode.name.dir</name>
       <value>file:///data/hadoop/hdfs/name</value>
    </property>
        <property>
           <name>dfs.datanode.data.dir</name>
           <value>file:///data/hadoop/hdfs/data</value>
        </property>
    <property>
           <name>dfs.namenode.secondary.http-address</name>
           <value>master:50090</value>
        </property>
    <property>
       <name>dfs.replication</name>
       <value>3</value>
    </property>
</configuration>

 

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
    </property>
       <property>
     <name>mapreduce.jobhistory.address</name>
         <value>master:10020</value>
       </property>
       <property>
         <name>mapreduce.jobhistory.webapp.address</name>
         <value>master:19888</value>    
       </property>
</configuration>

 

yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->
<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>master</value>
</property>
<property>
  <name>yarn.resourcemanager.address</name>
  <value>${yarn.resourcemanager.hostname}:8032</value>
</property>

<property>
  <name>yarn.resourcemanager.scheduler.address</name>
  <value>${yarn.resourcemanager.hostname}:8030</value>
</property>

<property>
  <name>yarn.resourcemanager.webapp.address</name>
  <value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.https.address</name>
  <value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
  <name>yarn.resourcemanager.resource-tracker.address</name>
  <value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
  <name>yarn.resourcemanager.admin.address</name>
  <value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
  <name>yarn.nodemanager.local-dirs</name>
  <value>/data/hadoop/yarn/local</value>
</property>

<property>
  <name>yarn.log-aggregation-enable</name>
  <value>true</value>
</property>

<property>
  <name>yarn.nodemanager.remote-app-log-dir</name>
  <value>/data/tmp/logs</value>
</property>

<property>
  <name>yarn.log.server.url</name>
  <value>http://master:19888/jobhistory/logs/</value>
  <desc>URL for jobhistory server</desc>
</property>

<property>
  <name>yarn.nodemanager.vmem-check-enabled</name>
  <value>false</value>
</property>

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>

<property> 
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> 
  <value>org.apache.hadoop.mapred.ShuffleHandler</value> 
</property> 
</configuration>

這個上面就基本是一臺上面的Hadoop配置完成，但是其他的幾臺不可能也是這麼配置吧;使用scp 直接copy過去就可以了

scp -r /usr/local/hadoop-2.7.3/ slave1:/usr/local/

scp -r /usr/local/hadoop-2.7.3/ slave2:/usr/local/

 這裏還是看你的服務器有多少臺來進行決定

在Master那臺上面格式化NameNode

在bin目錄下

./hdfs namenode -format

xxx has been successully formatted的話,就說明是成功了。

 

啓動在master節點上面

sbin 目錄下

./start-dfs.sh

./start-yarn.sh

./mr-jobhistory-daemon.sh start historyserver 

來進行啓動

然後你可以使用 Jps 來查詢啓動的進程

 

關閉的話就是

./stop-dfs.sh

./stop-yarn.sh

./mr-jobhistory-daemon.sh stop historyserver  

 

• http://192.168.119.141:50070/dfshealth.html#tab-overview   NameNode
• http://192.168.119.141:8088/cluster  ResourceManager
• http://192.168.119.141:19888/jobhistory  MapReduce JobHistroy Server

如果都可以訪問成功的話,並且上面的節點都有的話,那麼就說明你是配置成功了，沒有問題。