使用虛擬機(MobaXterm)從零開始搭建Spark1.3集羣

原創 小兴安岭mx 2019-03-20 05:47

準備工作

1. Hadoop：hadoop-2.4.1.tar.gz

2. Spark：下載編譯好的基於對應hadoop版本的版本：spark-1.3.0-bin-hadoop2.4.tgz

3.JAVA：jdk-7u80-linux-x64.tar.gz

4. scala：scala-2.11.8.tgz5

 

搭建環境

在MobaXterm界面連接服務器，使用命令sudo virt-manager打開virtual machine manager窗口，使用ubuntu16.04鏡像創建一個新的虛擬機，進行如下操作

  • 設置靜態ip地址      
    sudo vim /etc/network/interfaces

## 更改ip地址，將#iface ens3 inet dhcp改爲如下內容
# The primary network interface
auto ens3
iface ens3 inet static
address 192.168.122.54
netmask 255.255.255.0
gateway 192.168.122.1

## 重啓網絡
/etc/init.d/networking restart

## 注意：clone機器後仍然需要修改，我設置的靜態ip如下
master 192.168.122.54
slave1 192.168.122.55
slave2 192.168.122.56
slave3 192.168.122.57
slave4 192.168.122.58

     

  • 配置hosts文件 
    ## 修改主機名
sudo vim /etc/hostname 
## 改爲master，clone之後其他再統一修改

## 修改hosts文件，只保留localhost，剩餘內容進行追加(否則會出問題，後面會提到！)
sudo vim/etc/hosts

192.168.122.54 master
192.168.122.55 slave1
192.168.122.56 slave2 
192.168.122.57 slave3 
192.168.122.58 slave4

     

  • 搭建Java環境 
    ## 解壓
sudo tar -zxvf jdk-7u80-linux-x64.tar.gz -C ./software/
sudo mv jdkxxxx jdk

## 配置環境變量
sudo vim /etc/profile

export JAVA_HOME=/home/zmx/software/jdk
export JRE_HOME=/home/zmx/software/jdk/jre
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

     

  • 搭建scala環境 
    ## 解壓
sudo tar -zxvf scala-2.11.8.tgz -C software/
sudo mv scala-2.11.8/ scala

## 追加環境變量
sudo vim /etc/profile

## 最終效果如圖
export JAVA_HOME=/home/zmx/softwarek
export JRE_HOME=/home/zmx/softwarek/jre
export SCALA_HOME=/home/zmx/software/scala
export PATH=$SCALA_HOME/bin:$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
export CLASSPATH=$CLASSPATH:.:$JAVA_HOMEb:$JAVA_HOME/jreb

     

  • 關閉防火牆 
    sudo ufw disable
Firewall stopped and disabled on system startup

     

  • clone虛擬機：在MobaXterm界面輸入sudo virt-manager打開vm manager後進行clone

       

  • 搭建集羣，測試機器之間能否通信 
    ## clone機器後記得進行ip地址和host的更改,這樣我們一共有五臺機器：master、slave1-slave4
## master上進行如下操作，其他機器同理
ping slave1
ping slave2
ping slave3
ping slave4

     

  • 配置master-slave ssh 免密登陸 
    ## 在每臺機器上生成私鑰和公鑰
ssh-keygen -t rsa

## 將slave上的id_rsa.pub用scp命令發給master
scp ./.ssh/id_rsa.pub zmx@master:~/.ssh/id_rsa.pub.slave1
scp ./.ssh/id_rsa.pub zmx@master:~/.ssh/id_rsa.pub.slave2
scp ./.ssh/id_rsa.pub zmx@master:~/.ssh/id_rsa.pub.slave3
scp ./.ssh/id_rsa.pub zmx@master:~/.ssh/id_rsa.pub.slave4

## 在master上，將所有公鑰加到用於認證的公鑰文件authorized_keys中
zmx@master:~$ cat .ssh/id_rsa.pub* >> ~/.ssh/authorized_keys

## 將公鑰文件分發給slaves
scp .ssh/authorized_keys zmx@slave1:~/.ssh/
scp .ssh/authorized_keys zmx@slave2:~/.ssh/
scp .ssh/authorized_keys zmx@slave3:~/.ssh/
scp .ssh/authorized_keys zmx@slave4:~/.ssh/

## 最後在每臺主機上，用SSH命令，檢驗下是否能免密碼登錄
ssh slave1
ssh slave2
ssh slave3
ssh slave4

安裝Hadoop

  配置環境

  • 配置文件
    • hadoop-env.sh 
      ## 末尾增加
export HADOOP_IDENT_STRING=$USER
export JAVA_HOME=/home/zmx/software/jdk
export HADOOP_PREFIX=/home/zmx/software/hadoop-2.4.1

       

    • yarn-evn.sh 
      ## 末尾增加
export JAVA_HOME=/home/zmx/software/jdk

       

    • slaves: 加入master表示將master也視爲slave 
      master
slave1
slave2
slave3
slave4

       

    • core-site.xml 
      <?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://master:9000</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/home/zmx/software/hadoop-2.4.1/tmp</value>
  </property>
</configuration>

       

    • hdfs-site.xml 
      <?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>dfs.datanode.ipc.address</name>
    <value>0.0.0.0:50020</value>
  </property>
  <property>
    <name>dfs.datanode.http.address</name>
    <value>0.0.0.0:50075</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>
</configuration>

       

    • mapred-site.xml 
      <?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

       

    • yarn-site.xml 
      <?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>

  <property>  
    <name>yarn.resourcemanager.address</name>  
    <value>master:8032</value>  
  </property> 
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>  
    <value>master:8030</value>  
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>  
    <value>master:8031</value>  
  </property>
</configuration>

       

  • 分發文件夾給slave: 
    sudo chmod -R 777 ~/software/hadoop-2.4.1
scp -r ~/software/hadoop-2.4.1 zmx@slave1:~/software/
scp -r ~/software/hadoop-2.4.1 zmx@slave2:~/software/
scp -r ~/software/hadoop-2.4.1 zmx@slave3:~/software/
scp -r ~/software/hadoop-2.4.1 zmx@slave4:~/software/

     

  • 啓動hadoop 
    ## 進入hadoop目錄，格式化hdfs
bin/hdfs namenode -format

## 啓動hdfs
sbin/start-dfs.sh

## 啓動yarn
sbin/start-yarn.sh

     

  • 用jps命令查看hadoop進程 
    ## master上
ResourceManager
SecondaryNameNode
NameNode
DataNode(因爲把master看成自己的slave所以存在該進程，如果不想這樣，將hadoop的配置文件slaves中的master去掉即可)
NodeManager(因爲把master看成自己的slave所以存在該進程)

## slave上
DataNode 
NodeManager

     

  • 輸入yarn node -list查看節點信息 
    zmx@master:~/software/hadoop-2.4.1$ yarn node -list
19/03/18 10:01:32 INFO client.RMProxy: Connecting to ResourceManager at /192.168.122.54:8032
19/03/18 10:01:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Total Nodes:5
         Node-Id             Node-State Node-Http-Address       Number-of-Running-Containers
    slave4:36952                RUNNING       slave4:8042                                  0
    slave2:39254                RUNNING       slave2:8042                                  0
    master:38718                RUNNING       master:8042                                  0
    slave1:42168                RUNNING       slave1:8042                                  0
    slave3:43401                RUNNING       slave3:8042

     

 

安裝Spark

  • 更改配置文件
    • spark-env.sh：配置幾個基本的，可以根據自己機器的實際情況再進行配置 
      ​## 添加代碼到末尾
export SCALA_HOME=/home/zmx/software/scala
export JAVA_HOME=/home/zmx/software/jdk
export HADOOP_HOME=/home/zmx/software/hadoop-2.4.1
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
SPARK_MASTER_IP=master
SPARK_LOCAL_DIRS=/home/zmx/software/spark-1.3.0-bin-hadoop2.4
SPARK_DRIVER_MEMORY=512M

       

    • slaves：加上master表示將master也視爲worker，可以不加

      master
slave1
slave2
slave3
slave4

       

  • 分發給slave 
    scp -r ~/software/spark-1.3.0-bin-hadoop2.4 zmx@slave1:~/software/
scp -r ~/software/spark-1.3.0-bin-hadoop2.4 zmx@slave2:~/software/
scp -r ~/software/spark-1.3.0-bin-hadoop2.4 zmx@slave3:~/software/
scp -r ~/software/spark-1.3.0-bin-hadoop2.4 zmx@slave4:~/software/

     

  • 啓動Spark 
    sbin/start-all.sh

##啓動成功運行jps可以看到
master節點進程
Master

worker節點進程
Worker

     

使用yarn-cluster模式測試Hadoop和Spark是否安裝成功

   參考鏈接：通過跑較大數據集測試Hadoop 2.4.1是否安裝成功

zmx@master:~/software/spark-1.3.0-bin-hadoop2.4$ ./bin/spark-submit \
> --class org.apache.spark.examples.JavaWordCount \
> --master yarn-cluster \
> lib/spark-examples*.jar \
> input/testWordCountFile.txt
Spark assembly has been built with Hive, including Datanucleus jars on classpath
19/03/18 21:04:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/03/18 21:04:06 INFO RMProxy: Connecting to ResourceManager at /192.168.122.54:8032
19/03/18 21:04:06 INFO Client: Requesting a new application from cluster with 5 NodeManagers
19/03/18 21:04:06 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
19/03/18 21:04:06 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
19/03/18 21:04:06 INFO Client: Setting up container launch context for our AM
19/03/18 21:04:06 INFO Client: Preparing resources for our AM container
19/03/18 21:04:07 INFO Client: Uploading resource file:/home/zmx/software/spark-1.3.0-bin-hadoop2.4/lib/spark-assembly-1.3.0-hadoop2.4.0.jar -> hdfs://master:9000/user/zmx/.sparkStaging/application_1552885048834_0014/spark-assembly-1.3.0-hadoop2.4.0.jar
19/03/18 21:04:10 INFO Client: Uploading resource file:/home/zmx/software/spark-1.3.0-bin-hadoop2.4/lib/spark-examples-1.3.0-hadoop2.4.0.jar -> hdfs://master:9000/user/zmx/.sparkStaging/application_1552885048834_0014/spark-examples-1.3.0-hadoop2.4.0.jar
19/03/18 21:04:12 INFO Client: Setting up the launch environment for our AM container
19/03/18 21:04:12 INFO SecurityManager: Changing view acls to: zmx
19/03/18 21:04:12 INFO SecurityManager: Changing modify acls to: zmx
19/03/18 21:04:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(zmx); users with modify permissions: Set(zmx)
19/03/18 21:04:12 INFO Client: Submitting application 14 to ResourceManager
19/03/18 21:04:12 INFO YarnClientImpl: Submitted application application_1552885048834_0014
19/03/18 21:04:13 INFO Client: Application report for application_1552885048834_0014 (state: ACCEPTED)
19/03/18 21:04:13 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1552914252846
         final status: UNDEFINED
         tracking URL: http://master:8088/proxy/application_1552885048834_0014/
         user: zmx
19/03/18 21:04:14 INFO Client: Application report for application_1552885048834_0014 (state: ACCEPTED)
19/03/18 21:04:15 INFO Client: Application report for application_1552885048834_0014 (state: ACCEPTED)
19/03/18 21:04:16 INFO Client: Application report for application_1552885048834_0014 (state: ACCEPTED)
19/03/18 21:04:17 INFO Client: Application report for application_1552885048834_0014 (state: ACCEPTED)
19/03/18 21:04:18 INFO Client: Application report for application_1552885048834_0014 (state: ACCEPTED)
19/03/18 21:04:19 INFO Client: Application report for application_1552885048834_0014 (state: ACCEPTED)
19/03/18 21:04:20 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:20 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: slave4
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1552914252846
         final status: UNDEFINED
         tracking URL: http://master:8088/proxy/application_1552885048834_0014/
         user: zmx
19/03/18 21:04:21 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:22 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:23 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:24 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:25 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:26 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:27 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:28 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:29 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:30 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:31 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:32 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:33 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:34 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:35 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:37 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:38 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:39 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:40 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:41 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:42 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:43 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:44 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:45 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:46 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:47 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:48 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:49 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:50 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:51 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:52 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:53 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:54 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:55 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:56 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:57 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:58 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:59 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:00 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:01 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:02 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:03 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:04 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:05 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:06 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:07 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:08 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:09 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:10 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:11 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:12 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:13 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:14 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:15 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:16 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:17 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:18 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:19 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:20 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:21 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:22 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:23 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:24 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:25 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:26 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:27 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:28 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:29 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:30 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:31 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:32 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:33 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:34 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:35 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:36 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:37 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:38 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:39 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:40 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:41 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:42 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:43 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:44 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:45 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:46 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:47 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:48 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:49 INFO Client: Application report for application_1552885048834_0014 (state: FINISHED)
19/03/18 21:05:49 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: slave4
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1552914252846
         final status: SUCCEEDED
         tracking URL: http://master:8088/proxy/application_1552885048834_0014/A
         user: zmx

遇到的問題及解決方案

1. Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS 

錯誤原因：yarn-site.xml配置錯誤，修改所有節點上yare-site.xml配置文件，在該文件中配置ResourceManager Master節點所在地址即可解決問題

<property>  
    <name>yarn.resourcemanager.address</name>  
    <value>master:8032</value>  
</property> 
<property>
    <name>yarn.resourcemanager.scheduler.address</name>  
    <value>master:8030</value>  
</property>
<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>  
    <value>mster:8031</value>
</property>

2. hadoop啓動異常，日誌如下

java.lang.IllegalArgumentException: Does not contain a valid host:port authority:  master:8031 (configuration property 'yarn.resourcemanager.resource-tracker.address')
	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:196)
	at org.apache.hadoop.conf.Configuration.getSocketAddr(Configuration.java:1590)
	at org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.serviceInit(ResourceTrackerService.java:106)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:288)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:871)
2014-03-20 21:00:20,545 INFO org.apache.hadoop.service.AbstractService: Service ResourceManager failed in state INITED; cause: java.lang.IllegalArgumentException: Does not contain a valid host:port authority:  master:8031 (configuration property 'yarn.resourcemanager.resource-tracker.address')
java.lang.IllegalArgumentException: Does not contain a valid host:port authority:  master:8031 (configuration property 'yarn.resourcemanager.resource-tracker.address')
	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:196)
	at org.apache.hadoop.conf.Configuration.getSocketAddr(Configuration.java:1590)
	at org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.serviceInit(ResourceTrackerService.java:106)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:288)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:871)

yarn-site.xml配置錯誤，<value>標籤之間不能有空格

 

3. hadoop啓動異常，INFO org.apache.hadoop.ipc.Client: Retrying connect to server: maste/192.168.122.54:8031. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

問題：/etc/hosts文件多了條記錄

127.0.1.1 master

$ netstat -apn|grep 8031
tcp 0 0 127.0.1.1:8031 0.0.0.0:* LISTEN 4964/java

檢查端口8031時發現NameNode進程只監聽了127.0.1.1:8031端口，除了自己以外其他主機都不連接，所以導致出錯，刪除所有節點上hosts文件的該記錄：127.0.1.1 xxx(xxx爲自己的host)，重啓namenode即可

其他端口出現同樣問題，解決方法類似。

 

4. hadoop hdfs格式化遇到問題

FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:9000. Exiting. 
java.io.IOException: Incompatible clusterIDs in /home/lxh/hadoop/hdfs/data: namenode clusterID = CID-a3938a0b-57b5-458d-841c-d096e2b7a71c; datanode clusterID = CID-200e6206-98b5-44b2-9e48-262871884eeb
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:477)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:226)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:254)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:974)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:945)
    at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:278)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:220)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)
    at java.lang.Thread.run(Thread.java:745)

問題：命令 bin/hdfs namenode -format執行一次就行了，否則就會出現上述問題

解決方法：如果沒有多次format namenode仍出現上述問題，參照鏈接 https://blog.csdn.net/liuxinghao/article/details/40121843

如果之前多次執行了format命令，則在記錄hadoop配置之後，刪除所有節點的hadoop文件夾重新在master配置並分發 

 

5. hadoop啓動時出現permission denied問題：sbin/start-dfs.sh

   原因：權限不夠，執行chmod命令後再進行hadoop文件夾的分發

sudo chmod -R 777 ~/software/hadoop-2.4.1

  

參考鏈接

  1. 使用虛擬機從小白開始搭建Spark集羣
  2. Hadoop-2.4.1完全分佈式環境搭建
  3. hadoop 2.6全分佈安裝
  4. ubuntu關閉防火牆
  5. Hadoop集羣從節點出現錯誤： Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 0 time(s);
  6. hadoop下啓動異常之一
  7. Hadoop 集羣安裝及配置實戰
  8. hadoop2.x.x格式化遇到的問題
  9. 重新format namenode後，datanode無法正常啓動
  10. Hadoop2.4.1中wordcount示例程序測試過程
  11. Spark On YARN 集羣安裝部署
  12. SparkDoc：Dynamic Resource Allocation
  13. Spark Dynamic Allocation 分析
  14. spark入門筆記
發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章

在spark，MapReduce 或 Flink 程序裏面制定環境變量

以jdk來舉例，當我們需要 更高版本的jdk時，而默認的環境引用的是低版本的jdk，會引發如下錯誤： Exception in thread "main" java.lang.UnsupportedClassVersionError:

梦里却知是客 2020-07-08 11:59:58

Spark系列：Spark學習筆記

Spark 閱讀官方文檔 Spark Quick Start Spark Programming Guide Spark SQL, DataFrames and Datasets Guide Cluster Mode Overv

路越 2020-07-08 12:17:40

Spark 線性迴歸模型異常

java.lang.AssertionError: assertion failed: lapack.dppsv returned 9. at scala.Predef$.assert(Predef.scala:179)

路越 2020-07-08 12:17:29

spark日常報錯問題-持續性更新

1：spark運行過程中出現與driver鏈接異常，並存在磁盤讀寫一場: java.io.IOException: Failed to delete: /mnt/sd04/yarn/nm/usercache/hdfs/appcache/a

梦里却知是客 2020-07-08 11:59:58

大數據入門（五）windows上搭建單機版Hadoop2.8（踩坑記錄）

目錄安裝jdk1、下載jdk的包並安裝2、配置環境變量3、驗證Hadoop安裝和配置1、下載Hadoop包及安裝2、環境變量配置Hadoop的文件修改1、core-site.xml2、mapred-site.xml（hadoop2

33 Audrey 2020-07-08 12:35:24

大數據入門（七）win10上eclipse使用Hadoop的配置

目錄工具eclipse的Hadoop環境配置參考 系列： 大數據入門（一）環境搭建，VMware15+CentOS8.1 配置 https://blog.csdn.net/qq_34391511/article/details/1

33 Audrey 2020-07-08 12:35:23

學習Hadoop第一個MapReduce程序WordCount的詳解

WordCount官方代碼的詳解 作爲小白的我初次在hadoop裏接觸mapReduce程序，難免會有些費解以及難懂代碼所蘊含的意思，經過自己查閱資料以及消化，便得到下面代碼的的解釋： package homework; imp

柘月十七 2020-07-08 12:33:27

WordCount大變身做一些簡單的數據處理應用案列

目錄前言任務描述任務分析MapReduce默認排序規則代碼實現任務最終效果 ———————————————————————————————————————— 前言 WordCount算是我們學習hadoop的第一分佈式計算程序了，

柘月十七 2020-07-08 12:33:17

Hadoop集羣裏執行程序出現異常就要尋找日誌文件查看情況

當我們運行Hadoop執行我們的程序時，很多時候因爲我們的各種無法猜測得到的原因導致程序執行出現異常情況，當然這時的我們也會很苦惱並大聲尖叫“太TM難了，嗚嗚嗚~”，莫着急！ 出現這樣的情況就需要我們冷靜下來啦，現在我們就需要去尋

柘月十七 2020-07-08 12:33:16

hadoop入門習題集

以下是我上hadoop課程時由老師提供的習題集（利用Hadoop提供的shell命令完成任務）： 一、將HDFS中指定文件的內容輸出到終端中； 二、顯示HDFS中指定的文件的讀寫權限、大小、創建時間、路徑等信息； 三、給定HD

柘月十七 2020-07-08 12:33:16

關於IP地址與主機名映射的/etc/hosts文件配置

爲什麼要這樣做？ 首先，在hadoop工作就像是一個社團幫派，master是老大，而slave1、slave2等就是master的小弟。但是，slave並沒有像我們那麼聰明一眼就能分辨出自己的老大，它們是以ip地址作爲辨別的。那麼

柘月十七 2020-07-08 12:33:16

Hadoop-02 HDFS簡介 讀寫文件

Hadoop - HDFS 簡介 Hadoop Distributed File System，分佈式文件系統 架構 Block數據塊 基本存儲單位，一般大小爲64M 配置大的塊主要原因： 減少搜尋時間，一般硬盤傳輸速率比

LiuDi1999 2020-07-08 12:26:19

Hadoop-03 Yarn

YARN - ResourceManager 負責全局的資源管理和任務調度，把整個集羣當成計算資源池，只關注分配，不管應用，且不負責容錯 資源管理 以前資源是每個節點分成一個個的Map slot和Reduce slot，現在是

LiuDi1999 2020-07-08 12:26:19

Hadoop -04 MapReduce介紹

Hadoop-MapReduce MapReduce是一個框架，我們可以使用它來編寫應用程序，以可靠的方式並行地處理大量商品硬件羣集上的大量數據。 什麼是MapReduce？ MapReduce是一種基於java的分佈式計算的處理

LiuDi1999 2020-07-08 12:26:19

Hadoop-01 hadoop簡介安裝配置測試

Hadoop - 簡介 Hadoop可運行於一般的商用服務器上，具有高容錯、高可靠性、高擴展性等特點 特別適合寫一次，讀多次的場景 適合 大規模數據 流式數據（寫一次，讀多次） 商用硬件（一般硬件） 不適合 低延時的數據訪問

LiuDi1999 2020-07-08 12:26:19