Centos_Hadoop2.7.1環境搭建記錄

1、上傳安裝包到/home/hadoop/upload  路徑下

2、解壓縮

$tar -xzf hadoop-2.7.1.tar.gz

3、移動解壓文件至/app/hadoop

$ mv hadoop-2.7.1 /app/hadoop

4、創建子目錄

$cd /app/hadoop/hadoop-2.7.1/ 

$mkdir tmp 

$mkdir name

  $mkdir data 

$ll

5、配置hadoop-env.sh

5.1. 打開配置文件hadoop-env.sh

$cd /app/hadoop/hadoop-2.7.0/etc/hadoop 

$sudo vi hadoop-env.sh

*使用sudo vi hadoop-env.sh  命令時，提示"hadoop"用戶不在sudoers 文件中，

需要把當前用戶添加進去，步驟如下：

1 修改/etc/sudoers文件，進入超級用戶，因爲沒有寫權限，所以要先把寫權限加上chmod u+w /etc/sudoers。

2 編輯/etc/sudoers文件，找到這一 行："root ALL=(ALL) ALL"在起下面添加"hadoop ALL=(ALL) ALL"，然後保存。

3 最後恢復沒有寫權限模式，撤銷文件的寫權限，chmod u-w /etc/sudoers。

5.2. 加入配置內容，設置JAVA_HOME和PATH路徑

export JAVA_HOME=/usr/lib/java/jdk1.7.0_55 

export PATH=$PATH:/app/hadoop/hadoop-2.7.0/bin

5.3. 編譯配置文件hadoop-env.sh，並確認生效

$source hadoop-env.sh

$hadoop version

*查看版本，檢查配置是否正確

6、配置yarn-env.sh

配置JAVA_HOME,過程同上

7、配置core-site.xml

<configuration> 
<span style="white-space:pre">	</span><property>
  <span style="white-space:pre">		</span><name>fs.default.name</name>
  <span style="white-space:pre">		</span><value>hdfs://Master.Hadoop:9000</value> 
<span style="white-space:pre">	</span></property> 
 <span style="white-space:pre">	</span><property> 
  <span style="white-space:pre">		</span><name>fs.defaultFS</name> 
  <span style="white-space:pre">		</span><value>hdfs://Master.Hadoop:9000</value>
 <span style="white-space:pre">	</span></property> 
 <span style="white-space:pre">	</span><property> 
 <span style="white-space:pre">		</span> <name>io.file.buffer.size</name>
  <span style="white-space:pre">		</span><value>131072</value> 
 <span style="white-space:pre">	</span></property> 
 <span style="white-space:pre">	</span><property>
 <span style="white-space:pre">		</span> <name>hadoop.tmp.dir</name>
 <span style="white-space:pre">		</span> <value>file:/app/hadoop/hadoop-2.2.0/tmp</value>
 <span style="white-space:pre">		</span> <description>Abase for other temporary directories.</description> 
 <span style="white-space:pre">	</span></property> 
 <span style="white-space:pre">	</span><property> 
 <span style="white-space:pre">		</span> <name>hadoop.proxyuser.hduser.hosts</name> 
 <span style="white-space:pre">		</span> <value>*</value>
 <span style="white-space:pre">	</span></property> 
 <span style="white-space:pre">	</span><property> 
 <span style="white-space:pre">		</span> <name>hadoop.proxyuser.hduser.groups</name>
  <span style="white-space:pre">		</span><value>*</value> 
 <span style="white-space:pre">	</span></property>
</configuration>

8、配置hdfs-site.xml

<configuration> 
	<property> 
		<name>dfs.namenode.secondary.http-address</name> 
		<value>Master.Hadoop:9001</span>
	</property> 
	<property> 
		<name>dfs.namenode.name.dir</name> 
		<value>file:/app/hadoop/hadoop-2.7.1/name</value>
	</property> 
	<property> 
		<name>dfs.datanode.data.dir</name> 
		<value>file:/app/hadoop/hadoop-2.7.1/data</value> 
	</property> 
	<property> 
		<name>dfs.replication</name> 
		<value>2</value> 
	</property>
	<property> 
		<name>dfs.webhdfs.enabled</name>
		<value>true</value>
	</property> 
</configuration>

9、配置mapred-site.xml

$cp mapred-site.xml.template mapred-site.xml

<configuration> 
	<property> 
		<name>mapreduce.framework.name</name> 
		<value>yarn</value> 
	</property> 
	<property> 
		<name>mapreduce.jobhistory.address</name> 
		<value>Master.Hadoop:10020 </span>
	</property>
	<property> 
		<name>mapreduce.jobhistory.webapp.address</name> 
		<value>Master.Hadoop:19888 </span>
	</property> 
</configuration>

10、配置yarn-site.xml

<configuration> 
	<property> 
		<name>yarn.nodemanager.aux-services</name> 
		<value>mapreduce_shuffle</value> 
	</property> 
	<property> 
		<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> 
		<value>org.apache.hadoop.mapred.ShuffleHandler</value> 
	</property> 
	<property> 
		<name>yarn.resourcemanager.address</name> 
		<value>Master.Hadoop:8032</span>
	</property> 
	<property> 
		<name>yarn.resourcemanager.scheduler.address</name> 
		<value>Master.Hadoop:8030 </span>
	</property>
	<property> 
		<name>yarn.resourcemanager.resource-tracker.address</name> 
		<value>Master.Hadoop:8031</span>
	</property> 
	<property> 
		<name>yarn.resourcemanager.admin.address</name>
		<value>Master.Hadoop:8033 </span>
	</property> 
	<property> 
		<name>yarn.resourcemanager.webapp.address</name> 
		<value>Master.Hadoop:8088</span>
	</property> 
</configuration>

11、配置Slaves

$sudo vi slaves

在文件中加入

Master.Hadoop

Slave1.Hadoop 

Slave2.Hadoop

12、將hadoop-2.7.1分發到各節點

$scp -r hadoop-2.2.0 [email protected]:/app/hadoop/ 

$scp -r hadoop-2.2.0 [email protected]:/app/hadoop/

*SCP命令報錯：Bad configuration option:AuthorizedkeysFile   

找了很多資料，發現配置文件/etc/ssh/ssh_config中沒有此字段，死馬當活馬醫，屏蔽掉AuthorizedkeysFile這一行就OK了。

13、格式化NameNode

$./bin/hdfs namenode -format

14、啓動

sbin目錄下執行 ./start-all.sh