Yarn部署:
nn nn dn jn zkfc zk RS NM
node001 * * *
node002 * * * * * *
node003 * * * * *
node004 * * * *
mapred-site.xml:
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
yarn-site.xml:
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>cluster1</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>node003</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>node004</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>master1:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>master2:8088</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>node002:2181,node003:2181,node004:2181</value>
</property>
开始操作:(纵观配置发现path环境变量配置中必须有hadooop的sbin,否则好多命令不能用)
到hadoop的/etc/hadoop下
1、赋值修改名称:cp mapred-site.xml.template 为mapred-site.xml
拷贝以上内容
2、修改yarn-site.xml:拷贝以上内容
3、分发该配置到其他三台
4、node001:start-yarn.sh
5、手动启动resourceManager在node003和node004:
执行zkCli.sh后可看到yarn-leader-election ls /yarn-leader-election、ls /yarn-leader-election/cluster1
、get /yarn-leader-election/cluster1/ActiveStandbyElectorLock
6、跑任务实践:
for i in `seq 100000`;do echo "hello sxt $i" >> test.txt; done
新建目录:hdfs dfs -mkdir -p /user/root
上传文件:hdfs dfs -D dfs.blocksize=1048576 -put test.txt /user/root
切换到/opt/bigdata/hadoop-2.6.5/share/hadoop/mapreduce
执行任务 hadoop jar hadoop-mapreduce-examples-2.6.5.jar wordcount /user/root/test.txt /data/wc/output
下载查看结果:
mkdir data
cd data
hdfs dfs -ls /data/wc/output
hdfs dfs -get /data/wc/output/* ./