一、環境準備篇
vmware虛擬機安裝兩個centos7服務器（這裏介紹一下vmware虛擬機設置固定ip）
這樣vmware虛擬機固定ip網絡就搭建好了，搭建Hadoop機器說明：主：192.168.2.2，從192.168.2.3（虛擬機一開始要設置固定ip） 兩個服務器上的/etc/hosts如下： 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 192.168.2.2 master 192.168.2.3 slave0
二、hadoop安裝包下載部署，下載地址： http://archive.apache.org/dist/hadoop/core/hadoop-2.7.5/
1、cd /data/software/ 2、rz -y 3、tar -zxvf hadoop-2.7.5.tar.gz /usr/local/ 4、mv hadoop-2.7.5 hadoop 5、vi /usr/local/hadoop/etc/hadoop/hadoop-env.sh 找到export JAVA_HOME，修改爲export JAVA_HOME=/usr/local/jdk1.8.0_251 6、 vi /usr/local/hadoop/etc/hadoop/core-site.xml 需要在<configuration>和</configuration>之間加入的代碼： <!--指定namenode的地址--> <property> <name>fs.defaultFS</name> <value>hdfs://master:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/opt/hadoop/hadoopdata</value> </property> 7、vi /usr/local/hadoop/etc/hadoop/hdfs-site.xml 需要在<configuration>和</configuration>之間加入的代碼： <!--指定hdfs保存數據的副本數量--> <property> <name>dfs.replication</name> <value>1</value> </property> 8、vi /usr/local/hadoop/etc/hadoop/yarn-site.xml 需要在<configuration>和</configuration>之間加入的代碼： <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>master:18040</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>master:18030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>master:18025</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>master:18141</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>master:18088</value> </property> 9、cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml vi /usr/local/hadoop/etc/hadoop/mapred-site.xml 需要在<configuration>和</configuration>之間加入的代碼： <!--告訴hadoop以後MR運行在yarn上--> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> 10、vi /opt/hadoop/hadoop/etc/hadoop/slaves slave0 slave1 11、scp -r /usr/local/hadoop root@slave0:/usr/local 12、cd /usr/local/hadoop（兩個節點都做） vi ~/.bash_profile export HADOOP_HOME=/usr/local/hadoop export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH source ~/.bash_profile 13、mkdir /opt/hadoop/hadoopdata hadoop namenode -format 14、cd /usr/local/hadoop/sbin start-all.sh stop-all.sh
在13做完以後需要創建hadoop用戶，hadoop組，然後使用chown命令將/usr/local/hadoop授權給hadoop組下的Hadoop用戶，檢查兩個服務器上的進程，進程明細如下：
192.168.2.2主節點 18784 SecondaryNameNode 18579 NameNode 18942 ResourceManager 21726 Jps 192.168.2.3從節點 129605 NodeManager 1445 Jps 129480 DataNode
如果從節點上datanode沒有啓動，可以查看從節點上datanode日誌，日誌目錄 ：/usr/local/hadoop/logs/hadoop-hadoop-datanode-slave0.log
cd /usr/local/hadoop/bin
echo "abc" file.txt
ls -l
./hdfs dfs -ls /
./hdfs dfs -put file.txt /
這樣就上傳了一個文件到hdfs中，可以打開地址http://192.168.2.2:50070/查看根目錄下是否存在file.txt
三、hive安裝包下載部署，下載地址： http://mirror.bit.edu.cn/apache/hive/
1、 tar -zxvf apache-hive-2.1.1-bin.tar.gz -C /usr/local/
2、 mv apache-hive-2.1.1-bin hive
3、 vi /etc/profile
export HIVE_HOME=/usr/local/hive
export PATH=$PATH:$HIVE_HOME/bin
source /etc.profile
4、cd /usr/local/hive/conf/
vi hive-site.xml
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://192.168.2.2:3306/hivedb?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
<description>username to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>1234</value>
<description>password to use against metastore database</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
<description>hive default warehouse, if nessecory, change it</description>
</property>
</configuration>
5、複製mysql的驅動程序到hive/lib下面
6、cd /usr/local/hive/bin
schematool -dbType mysql -initSchema
7、執行hive命令
create database test_work
create table course (id int,name string);8、查詢hadoop目錄
hadoop fs -lsr /
drwxr-xr-x - hadoop supergroup 0 2020-08-28 22:53 /user
drwxr-xr-x - hadoop supergroup 0 2020-08-28 22:53 /user/hive
drwxr-xr-x - hadoop supergroup 0 2020-08-28 22:53 /user/hive/warehouse
drwxr-xr-x - hadoop supergroup 0 2020-08-28 22:53 /user/hive/warehouse/test_work.db
drwxr-xr-x - hadoop supergroup 0 2020-08-28 22:54 /user/hive/warehouse/test_work.db/course
9、查詢hivedb數據庫的DBS、TBLS差異