[TOP]
ip设置:
服务器中共虚拟了6台虚拟机:
hadoop1 :内存8G,硬盘2T
hadoop2 :内存8G,硬盘2T
hadoop3 :内存8G,硬盘2T
zookeeper :内存8G,硬盘2T
redis :内存8G,硬盘2T
ethings :内存8G,硬盘2T
192.168.0.10 hadoop1 // hadoop2.7.4 + zookeeper3.4.10 + hbase1.2.6 + hive2.1.1 + mariadb5.5
192.168.0.11 hadoop2 // hadoop2.7.4 + zookeeper3.4.10 + hbase1.2.6
192.168.0.12 hadoop3 // hadoop2.7.4 + zookeeper3.4.10 + hbase1.2.6
192.168.0.13 zookeeper // (备用)
192.168.0.14 redis // redis4.0.1 + mysql(mariadb5.5)
192.168.0.15 ethings // 应用平台
192.168.0.16 hadoop3 // hadoop2.7.4 + zookeeper3.4.10 + hbase1.2.6
192.168.0.17 hadoop3 // hadoop2.7.4 + zookeeper3.4.10 + hbase1.2.6
这是整个在给的服务器上搭建的环境,
目前数据存储在 hadoop2 和 hadoop3 上。
存储平台开关
启动:
1. 先启动zookeeper:
hadoop1:zkServer.sh start
hadoop2:zkServer.sh start
hadoop3:zkServer.sh start
2. 启动hbase
hadoop1:start-hbase.sh
3. 启动hadoop集群
hadoop1:start-all.sh
关闭:
1. 先关闭zookeeper:
hadoop1:zkServer.sh stop
hadoop2:zkServer.sh stop
hadoop3:zkServer.sh stop
2. 关闭hbase
hadoop1:stop-hbase.sh
3. 关闭hadoop集群
hadoop1:stop-all.sh
hadoop
core-site.xml
<configuration>
<!-- 指定HDFS老大(namenode)的通信地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop1:9000</value>
</property>
<!-- 指定hadoop运行时产生文件的存储路径 -->
<property>
<name>hadoop.tmp.dir</name>
<value>file:/usr/local/hadoop/tmp</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop1:9001</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop/hdfs/data</value>
</property>
<!-- 设置hdfs副本数量 -->
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>
客户端,添加环境变量:HADOOP_USER_NAME=hadoop (如果在客户端的IDE中调试需要设置这个环境变量,如eclipse、idea等),这样就不会发生访问权限问题了。
另外,说一下,建立虚拟机时候,就默认使用hadoop用户建立,这样就不用专门去建立这个用户和组了。
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop1:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop1:19888</value>
</property>
</configuration>
slaves
hadoop2
hadoop3
hadoop4
hadoop5
yarn-site.xml
<configuration>
<!-- reducer取数据的方式是mapreduce_shuffle -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop1:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hadoop1:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop1:8088</value>
</property>
</configuration>
hbase
hbase-site.xml
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://hadoop1:9000/hbase</value>
</property>
<value>hadoop1,hadoop2,hadoop3,hadoop4,hadoop5</value>
</property>
<property>
<name>zookeeper.session.timeout</name>
<value>60000000</value>
</property>
<property>
<name>dfs.support.append</name>
<value>true</value>
</property>
</configuration>
hbase-env.sh
export HBASE_OPTS="-XX:+UseConcMarkSweepGC"
# Configure PermSize. Only needed in JDK7. You can safely remove it for JDK8+
export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m"
export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m"
regionservers
hadoop2
hadoop3
hadoop4
hadoop5
vbox设置共享目录
mount -t vboxsf share /home/hadoop/mount_point/
虚拟机中需要安装VBoxGuestAdditions.iso
挂载: mount /dev/cdrom /home/hadoop/mount_point/
cd /home/hadoop/mount_point/
sh ./VBoxLinuxAdditions.run
执行过程中可能有错,根据错误日志修改
需要
sudo yum install gcc kernal kernal-devel
成功后就可以挂载了
mount -t vboxsf share /home/hadoop/mount_point/
Zookeeper设置
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/local/zookeeper/tmp/data
dataLogDir=/usr/local/zookeeper/tmp/logs
clientPort=2181
server.1=hadoop1:2888:3888
server.2=hadoop2:2888:3888
server.3=hadoop3:2888:3888
#server.4=hadoop4:2888:3888
#server.5=hadoop5:2888:3888
#maxClientCnxns=60
#autopurge.snapRetainCount=3
#autopurge.purgeInterval=1
配置好后在dataDir
目录中创建myid,并相应的设置 1,2,3,4,5,
HIVE 设置
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&characterEncoding=UTF-8&useSSL=false</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>root</value>
</property>
<!-- 设置 hive仓库的HDFS上的位置 -->
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
</property>
<!--资源临时文件存放位置 -->
<property>
<name>hive.downloaded.resources.dir</name>
<value>/usr/local/hive/tmp/${hive.session.id}_resources</value>
</property>
<!-- Hive在0.9版本之前需要设置hive.exec.dynamic.partition为true, Hive在0.9版本之后默认为true -->
<property>
<name>hive.exec.dynamic.partition</name>
<value>true</value>
</property>
<property>
<name>hive.exec.dynamic.partition.mode</name>
<value>nonstrict</value>
</property>
<!-- 修改日志位置 -->
<property>
<name>hive.exec.local.scratchdir</name>
<value>/usr/local/hive/tmp/HiveJobsLog</value>
</property>
<property>
<name>hive.downloaded.resources.dir</name>
<value>/usr/local/hive/tmp/ResourcesLog</value>
</property>
<property>
<name>hive.querylog.location</name>
<value>/usr/local/hive/tmp/HiveRunLog</value>
</property>
<property>
<name>hive.server2.logging.operation.log.location</name>
<value>/usr/local/hive/tmp/OperationLogs</value>
</property>
<!-- 配置HWI接口 -->
<property>
<name>hive.hwi.war.file</name>
<value>${env:HWI_WAR_FILE}</value>
</property>
<property>
<name>hive.hwi.listen.host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>hive.hwi.listen.port</name>
<value>9999</value>
</property>
<!-- Hiveserver2已经不再需要hive.metastore.local这个配置项了(hive.metastore.uris为空,则表示是metastore在本地,否则就是远程)远程的话直接配置hive.metastore.uris即可 -->
<!-- property>
<name>hive.metastore.uris</name>
<value>thrift://m1:9083</value>
<description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
</property -->
<property>
<name>hive.server2.thrift.bind.host</name>
<value>hadoop1</value>
</property>
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
</property>
<property>
<name>hive.server2.thrift.http.port</name>
<value>10001</value>
</property>
<property>
<name>hive.server2.thrift.http.path</name>
<value>cliservice</value>
</property>
<!-- HiveServer2的WEB UI -->
<property>
<name>hive.server2.webui.host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>hive.server2.webui.port</name>
<value>10002</value>
</property>
<property>
<name>hive.scratch.dir.permission</name>
<value>755</value>
</property>
<!-- 下面hive.aux.jars.path这个属性里面你这个jar包地址如果是本地的记住前面要加file://不然找不到, 而且会报org.apache.hadoop.hive.contrib.serde2.RegexSerDe错误 -->
<property>
<name>hive.aux.jars.path</name>
<value/>
</property>
<property>
<name>hive.server2.enable.doAs</name>
<value>true</value>
</property>
<property>
<name>hive.auto.convert.join</name>
<value>true</value>
</property>
<property>
<name>spark.dynamicAllocation.enabled</name>
<value>true</value>
<description>动态分配资源</description>
</property>
<!-- 使用Hive on spark时,若不设置下列该配置会出现内存溢出异常 -->
<property>
<name>spark.driver.extraJavaOptions</name>
<value>-XX:PermSize=128M -XX:MaxPermSize=512M</value>
</property>
<property>
<name>datanucleus.autoCreateSchema</name>
<value>true</value>
</property>
<property>
<name>datanucleus.autoCreateTables</name>
<value>true</value>
</property>
<property>
<name>datanucleus.autoCreateColumns</name>
<value>true</value>
</property>
</configuration>
hive的配置感觉是最复杂的了,上面使用的是mysql作为元数据管理,如果用centos7的话,系统默认自带mariadb数据库跟mysql是一样的。
Redis4.0.1
安装:
$ wget http://download.redis.io/releases/redis-4.0.1.tar.gz
$ tar xzf redis-4.0.1.tar.gz
$ cd redis-4.0.1
$ make
make成功后,执行
$ src/redis-server
测试:
$ src/redis-cli
redis> set foo bar
OK
redis> get foo
"bar"
如果make不成功,可以参考README.md,使用
make MALLOC=libc
再编译一次,默认使用
make MALLOC=jemalloc
VirtualBox 磁盘复制
虚拟机做好一个后,复制一下就会得到另一台虚拟机,但是有时候,并不是通过VirtualBox界面工具复制的,直接手动复制粘贴,这样这个虚拟机是启动不来的,所以需要如下方法:
cmd至virtualBox运行目录后,执行
VBoxManage.exe internalcommands sethduuid G:\vbox\xxx.vdi
将修改VDI的UUID
修改成功提示UUID changed to: 428079cd-830d-49b1-bfde-feac051b4d3e
run VBoxManage internalcommands sethduuid <VDI/VMDK file>
twice (the first time is just to conveniently generate an UUID, you could use any other UUID generation method instead)
open the .vbox
file in a text editor
replace the UUID found in <Machine uuid="{...}"
with the UUID you got when you ran sethduuid the first time
replace the UUID found in <HardDisk uuid="{...}"
and in <Image uuid="{}"
(towards the end) with the UUID you got when you ran sethduuid the second time
Spark
spark-env.sh
export SCALA_HOME=/usr/local/scala
export JAVA_HOME=/usr/local/jdk
export SPARK_MASTER_IP=hadoop1
export SPARK_WORKER_MEMORY=4G
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
我这里虚拟机的内存是8G的。
设置静态IP地址
[hadoop@zookeeper network-scripts]$ cat ifcfg-enp0s3
TYPE="Ethernet"
#BOOTPROTO="dhcp"
BOOTPROTO="static"
IPADDR=192.168.0.10
NETMASK=255.255.255.0
DEFROUTE="yes"
PEERDNS="yes"
PEERROUTES="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_PEERDNS="yes"
IPV6_PEERROUTES="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="enp0s3"
UUID="09f62fa6-36bc-4782-95ad-63fda20b194f"
DEVICE="enp0s3"
ONBOOT="yes"
关闭防火墙
sudo systemctl stop firewalld.service
sudo systemctl disable firewalld.service
sudo systemctl status firewalld.service
启动一个服务:systemctl start firewalld.service
关闭一个服务:systemctl stop firewalld.service
重启一个服务:systemctl restart firewalld.service
显示一个服务的状态:systemctl status firewalld.service
在开机时启用一个服务:systemctl enable firewalld.service
在开机时禁用一个服务:systemctl disable firewalld.service
查看服务是否开机启动:systemctl is-enabled firewalld.service
查看已启动的服务列表:systemctl list-unit-files|grep enabled
从VirtulBox转VMware磁盘
vmkload_mod multiextent
vmkfstools -i hadoop3-disk1.vmdk hadoop3-disk2.vmdk -d thin
vmkfstools -U hadoop3-disk1.vmdk
vmkfstools -E hadoop3-disk2.vmdk hadoop3-disk1.vmdk
vmkload_mod -u multiextent
Hadoop退出安全模式
1. 在HDFS配置文件中修改安全模式阀值
在hdfs-site.xml中设置安全阀值属性,属性值默认为0.999f,如果设为1则不进行安全检查
<property>
<name>dfs.safemode.threshold.pct</name>
<value>0.999f</value>
<description>
Specifies the percentage of blocks that should satisfy
the minimal replication requirement defined by dfs.replication.min.
Values less than or equal to 0 mean not to wait for any particular
percentage of blocks before exiting safemode.
Values greater than 1 will make safe mode permanent.
</description>
</property>
因为是在配置文件中进行硬修改,不利于管理员操作和修改,因此不推荐此方式
2. 直接在bash输入指令脱离安全模式(推荐)
在安全模式下输入指令:
hadoop dfsadmin -safemode leave
即可退出安全模式。
hdfs文件保存到本地
hadoop fs -get [-ignorecrc] [-crc] 复制文件到本地文件系统
hadoop fs -get hdfs://host:port/user/hadoop/file localfile
各模式下运行spark自带实例SparkPi
2.1 local模式
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master local lib/spark-examples-1.0.0-hadoop2.2.0.jar
2.2 standalone模式
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://192.168.0.10:7077 lib/spark-examples-1.0.0-hadoop2.2.0.jar
2.3 on-yarn-cluster模式
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster lib/spark-examples-1.0.0-hadoop2.2.0.jar
2.4 on-yarn-client模式
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client lib/spark-examples-1.0.0-hadoop2.2.0.jar
2.5 参考
http://spark.apache.org/docs/latest/submitting-applications.html