Cloudera Manager安裝部署手冊

1. 安裝前的準備

 準備好Ubuntu離線source，配置/etc/apt/source.list
 準備好Cloudera Hadoop離線source，配置/etc/apt/source.list.d/cloudera.list
 同步集羣的/etc/hosts
 ssh免密碼登錄，cloudera-scm-server所在機器能夠免密碼登錄其他cloudera-scm-agent機器或者所有機器的root用戶密碼相同

2. 系統環境

OS：ubuntu12.04 64bit
CDH：cdh3u6
Cloudera manager：4.8.2

3. 數據庫的配置

3.1 安裝配置MySQL

Cloudera Manager支持三種數據庫類型：MySQL、PostgreSQL、Oracle。我們選擇使用MySQL。
apt-get install mysql-server
service mysql start
在/etc/mysql/conf.d下創建mysql_cloudera_manager.cnf文件，加入以下內容：
[mysqld]
transaction-isolation=READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links=0


key_buffer              = 16M
key_buffer_size         = 32M
max_allowed_packet      = 16M
thread_stack            = 256K
thread_cache_size       = 64
query_cache_limit       = 8M
query_cache_size        = 64M
query_cache_type        = 1
# Important: see Configuring the Databases and Setting max_connections
max_connections         = 550


# log-bin should be on a disk with enough free space
#log-bin=/x/home/mysql/logs/binary/mysql_binary_log


# For MySQL version 5.1.8 or later. Comment out binlog_format for older versions.
binlog_format           = mixed


read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M


# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2

innodb_log_buffer_size          = 64M

#cant't over 4G on 32-bit OS

innodb_buffer_pool_size         = 4G
innodb_thread_concurrency       = 8
innodb_flush_method             = O_DIRECT
innodb_log_file_size = 512M


[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

3.2 配置innodb

mv /var/lib/mysql/ib_logfile* ~
重啓mysql
service mysql restart

3.3 安裝mysql-connector

apt-get install libmysql-java

3.4 創建數據庫

需要創建的數據庫列表：
amon smon ran hmon hive
mysql -u root -p iflytek
create database amon DEFAULT CHARACTER SET utf8;
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';
create database smon DEFAULT CHARACTER SET utf8;
grant all on smon.* TO 'smon'@'%' IDENTIFIED BY 'smon_password';
create database rman DEFAULT CHARACTER SET utf8;
grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password';
create database hmon DEFAULT CHARACTER SET utf8;
grant all on hmon.* TO 'hmon'@'%' IDENTIFIED BY 'hmon_password';
create database hive DEFAULT CHARACTER SET utf8;
grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';
備份數據庫
mysqldump -piflytek amon > /tmp/amon-backup.sql
和其他集羣數據庫

3.5 安裝Cloudera Manager

添加Cloudera Manager的源
創建/etc/apt/sources.list.d/cloudera-manager.list文件，並在文件中添加
deb [arch=amd64] http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh precise-cdh4 contrib
deb-src http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh precise-cdh4 contrib
deb http://archive.cloudera.com/cm4/ubuntu/precise/amd64/cm precise-cm4 contrib
deb-src http://archive.cloudera.com/cm4/ubuntu/precise/amd64/cm precise-cm4 contrib
保存後下載key
curl -s http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh/archive.key | sudo apt-key add -
若上述命令執行失敗，請自己下載key文件導入。apt-key add [file path]
在Server機器上執行
apt-get install cloudera-manager-daemons cloudera-manager-server 
在agent機器上執行
apt-get installation cloudera-manager-agent cloudera-manager-daemons

3.6 爲cloudera manager準備數據庫

執行/usr/share/cmf/schema/scm_prepare_database.sh database-type [options] database-name username password
如
./scm_prepare_database.sh mysql amon root xxx
./scm_prepare_database.sh mysql smon root xxx
./scm_prepare_database.sh mysql hmon root xxx
./scm_prepare_database.sh mysql rman root xxx
./scm_prepare_database.sh mysql hive root xxx

3.7 安裝HDFS

指定一臺機器做namendoe
apt-get install namenode
在指定的DataNode機器上執行
apt-get install datanode

3.8 啓動cloudera manager Server和agent

在所有機器上創建一些保存日誌的目錄：
mkdir -p /var/log/cloudera-scm-headlamp
mkdir -p /var/log/cloudera-scm-firehose
mkdir -p /var/log/cloudera-scm-alertpublisher
mkdir -p /var/log/cloudera-scm-eventserver
mkdir -p /var/lib/cloudera-scm-headlamp
mkdir -p /var/lib/cloudera-scm-firehose
mkdir -p /var/lib/cloudera-scm-alertpublisher
mkdir -p /var/lib/cloudera-scm-eventserver
創建完成後修改owner
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-*
chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-*
如果使用tar包安裝需要chown到cloudera-scm用戶組下。如果使用apt-get方式安裝不需要chown。默認使用的是root。
修改所有agent的JDK配置
在/etc/default/ cloudera-scm-agent中增加
export CMF_AGENT_JAVA_HOME=/usr/local/jdk6
在Server機器上執行：
service cloudera-scm-server start
分別登錄agent機器執行：
service cloudera-scm-agent start

3.9 登錄web端

登錄部署有scm-server機器的http://ServerIP:7180
用戶名：admin
密碼：admin

3.10 數據庫設置

數據庫主機名稱：localhost，用戶名：root，數據庫名稱上面那5個，密碼：xxxx，即MySQL數據庫的密碼。

4. web端使用

略