在Ubuntu中安裝HBase
R利劍NoSQL系列文章,主要介紹通過R語言連接使用nosql數據庫。涉及的NoSQL產品,包括Redis, MongoDB, HBase, Hive, Cassandra, Neo4j。希望通過我的介紹讓廣大的R語言愛好者,有更多的開發選擇,做出更多地激動人心的應用。
關於作者:
- 張丹(Conan), 程序員Java,R,PHP,Javascript
- weibo:@Conan_Z
- blog: http://blog.fens.me
- email: [email protected]
轉載請註明出處:
http://blog.fens.me/linux-hbase-install/
前言
HBase是Hadoop家族中的一個分佈式數據庫產品,HBase支持高併發讀寫,列式數據存儲,高效的索引,自動分片,自動Region遷移等許多優點,已經越來越多的被界業認可並實施。
目錄
- 在Ubuntu中環境準備
- HBase安裝
- Thrift安裝
1 在Ubuntu中環境準備
HBase是基於Java開發的運行Hadoop平臺上分佈式NoSQL數據庫軟件,HBase沒有提供Windows系統安裝版本。我在這裏也只介紹HBase在Linux Ubuntu系統中的安裝。
由於HBase是運行在Hadoop平臺上面的,因此我們需要先安裝好Hadoop的環境,Hadoop的安裝請參考文章:[Hadoop歷史版本安裝](http://blog.fens.me/hadoop-history-source-install/)
HBase沒有提供apt的軟件源安裝,我們需要自己去官方網絡下載HBase軟件包進行安裝。HBase下載頁:http://www.apache.org/dyn/closer.cgi/hbase/
系統環境:
- Linux Ubuntu 12.04.2 LTS 64bit server
- Java JDK 1.6.0_45
- Hadoop 1.1.2
2 HBase安裝
2.1 下載HBase
# 通過wget命令下載
~ wget http://www.gaidso.com/apache/hbase/stable/hbase-0.94.18.tar.gz
# 解壓HBase
~ tar xvf hbase-0.94.18.tar.gz
# 移動HBase目錄到文件夾
~ mv hbase-0.94.18/ /home/conan/hadoop/
# 進入目錄
~ cd /home/conan/hadoop/hbase-0.94.18
2.2 配置HBase
2.2.1 修改啓動文件hbase-env.sh
~ vi conf/hbase-env.sh
#打開註釋
export JAVA_HOME=/home/conan/toolkit/jdk16
export HBASE_CLASSPATH=/home/conan/hadoop/hadoop-1.1.2/conf
export HBASE_MANAGES_ZK=true
2.2.2 修改配置文件 hbase-site.xml
~ vi conf/hbase-site.xml
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://master:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>master</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/conan/hadoop/hdata</value>
</property>
</configuration>
複製hadoop環境的配置文件和類庫
~ cp ~/hadoop/hadoop-1.1.2/conf/hdfs-site.xml conf/
~ cp ~/hadoop/hadoop-1.1.2/hadoop-core-1.1.2.jar lib/
~ mkdir /home/conan/hadoop/hdata
2.3 啓動hadoop和hbase
~ /home/conan/hadoop/hadoop-1.1.2/bin/start-all.sh
~ /home/conan/hadoop/hbase-0.94.18/bin/start-hbase.sh
# 查看hbase進程
~ jps
13838 TaskTracker
13541 JobTracker
15946 HMaster
16756 Jps
12851 NameNode
13450 SecondaryNameNode
13133 DataNode
15817 HQuorumPeer
16283 HRegionServer
2.4 打開HBase命令行客戶端訪問Hbase
~ bin/hbase shell
HBase Shell; enter 'help' for list of supported commands.
Type "exit" to leave the HBase Shell
Version 0.94.18, r1577788, Sat Mar 15 04:46:47 UTC 2014
hbase(main):002:0> help
HBase Shell, version 0.94.18, r1577788, Sat Mar 15 04:46:47 UTC 2014
Type 'help "COMMAND"', (e.g. 'help "get"' -- the quotes are necessary) for help on a specific command.
Commands are grouped. Type 'help "COMMAND_GROUP"', (e.g. 'help "general"') for help on a command group.
COMMAND GROUPS:
Group name: general
Commands: status, version, whoami
Group name: ddl
Commands: alter, alter_async, alter_status, create, describe, disable, disable_all, drop, drop_all, enable, enable_all, exists, is_disabled, is_enabled, list, show_filters
Group name: dml
Commands: count, delete, deleteall, get, get_counter, incr, put, scan, truncate
Group name: tools
Commands: assign, balance_switch, balancer, close_region, compact, flush, hlog_roll, major_compact, move, split, unassign, zk_dump
Group name: replication
Commands: add_peer, disable_peer, enable_peer, list_peers, list_replicated_tables, remove_peer, start_replication, stop_replication
Group name: snapshot
Commands: clone_snapshot, delete_snapshot, list_snapshots, restore_snapshot, snapshot
Group name: security
Commands: grant, revoke, user_permission
SHELL USAGE:
Quote all names in HBase Shell such as table and column names. Commas delimit
command parameters. Type after entering a command to run it.
Dictionaries of configuration used in the creation and alteration of tables are
Ruby Hashes. They look like this:
{'key1' => 'value1', 'key2' => 'value2', ...}
and are opened and closed with curley-braces. Key/values are delimited by the
'=>' character combination. Usually keys are predefined constants such as
NAME, VERSIONS, COMPRESSION, etc. Constants do not need to be quoted. Type
'Object.constants' to see a (messy) list of all constants in the environment.
If you are using binary keys or values and need to enter them in the shell, use
double-quote'd hexadecimal representation. For example:
hbase> get 't1', "key\x03\x3f\xcd"
hbase> get 't1', "key\003\023\011"
hbase> put 't1', "test\xef\xff", 'f1:', "\x01\x33\x40"
The HBase shell is the (J)Ruby IRB with the above HBase-specific commands added.
For more on the HBase Shell, see http://hbase.apache.org/docs/current/book.html
2.5 HBase簡單命令操作
#創建一個新表student
hbase(main):003:0> create 'student','info'
0 row(s) in 1.2680 seconds
#查看所有的表
hbase(main):004:0> list
TABLE
student
1 row(s) in 0.0330 seconds
#查看student的表結構
hbase(main):005:0> describe 'student'
DESCRIPTION ENABLED
'student', {NAME => 'info', DATA_BLOCK_ENCODING => 'NONE', true
BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS
=> '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL =
> '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE =
> '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'true',
BLOCKCACHE => 'true'}
1 row(s) in 0.1100 seconds
#同student表中插入一條數據
hbase(main):007:0> put 'student','mary','info:age','19'
0 row(s) in 0.0490 seconds
#從student表中取出mary的數據
hbase(main):008:0> get 'student','mary'
COLUMN CELL
info:age timestamp=1396366643298, value=19
1 row(s) in 0.0190 seconds
#讓student表失效
hbase(main):009:0> disable 'student'
0 row(s) in 1.2400 seconds
#列出所有表
hbase(main):010:0> list
TABLE
student
1 row(s) in 0.0310 seconds
#刪除student表
hbase(main):013:0> drop 'student'
0 row(s) in 1.1100 seconds
#列出所有表
hbase(main):014:0> list
TABLE
0 row(s) in 0.0400 seconds
3 Thrift安裝
安裝完成HBase後,我們還需要安裝Thrift,因爲其他語言調用HBase的時候,是通過Thrift連接的。
Thrift是需要本地編譯的,官方沒有提供二進制安裝包,首先下載thrift-0.9.1,Thrift下載頁:http://thrift.apache.org/download
3.1 下載thrift
下載Thrift有兩種方式,直接下載源代碼發行包,或者通過git下載源代碼,請選擇其中一種方式下載。
3.1.1 直接下載源代碼發行包 thrift-0.9.1.tar.gz
~ wget http://apache.fayea.com/apache-mirror/thrift/0.9.1/thrift-0.9.1.tar.gz
~ tar xvf thrift-0.9.1.tar.gz
~ mv thrift-0.9.1/ /home/conan/hadoop/
~ cd /home/conan/hadoop/
注:後文中的各種錯誤,都是這個包引起的
3.1.2 通過git下載源代碼
~ git clone https://git-wip-us.apache.org/repos/asf/thrift.git thrift-git
~ mv thrift-git/ /home/conan/hadoop/
~ cd /home/conan/hadoop/
爲了避免各種出錯,建議使用git下載源代碼安裝
3.2 通過thrift-0.9.1.tar.gz 發行包安裝Thrift
Thrift是需要本地編譯的,在Thrift解壓目錄輸入./configure,會列Thrift在當前機器所支持的語言環境。
3.2.1 安裝Thrift的依賴包
sudo apt-get install libboost-dev libboost-test-dev libboost-program-options-dev libevent-dev automake libtool flex bison pkg-config g++ libssl-dev
如果只是爲了連接rhbase,默認配置就可以了。如果除了希望支持rhbase訪問,還支持PHP,Python,C++等語言的訪問,就需要在系統中,裝一些額外的類庫。大家可以根據自己的要求,安裝對應的軟件包並設置Thrift的編譯參數。
生成配置腳本
~ ./configure
//省略部分日誌輸出
thrift 0.9.1
Building C++ Library ......... : yes
Building C (GLib) Library .... : no
Building Java Library ........ : yes
Building C# Library .......... : no
Building Python Library ...... : yes
Building Ruby Library ........ : no
Building Haskell Library ..... : no
Building Perl Library ........ : no
Building PHP Library ......... : no
Building Erlang Library ...... : no
Building Go Library .......... : no
Building D Library ........... : no
C++ Library:
Build TZlibTransport ...... : yes
Build TNonblockingServer .. : yes
Build TQTcpServer (Qt) .... : no
Java Library:
Using javac ............... : javac
Using java ................ : java
Using ant ................. : /home/conan/toolkit/ant184/bin/ant
Python Library:
Using Python .............. : /usr/bin/python
If something is missing that you think should be present,
please skim the output of configure to find the missing
component. Details are present in config.log.
我本機的已支持C++, Java與Thrift的通信。
3.2.2 增加Python語言的通信
雖然Python已被顯示支持與Thrift但在後面編譯過程中,還是缺少一些Python的庫,我們需要再增加Python的依賴庫
安裝Python的依賴包
sudo apt-get install python-all python-all-dev python-all-dbg
3.2.3 增加PHP語言的通信
安裝PHP的依賴包
sudo apt-get install php5-dev php5-cli phpunit
生成配置腳本
~ ./configure --enable-thrift_protocol
//省略部分日誌輸出
thrift 0.9.1
Building C++ Library ......... : yes
Building C (GLib) Library .... : no
Building Java Library ........ : yes
Building C# Library .......... : no
Building Python Library ...... : yes
Building Ruby Library ........ : no
Building Haskell Library ..... : no
Building Perl Library ........ : no
Building PHP Library ......... : yes
Building Erlang Library ...... : no
Building Go Library .......... : no
Building D Library ........... : no
C++ Library:
Build TZlibTransport ...... : yes
Build TNonblockingServer .. : yes
Build TQTcpServer (Qt) .... : no
Java Library:
Using javac ............... : javac
Using java ................ : java
Using ant ................. : /home/conan/toolkit/ant184/bin/ant
Python Library:
Using Python .............. : /usr/bin/python
PHP Library:
Using php-config .......... : /usr/bin/php-config
If something is missing that you think should be present,
please skim the output of configure to find the missing
component. Details are present in config.log.
我們看到Thrift的配置中,增加了對PHP語言的支持。
3.2.4 編譯和安裝
# 編譯Thrift
~ make
//省略部分日誌
make[5]: 正在進入目錄 `/home/conan/hadoop/thrift-0.9.1/lib/php/src/ext/thrift_protocol'
make[5]: *** 沒有指明目標並且找不到 makefile。 停止。
make[5]:正在離開目錄 `/home/conan/hadoop/thrift-0.9.1/lib/php/src/ext/thrift_protocol'
make[4]: *** [src/ext/thrift_protocol/modules/thrift_protocol.so] 錯誤 2
make[4]:正在離開目錄 `/home/conan/hadoop/thrift-0.9.1/lib/php'
make[3]: *** [all-recursive] 錯誤 1
make[3]:正在離開目錄 `/home/conan/hadoop/thrift-0.9.1/lib/php'
make[2]: *** [all-recursive] 錯誤 1
make[2]:正在離開目錄 `/home/conan/hadoop/thrift-0.9.1/lib'
make[1]: *** [all-recursive] 錯誤 1
make[1]:正在離開目錄 `/home/conan/hadoop/thrift-0.9.1'
make: *** [all] 錯誤 2
在make生成過程,出現PHP的編譯錯誤。從Thrift的錯誤列表中,我們可以找到錯誤描述( https://issues.apache.org/jira/browse/THRIFT-2265 ),這是由於Thrift-0.9.1發行包,打包時缺少了PHP擴展文件造成的錯誤,並在Thrift-0.9.2版本中修復。
所以,我們如果還想繼續使用Thrift-0.9.1版本,則不能支持PHP語言。
# 生成配置信息,不包括PHP模塊
~ ./configure --without-php_extension
# 編譯Thrift
~ make
編譯過程中,又出現了C++編譯錯誤。
Makefile:832: 警告:覆蓋關於目標“gen-cpp/ThriftTest.cpp”的命令
Makefile:829: 警告:忽略關於目標“gen-cpp/ThriftTest.cpp”的舊命令
/bin/bash ../../libtool --tag=CXX --mode=link g++ -Wall -g -O2 -L/usr/lib -o libtestgencpp.la ThriftTest_constants.lo ThriftTest_types.lo ../../lib/cpp/libthrift.la -lssl -lcrypto -lrt -lpthread
libtool: link: ar cru .libs/libtestgencpp.a .libs/ThriftTest_constants.o .libs/ThriftTest_types.o
ar: .libs/ThriftTest_constants.o: No such file or directory
make[3]: *** [libtestgencpp.la] 錯誤 1
make[3]:正在離開目錄 `/home/conan/hadoop/thrift-0.9.1/test/cpp'
make[2]: *** [all-recursive] 錯誤 1
make[2]:正在離開目錄 `/home/conan/hadoop/thrift-0.9.1/test'
make[1]: *** [all-recursive] 錯誤 1
make[1]:正在離開目錄 `/home/conan/hadoop/thrift-0.9.1'
make: *** [all] 錯誤 2
對於上面的2個編譯錯誤,我決定換成git源代碼的版本重新操作。
3.2 通過git源代碼安裝Thrift
運行安裝命令
# 進行thrift-git目錄
~ cd /home/conan/hadoop/thrift-git
# 複製0.9.1標籤到新分支thrift-0.9.1
~ git checkout -b thrift-0.9.1 0.9.1
# 產生配置腳本
~ ./bootstrap.sh
# 生成配置信息
~ ./configure
# 編譯Thrift
~ make
# 安裝Thrift
~ sudo make install
走了許多彎路,終於使用git源代碼版本安裝好了Thrift。
查看thrift版本
~ thrift -version
Thrift version 0.9.1
接下來,我們啓動HBase的Thrift Server服務
# 啓動HBase的Thrift服務
~ /home/conan/hadoop/hbase-0.94.18/bin/hbase-daemon.sh start thrift
starting thrift, logging to /home/conan/hadoop/hbase-0.94.18/bin/../logs/hbase-conan-thrift-master.out
# 檢查系統進程
~ jps
13838 TaskTracker
13541 JobTracker
15946 HMaster
32120 Jps
12851 NameNode
13450 SecondaryNameNode
13133 DataNode
32001 ThriftServer
15817 HQuorumPeer
16283 HRegionServer
我們看到ThriftServer已被啓動,後面我們就可以使用多種語言,通過Thrift來訪問HBase了,這樣就完成了HBase的安裝。