HBase的簡單性能測試

測試服務器爲Dell  E5410 的Debian linux 2.6，配置爲：

1、4核，2.33GHz

2、內存3G

3、SATA硬盤2T

 

web服務器：tomcat5.5

打壓工具：Apache Bench

Hadoop：0.20.2

HBase：0.20.6 ， 測試基於Hdfs 

 

測試思路：

1、因爲HBase內置了連接池，所以客戶端程序相對簡單；

2、每個請求做1000次插入；

 

連接管理器：

package lab.winston;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.HBaseAdmin;

public class HbaseManager {
	static private HbaseManager instance; // 唯一實例
	static HBaseConfiguration cfg = null;

	static synchronized public  HbaseManager getInstance(final String ip, int port, int poolSize) throws IOException {
		if (instance == null) {
			instance = new HbaseManager(ip,port,poolSize);
		}		
		return instance;
	}

	private HbaseManager() {
	}
	
	private HbaseManager(final String ip, int port, int poolSize) throws IOException  {
		init(ip,port,poolSize);
	}

	public void init(final String ip, int port, int poolSize)
			throws IOException {
		if (cfg == null) {
	        Configuration HBASE_CONFIG = new Configuration();
	        HBASE_CONFIG.set("hbase.zookeeper.quorum", ip);
	        HBASE_CONFIG.set("hbase.zookeeper.property.clientPort", String.valueOf(port));
	        cfg = new HBaseConfiguration(HBASE_CONFIG);
	        
	        HBaseAdmin admin = new HBaseAdmin(cfg);
	        if (admin.tableExists("test")) {
	            System.out.println("table   Exists!!!");
	        }
	        else{
	            HTableDescriptor tableDesc = new HTableDescriptor("test");
	            tableDesc.addFamily(new HColumnDescriptor("name"));
	            admin.createTable(tableDesc);
	            System.out.println("create table ok .");
	        }
		}
	}
}

測試Servlet：

package lab.winston;

import java.io.IOException;
import java.util.UUID;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.util.Bytes;

public class InsertHbaseServlet extends HttpServlet {

	public void doGet(HttpServletRequest req, HttpServletResponse resp)
			throws ServletException, IOException {
		Integer insertNum = Integer.valueOf(req.getParameter("insertNum")
				.toString());
		resp.setContentType("text/html;charset=UTF-8");
		resp.setHeader("Cache-Control", "no-cache");

		HBaseConfiguration cfg = HbaseManager.getInstance("127.0.0.1", 2181, 0).cfg;
		HTable table = new HTable(cfg, "test");
		for(int i=0;i<insertNum;i++){
			String uuid = UUID.randomUUID().toString();
			Put put = new Put(Bytes.toBytes(uuid));
			put.add(Bytes.toBytes("name"), Bytes.toBytes("id"), Bytes
					.toBytes("10000"));		
			put.add(Bytes.toBytes("name"), Bytes.toBytes("value"), Bytes
					.toBytes("this is hbase insert!"));
			table.put(put);			
		}
		System.out.println("add data ok. insertNum:"+insertNum);
		resp.getWriter().write(cfg + ": Insert hbase successed!");
	}

	public void doPost(HttpServletRequest request, HttpServletResponse response)
			throws ServletException, IOException {
	}
}

5個線程併發，插入20萬記錄：

ab -n 200 -c 5 http://localhost:8080/labWeb/InsertHbase.do?insertNum=1000

結果爲：

Concurrency Level:      5
Time taken for tests:   40.251845 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      66000 bytes
HTML transferred:       28600 bytes
Requests per second:    4.97 [#/sec] (mean)
Time per request:       1006.296 [ms] (mean)
Time per request:       201.259 [ms] (mean, across all concurrent requests)
Transfer rate:          1.59 [Kbytes/sec] received

cpu負載：90%以上

耗時40秒，平均每秒插入（QPS）爲5000。

 

10個線程併發，插入20萬記錄：

ab -n 200 -c 10 http://localhost:8080/labWeb/InsertHbase.do?insertNum=1000

結果爲：

Concurrency Level:      10
Time taken for tests:   56.608753 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      66000 bytes
HTML transferred:       28600 bytes
Requests per second:    3.53 [#/sec] (mean)
Time per request:       2830.438 [ms] (mean)
Time per request:       283.044 [ms] (mean, across all concurrent requests)
Transfer rate:          1.13 [Kbytes/sec] received

cpu負載：90%以上

耗時56秒，平均每秒插入（QPS）爲3600。