這兩天主要是在最新版本的YCSB的基礎上用mapkeeperclient打包並連接mapkeeper端的server。不得不說ycsb在創建mapkeeper client的時候就是有問題的。我踩了很多坑,我要記下來做筆記。
首先是下載最新的YCSB:
https://github.com/brianfrankcooper/YCSB.git
然後執行下面命令:
cd YCSB
vim pom.xml
在pom.xml文件中把有關mapkeeper的部分取消註釋:
取消註釋部分:
<mapkeeper.version>1.0</mapkeeper.version>
<module>mapkeeper</module>
在mapkeeper目錄下的pom.xml文件中添加:
<dependency>
<groupId>org.apache.thrift</groupId>
<artifactId>libthrift</artifactId>
<version>0.6.1</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.25</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.25</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.25</version>
</dependency>
敲黑板的重點是這裏thrift一定要是0.6.1版本的!!!我試過最新0.12.0還有0.8.0都不行。這纔是我這幾天一直mvn打包的時候報錯的原因,如果不用這個會報以下錯誤:
Exception in thread "Thread-2" java.lang.IncompatibleClassChangeError: Implementing class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.yahoo.ycsb.db.MapKeeperClient.init(MapKeeperClient.java:69)
at com.yahoo.ycsb.DBWrapper.init(DBWrapper.java:86)
at com.yahoo.ycsb.ClientThread.run(ClientThread.java:91)
at java.lang.Thread.run(Thread.java:748)
我在網上搜索了好久也不知道上面是啥錯誤,原來是mapkeeper在mvn打包的時候只支持0.6.1版本。然後就是對於MapKeeperClient的修改了,在這裏最新版的MapKeeperClient.java文件中是有一些問題的就是把一些本該是用com.yahoo.ycsb.Status類型的表示用常用的int表示了。所以會報checkstyle的錯誤。同時格式也會不對顯示類似於本該空着幾格實際空了幾格的報錯,因爲可能是tab問題。所以需要手動修改。修改如下完整的mapkeeperclient文件:
/**
* Copyright (c) 2012 YCSB contributors. All rights reserved.
* <p>
* Licensed under the Apache License, Version 2.0 (the "License"); you
* may not use this file except in compliance with the License. You
* may obtain a copy of the License at
* <p>
* http://www.apache.org/licenses/LICENSE-2.0
* <p>
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
* implied. See the License for the specific language governing
* permissions and limitations under the License. See accompanying
* LICENSE file.
*/
package com.yahoo.ycsb.db;
import java.nio.ByteBuffer;
import java.util.HashMap;
import java.util.Properties;
import java.util.Set;
import java.util.Vector;
import java.util.Map;
import org.apache.thrift.TException;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.apache.thrift.protocol.TProtocol;
import org.apache.thrift.transport.TFramedTransport;
import org.apache.thrift.transport.TSocket;
import org.apache.thrift.transport.TTransport;
import com.yahoo.mapkeeper.BinaryResponse;
import com.yahoo.mapkeeper.MapKeeper;
import com.yahoo.mapkeeper.Record;
import com.yahoo.mapkeeper.RecordListResponse;
import com.yahoo.mapkeeper.ResponseCode;
import com.yahoo.mapkeeper.ScanOrder;
import com.yahoo.ycsb.ByteIterator;
import com.yahoo.ycsb.DB;
import com.yahoo.ycsb.StringByteIterator;
import com.yahoo.ycsb.workloads.CoreWorkload;
import com.yahoo.ycsb.Status;
/**
* YCSB binding for <a href="http://redis.io/">Redis</a>.
*
* See {@code redis/README.md} for details.
*/
public class MapKeeperClient extends DB {
private static final String HOST = "mapkeeper.host";
private static final String HOST_DEFAULT = "localhost";
private static final String PORT = "mapkeeper.port";
private static final String PORT_DEFAULT = "9090";
private MapKeeper.Client c;
private boolean writeallfields;
private static boolean initteddb = false;
private static synchronized void initDB(Properties p, MapKeeper.Client c) throws TException {
if (!initteddb) {
initteddb = true;
c.addMap(p.getProperty(CoreWorkload.TABLENAME_PROPERTY, CoreWorkload.TABLENAME_PROPERTY_DEFAULT));
}
}
public void init() {
String host = getProperties().getProperty(HOST, HOST_DEFAULT);
int port = Integer.parseInt(getProperties().getProperty(PORT, PORT_DEFAULT));
TTransport tr = new TFramedTransport(new TSocket(host, port));
TProtocol proto = new TBinaryProtocol(tr);
c = new MapKeeper.Client(proto);
try {
tr.open();
initDB(getProperties(), c);
} catch (TException e) {
throw new RuntimeException(e);
}
writeallfields = Boolean.parseBoolean(getProperties().getProperty(CoreWorkload.WRITE_ALL_FIELDS_PROPERTY,
CoreWorkload.WRITE_ALL_FIELDS_PROPERTY_DEFAULT));
}
ByteBuffer encode(HashMap<String, ByteIterator> values) {
int len = 0;
for (Map.Entry<String, ByteIterator> entry : values.entrySet()) {
len += (entry.getKey().length() + 1 + entry.getValue().bytesLeft() + 1);
}
byte[] array = new byte[len];
int i = 0;
for (Map.Entry<String, ByteIterator> entry : values.entrySet()) {
for (int j = 0; j < entry.getKey().length(); j++) {
array[i] = (byte) entry.getKey().charAt(j);
i++;
}
array[i] = '\t';
// XXX would like to use sane delimiter (null, 254, 255, ...) but java makes this nearly impossible
i++;
ByteIterator v = entry.getValue();
i = v.nextBuf(array, i);
array[i] = '\t';
i++;
}
array[array.length - 1] = 0;
ByteBuffer buf = ByteBuffer.wrap(array);
buf.rewind();
return buf;
}
void decode(Set<String> fields, String tups, HashMap<String, ByteIterator> tup) {
String[] tok = tups.split("\\t");
if (tok.length == 0) {
throw new IllegalStateException("split returned empty array!");
}
for (int i = 0; i < tok.length; i += 2) {
if (fields == null || fields.contains(tok[i])) {
if (tok.length < i + 2) {
throw new IllegalStateException("Couldn't parse tuple <" + tups + "> at index " + i);
}
if (tok[i] == null || tok[i + 1] == null) {
throw new NullPointerException("Key is " + tok[i] + " val is + " + tok[i + 1]);
}
tup.put(tok[i], new StringByteIterator(tok[i + 1]));
}
}
if (tok.length == 0) {
System.err.println("Empty tuple: " + tups);
}
}
Status ycsbThriftRet(BinaryResponse succ, ResponseCode zero, ResponseCode one) {
return ycsbThriftRet(succ.responseCode, zero, one);
}
Status ycsbThriftRet(ResponseCode rc, ResponseCode zero, ResponseCode one) {
return rc == zero ? Status.OK : rc == one ? Status.NOT_FOUND : Status.ERROR;
}
ByteBuffer bufStr(String str) {
ByteBuffer buf = ByteBuffer.wrap(str.getBytes());
return buf;
}
String strResponse(BinaryResponse buf) {
return new String(buf.value.array());
}
@Override
public Status read(String table, String key, Set<String> fields,
Map<String, ByteIterator> result) {
try {
ByteBuffer buf = bufStr(key);
BinaryResponse succ = c.get(table, buf);
Status ret = ycsbThriftRet(
succ,
ResponseCode.RecordExists,
ResponseCode.RecordNotFound);
if (ret == Status.OK) {
decode(fields, strResponse(succ), (HashMap<String, ByteIterator>)result);
}
return ret;
} catch (TException e) {
e.printStackTrace();
return Status.ERROR;
}
}
@Override
public Status scan(String table, String startkey, int recordcount,
Set<String> fields, Vector<HashMap<String, ByteIterator>> result) {
try {
//XXX what to pass in for nulls / zeros?
RecordListResponse res = c.scan(table, ScanOrder.Ascending, bufStr(startkey), true, null, false, recordcount, 0);
Status ret = ycsbThriftRet(res.responseCode, ResponseCode.Success, ResponseCode.ScanEnded);
if (ret == Status.OK) {
for (Record r : res.records) {
HashMap<String, ByteIterator> tuple = new HashMap<String, ByteIterator>();
// Note: r.getKey() and r.getValue() call special helper methods that trim the buffer
// to an appropriate length, and memcpy it to a byte[]. Trying to manipulate the ByteBuffer
// directly leads to trouble.
tuple.put("key", new StringByteIterator(new String(r.getKey())));
decode(fields, new String(r.getValue())/*strBuf(r.bufferForValue())*/, tuple);
result.add(tuple);
}
}
return ret;
} catch (TException e) {
e.printStackTrace();
return Status.ERROR;
}
}
@Override
public Status update(String table, String key,
Map<String, ByteIterator> values) {
try {
if (!writeallfields) {
HashMap<String, ByteIterator> oldval = new HashMap<String, ByteIterator>();
read(table, key, null, oldval);
for (Map.Entry<String, ByteIterator> entry : values.entrySet()) {
oldval.put(entry.getKey(), entry.getValue());
}
values = oldval;
}
ResponseCode succ = c.update(table, bufStr(key), encode((HashMap<String, ByteIterator>)values));
return ycsbThriftRet(succ, ResponseCode.RecordExists, ResponseCode.RecordNotFound);
} catch (TException e) {
e.printStackTrace();
return Status.ERROR;
}
}
@Override
public Status insert(String table, String key, Map<String, ByteIterator> values) {
try {
Status ret = ycsbThriftRet(c.insert(table, bufStr(key), encode((HashMap<String, ByteIterator>)values)),
ResponseCode.Success, ResponseCode.RecordExists);
return ret;
} catch (TException e) {
e.printStackTrace();
return Status.ERROR;
}
}
@Override
public Status delete(String table, String key) {
try {
return ycsbThriftRet(c.remove(table, bufStr(key)), ResponseCode.Success, ResponseCode.RecordExists);
} catch (TException e) {
e.printStackTrace();
return Status.ERROR;
}
}
}
同時在該目錄下添加一個package-info.java文件。內容是(雖然很簡單但是對於打包很重要)
package com.yahoo.ycsb.db;
然後就是使用mvn打包了:
編譯所有數據庫的話:
mvn clean package
只編譯mapkeeper的話:
mvn -pl com.yahoo.ycsb:mapkeeper-binding -am clean package
然後就可以編譯成功了。
然後就是修改YCSB/bin中的bindings.properties:
#添加
mapkeeper:com.yahoo.ycsb.db.MapKeeperClient
然後就是修改YCSB/bin中的ycsb:
#添加
"mapkeeper" : "com.yahoo.ycsb.db.MapKeeperClient",
然後就大功告成了,在clinet端:
bin/ycsb load mapkeeper -s -P workloads/workloada -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090"
bin/ycsb run mapkeeper -s -P workloads/workloada -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090"
在mapkeeper server端開啓:
./mapkeeper_RHDB 0 1 1 /home/clever_wr/data
不報錯就顯示成功連接