對於分佈式系統而言，RPC毫無疑問是非常重要的，其負責機器之間的通信，而在hadoop中，相應的RPC調用更是不計其數，這裏僅僅簡單提供一個Hadoop IPC的一個小例子，供大家研究。
（1）繼承自VersionedProtocol的接口
import org.apache.hadoop.ipc.VersionedProtocol;
public interface IPCQueryStatus extends VersionedProtocol {
IPCFileStatus getFileStatus(String fileName);
}在hadoop內，遠程調用的接口都需要繼承本接口，讓我們看下VersionedProtocol的源碼和其中的方法，非常簡單：
/**
* Superclass of all protocols that use Hadoop RPC.
* Subclasses of this interface are also supposed to have
* a static final long versionID field.
*/
public interface VersionedProtocol {
/**
* Return protocol version corresponding to protocol interface.
* @param protocol The classname of the protocol interface
* @param clientVersion The version of the protocol that the client speaks
* @return the version that the server will speak
*/
public long getProtocolVersion(String protocol,
long clientVersion) throws IOException;
}其中就只有一個方法，getProtocolVersion，兩個參數，分別是協議接口的類名，與客戶端接口協議的版本，返回值則是服務器端接口的版本；這種設計考量到了客戶端與服務端接口協議不一致的問題。
（2）接口的實現類
public class IPCQueryStatusImpl implements IPCQueryStatus {
/**
* @description:
* @author:
* @param:
* @return:
**/
@Override
public long getProtocolVersion(String protocol, long clientVersion)
throws IOException {
// TODO Auto-generated method stub
System.out.println("protocol:" + protocol);
System.out.println("clientVersion:" + clientVersion);
return IPCQueryServer.IPV_VER;
}
/**
* @description:
* @author:
* @param:
* @return:
**/
@Override
public IPCFileStatus getFileStatus(String fileName) {
// TODO Auto-generated method stub
return new IPCFileStatus(fileName);
}
}這是毫無疑問的，必須有實現類才能執行方法，其中實現了VersionedProtocol中的默認方法，實現了自定義接口中的其它方法。
（3）服務端
public class IPCQueryServer {
public static final int IPC_PORT = 32121;
public static final long IPV_VER = 5473l;
public static void main(String[] args) throws IOException {
IPCQueryStatusImpl query = new IPCQueryStatusImpl();
Server server = RPC.getServer(query, "0.0.0.0", IPC_PORT,
new Configuration());
server.start();
}
}開啓客戶端，能夠看到，這裏牽涉到了一個RPC類，非常關鍵，下面看下其源碼，其源碼非常長，單單截取出比較關鍵的地方來看：
/**
* Construct a server for a protocol implementation instance listening on a
* port and address.
*/
public static Server getServer(final Object instance,
final String bindAddress, final int port, Configuration conf)
throws IOException {
return getServer(instance, bindAddress, port, 1, false, conf);
}這是我們新建服務器的代碼，發現其中調用了getServer方法，一直點下去，發現調用的是這個方法：
@SuppressWarnings("unchecked")
protected Server(String bindAddress, int port,
Class<? extends Writable> paramClass, int handlerCount,
Configuration conf, String serverName, SecretManager<? extends TokenIdentifier> secretManager)
throws IOException {
this.bindAddress = bindAddress;
this.conf = conf;
this.port = port;
this.paramClass = paramClass;
this.handlerCount = handlerCount;
this.socketSendBufferSize = 0;
this.maxQueueSize = handlerCount * conf.getInt(
IPC_SERVER_HANDLER_QUEUE_SIZE_KEY,
IPC_SERVER_HANDLER_QUEUE_SIZE_DEFAULT);
this.maxRespSize = conf.getInt(IPC_SERVER_RPC_MAX_RESPONSE_SIZE_KEY,
IPC_SERVER_RPC_MAX_RESPONSE_SIZE_DEFAULT);
this.readThreads = conf.getInt(
IPC_SERVER_RPC_READ_THREADS_KEY,
IPC_SERVER_RPC_READ_THREADS_DEFAULT);
this.callQueue = new LinkedBlockingQueue<Call>(maxQueueSize);
this.maxIdleTime = 2*conf.getInt("ipc.client.connection.maxidletime", 1000);
this.maxConnectionsToNuke = conf.getInt("ipc.client.kill.max", 10);
this.thresholdIdleConnections = conf.getInt("ipc.client.idlethreshold", 4000);
this.secretManager = (SecretManager<TokenIdentifier>) secretManager;
this.authorize =
conf.getBoolean(HADOOP_SECURITY_AUTHORIZATION, false);
this.isSecurityEnabled = UserGroupInformation.isSecurityEnabled();
// Start the listener here and let it bind to the port
listener = new Listener();
this.port = listener.getAddress().getPort();
this.rpcMetrics = RpcInstrumentation.create(serverName, this.port);
this.tcpNoDelay = conf.getBoolean("ipc.server.tcpnodelay", false);
// Create the responder here
responder = new Responder();
if (isSecurityEnabled) {
SaslRpcServer.init(conf);
}
}利用我們的傳入的地址和端口號，新建了一個服務端，內部代碼暫時沒有過分深究，其他的賦值比較容易看懂，我們可以看到，這裏利用了一個Listener和Responder有點陌生，看看源碼如何：
先放個Listener的源碼：
/** Listens on the socket. Creates jobs for the handler threads*/
private class Listener extends Thread {
private ServerSocketChannel acceptChannel = null; //the accept channel
private Selector selector = null; //the selector that we use for the server
private Reader[] readers = null;
private int currentReader = 0;
private InetSocketAddress address; //the address we bind at
private Random rand = new Random();
private long lastCleanupRunTime = 0; //the last time when a cleanup connec-
//-tion (for idle connections) ran
private long cleanupInterval = 10000; //the minimum interval between
//two cleanup runs
private int backlogLength = conf.getInt("ipc.server.listen.queue.size", 128);
private ExecutorService readPool;
public Listener() throws IOException {
address = new InetSocketAddress(bindAddress, port);
// Create a new server socket and set to non blocking mode
acceptChannel = ServerSocketChannel.open();
acceptChannel.configureBlocking(false);
// Bind the server socket to the local host and port
bind(acceptChannel.socket(), address, backlogLength);
port = acceptChannel.socket().getLocalPort(); //Could be an ephemeral port
// create a selector;
selector= Selector.open();
readers = new Reader[readThreads];
readPool = Executors.newFixedThreadPool(readThreads);
for (int i = 0; i < readThreads; i++) {
Selector readSelector = Selector.open();
Reader reader = new Reader(readSelector);
readers[i] = reader;
readPool.execute(reader);
}
// Register accepts on the server socket with the selector.
acceptChannel.register(selector, SelectionKey.OP_ACCEPT);
this.setName("IPC Server listener on " + port);
this.setDaemon(true);
}服務器端使用這個Listener監聽客戶端的連接，可以看到，這裏利用的是Java NIO的機制，並利用內部的線程池加以處理，代碼很容易看懂，看不懂的可以認真研究下Java NIO的機制。
再來個Responder的源碼：
private class Responder extends Thread {
private Selector writeSelector;
private int pending; // connections waiting to register
final static int PURGE_INTERVAL = 900000; // 15mins
Responder() throws IOException {
this.setName("IPC Server Responder");
this.setDaemon(true);
writeSelector = Selector.open(); // create a selector
pending = 0;
}
@Override
public void run() {
LOG.info(getName() + ": starting");
SERVER.set(Server.this);
long lastPurgeTime = 0; // last check for old calls.
while (running) {
try {
waitPending(); // If a channel is being registered, wait.
writeSelector.select(PURGE_INTERVAL);
Iterator<SelectionKey> iter = writeSelector.selectedKeys().iterator();
while (iter.hasNext()) {
SelectionKey key = iter.next();
iter.remove();
try {
if (key.isValid() && key.isWritable()) {
doAsyncWrite(key);
}
} catch (IOException e) {
LOG.info(getName() + ": doAsyncWrite threw exception " + e);
}
}
long now = System.currentTimeMillis();
if (now < lastPurgeTime + PURGE_INTERVAL) {
continue;
}
lastPurgeTime = now;
//
// If there were some calls that have not been sent out for a
// long time, discard them.
//
LOG.debug("Checking for old call responses.");
ArrayList<Call> calls;
// get the list of channels from list of keys.
synchronized (writeSelector.keys()) {
calls = new ArrayList<Call>(writeSelector.keys().size());
iter = writeSelector.keys().iterator();
while (iter.hasNext()) {
SelectionKey key = iter.next();
Call call = (Call)key.attachment();
if (call != null && key.channel() == call.connection.channel) {
calls.add(call);
}
}
}
for(Call call : calls) {
try {
doPurge(call, now);
} catch (IOException e) {
LOG.warn("Error in purging old calls " + e);
}
}
} catch (OutOfMemoryError e) {
//
// we can run out of memory if we have too many threads
// log the event and sleep for a minute and give
// some thread(s) a chance to finish
//
LOG.warn("Out of Memory in server select", e);
try { Thread.sleep(60000); } catch (Exception ie) {}
} catch (Exception e) {
LOG.warn("Exception in Responder " +
StringUtils.stringifyException(e));
}
}
LOG.info("Stopping " + this.getName());
}在日常的使用中，我們傳入對應的地址，端口號，即可新建一個服務端供我們使用，建議深入細節，加深自己對於NIO使用的理解。
（4）客戶端
服務端OK了，我們要新建客戶端來對連接服務端：
public class IPCQueryClient {
public static void main(String[] args) {
try {
// 準備一個socket地址
InetSocketAddress addr = new InetSocketAddress("localhost",
IPCQueryServer.IPC_PORT);
IPCQueryStatus query = (IPCQueryStatus) RPC.getProxy(
IPCQueryStatus.class, IPCQueryServer.IPV_VER, addr,
new Configuration());
System.out.println(query.getFileStatus("/test").getFilename());
RPC.stopProxy(query);
} catch (Exception e) {
e.printStackTrace();
}
}
}可以看出，這裏我們還是利用了RPC來獲取遠程服務，所以，對於Hadoop的遠程調用來說，代碼的精華部分都在RPC內，我們看下getProxy的代碼：
/**
* Construct a client-side proxy object that implements the named protocol,
* talking to a server at the named address.
*/
public static VersionedProtocol getProxy(
Class<? extends VersionedProtocol> protocol, long clientVersion,
InetSocketAddress addr, UserGroupInformation ticket,
Configuration conf, SocketFactory factory, int rpcTimeout)
throws IOException {
if (UserGroupInformation.isSecurityEnabled()) {
SaslRpcServer.init(conf);
}
VersionedProtocol proxy = (VersionedProtocol) Proxy.newProxyInstance(
protocol.getClassLoader(), new Class[] { protocol },
new Invoker(protocol, addr, ticket, conf, factory, rpcTimeout));
long serverVersion = proxy.getProtocolVersion(protocol.getName(),
clientVersion);
if (serverVersion == clientVersion) {
return proxy;
} else {
throw new VersionMismatch(protocol.getName(), clientVersion,
serverVersion);
}
}一路下來，到了這部分代碼，可以看到，裏面有對於客戶端版本和服務器端版本是否一致的判斷，如果不一致，就會拋出版本不一致的異常。
同時還可以發現，在返回結果之前，還有一次權限判斷，對於傳入的UserGroupInformation，其實就相當於我們的用戶組信息，需要檢測其是否有調用接口的權限。
我們這裏可以看到，其實返回的代碼，完全採用的就是Java動態代理的形式來做的。
綜上所述：想要研究清楚Hadoop IPC的機制，必須對Java之NIO和Java動態代理了解透徹，這是精髓。
（5）上面的代碼還缺少了一個IPCFileStatus的類，代碼如下：
public class IPCFileStatus implements Writable {
private String filename;
public IPCFileStatus() {
}
/**
* @param filename
*/
public IPCFileStatus(String filename) {
this.filename = filename;
}
/**
* @return the filename
*/
public String getFilename() {
return filename;
}
/**
* @param filename
* the filename to set
*/
public void setFilename(String filename) {
this.filename = filename;
}
/**
* @description:
* @author:
* @param:
* @return:
**/
@Override
public void write(DataOutput out) throws IOException {
// TODO Auto-generated method stub
Text.writeString(out, filename);
}
/**
* @description:
* @author:
* @param:
* @return:
**/
@Override
public void readFields(DataInput in) throws IOException {
// TODO Auto-generated method stub
this.filename = Text.readString(in);
}
@Override
public String toString() {
return filename;
}
}日常使用下，如此安排即可，但想要深入瞭解Hadoop IPC的機制，還是需要認真瞭解NIO和動態代理。
