對網絡編程來說，最基本的三要素是IO， 協議（編解碼），服務器端線程模型。這篇來看看ZooKeeper是如何實現高性能的網絡程序。
IO模型
ZooKeeper默認提供了兩種網絡IO的實現，一個是Java原生的NIO，一個是基於Netty的IO。先從ServerCnxn這個抽象類看起，它表示一個從客戶端到服務器端的網絡連接。ServerCnxn實現了Stat服務器端統計接口，Watcher接口。 Watcher接口裏面定義了KeeperState和EventType這兩個枚舉類型。
ServerCnxn有兩個默認實現類，一個是基於JDK原生NIO的NIOServerCnxn，一個是基於Netty的NettyServerCnxn。
// ServerCnxn的屬性
public abstract class ServerCnxn implements Stats, Watcher {
protected abstract ServerStats serverStats();
protected final Date established = new Date();
protected final AtomicLong packetsReceived = new AtomicLong();
protected final AtomicLong packetsSent = new AtomicLong();
protected long minLatency;
protected long maxLatency;
protected String lastOp;
protected long lastCxid;
protected long lastZxid;
protected long lastResponseTime;
protected long lastLatency;
protected long count;
protected long totalLatency;
}
重點看一下NIOServerCnxn，它處理和客戶端的連接。它的唯一構造函數需要4個參數：ZooKeeperServer， SocketChannel， SelectionKey， NIOServerCnxnFactory。NIOServerCnxn本質上是對SocketChannel的封裝，它提供了對SocketChannel讀寫的方法。
public NIOServerCnxn(ZooKeeperServer zk, SocketChannel sock,
SelectionKey sk, NIOServerCnxnFactory factory) throws IOException {
this.zkServer = zk;
this.sock = sock;
this.sk = sk;
this.factory = factory;
if (this.factory.login != null) {
this.zooKeeperSaslServer = new ZooKeeperSaslServer(factory.login);
}
if (zk != null) {
outstandingLimit = zk.getGlobalOutstandingLimit();
}
sock.socket().setTcpNoDelay(true);
/* set socket linger to false, so that socket close does not
* block */
sock.socket().setSoLinger(false, -1);
InetAddress addr = ((InetSocketAddress) sock.socket()
.getRemoteSocketAddress()).getAddress();
authInfo.add(new Id("ip", addr.getHostAddress()));
sk.interestOps(SelectionKey.OP_READ);
}NIOServerCnxn的核心方法是doIO, 它實現了SelectionKey被Selector選出後，SocketChannel如何進行讀寫
SocketChannel從客戶端讀數據的過程：
1. NIOServerCnxc維護了兩個讀數據的ByteBuffer, 一個是 lenBuffer = ByteBuffer.allocate(4)， 4個字節的ByteBuffer，表示是否是4個字符的命令消息，比如ruok, conf這種命令。ByteBuffer incomingBuffer表示用來存放讀數據ByteBuffer, 初始狀態下incomingBuffer指向lenBuffer
2. SocketChannel先向incomingBuffer寫入數據，如果寫入的長度小於0就拋異常。如果正常寫入，並且incomingBuffer寫滿了，如果incomingBuffer是指向lenBuffer，表示這次讀的是4個字節的長度。
3. readLength方法會判斷是否是4字符命令，首先調用checkFourLetterWord來判斷是否是4字符命令
4. 在checkFourLetterWord中，如果是4字符命令，就調用對應的線程CommandThread，啓動單獨的線程去執行對應的命令，具體的如何寫會在後面說.
如果不是4字符命令，就會incomingBuffer分配對應長度的ByteBuffer， incomingBuffer = ByteBuffer.allocate(len); 不在指向lenBuffer
5. 如果不是4字符命令，進入到readPayload分支。在readPayload判斷incomingBuffer是否滿包，如果不是，就嘗試讀一次SocketChanel。如果這時候滿包了，就調用flip方法切換到讀模式，如果是第一次讀到請求，就進入readConnectRequest，如果不是就進入到readRequest。 最後 incomingBuffer = lenBuffer; 再次指向lenBuffer，讀下一個請求。
void doIO(SelectionKey k) throws InterruptedException {
try {
if (isSocketOpen() == false) {
LOG.warn("trying to do i/o on a null socket for session:0x"
+ Long.toHexString(sessionId));
return;
}
if (k.isReadable()) {
int rc = sock.read(incomingBuffer);
if (rc < 0) {
throw new EndOfStreamException(
"Unable to read additional data from client sessionid 0x"
+ Long.toHexString(sessionId)
+ ", likely client has closed socket");
}
if (incomingBuffer.remaining() == 0) {
boolean isPayload;
if (incomingBuffer == lenBuffer) { // start of next request
incomingBuffer.flip();
isPayload = readLength(k);
incomingBuffer.clear();
} else {
// continuation
isPayload = true;
}
if (isPayload) { // not the case for 4letterword
readPayload();
}
else {
// four letter words take care
// need not do anything else
return;
}
}
}
............
}
private boolean readLength(SelectionKey k) throws IOException {
// Read the length, now get the buffer
int len = lenBuffer.getInt();
if (!initialized && checkFourLetterWord(sk, len)) {
return false;
}
if (len < 0 || len > BinaryInputArchive.maxBuffer) {
throw new IOException("Len error " + len);
}
if (zkServer == null) {
throw new IOException("ZooKeeperServer not running");
}
incomingBuffer = ByteBuffer.allocate(len);
return true;
}
private boolean checkFourLetterWord(final SelectionKey k, final int len)
throws IOException
{
// We take advantage of the limited size of the length to look
// for cmds. They are all 4-bytes which fits inside of an int
String cmd = cmd2String.get(len);
if (cmd == null) {
return false;
}
LOG.info("Processing " + cmd + " command from "
+ sock.socket().getRemoteSocketAddress());
packetReceived();
/** cancel the selection key to remove the socket handling
* from selector. This is to prevent netcat problem wherein
* netcat immediately closes the sending side after sending the
* commands and still keeps the receiving channel open.
* The idea is to remove the selectionkey from the selector
* so that the selector does not notice the closed read on the
* socket channel and keep the socket alive to write the data to
* and makes sure to close the socket after its done writing the data
*/
if (k != null) {
try {
k.cancel();
} catch(Exception e) {
LOG.error("Error cancelling command selection key ", e);
}
}
final PrintWriter pwriter = new PrintWriter(
new BufferedWriter(new SendBufferWriter()));
if (len == ruokCmd) {
RuokCommand ruok = new RuokCommand(pwriter);
ruok.start();
return true;
} else if (len == getTraceMaskCmd) {
TraceMaskCommand tmask = new TraceMaskCommand(pwriter);
tmask.start();
return true;
} else if (len == setTraceMaskCmd) {
int rc = sock.read(incomingBuffer);
if (rc < 0) {
throw new IOException("Read error");
}
incomingBuffer.flip();
long traceMask = incomingBuffer.getLong();
ZooTrace.setTextTraceLevel(traceMask);
SetTraceMaskCommand setMask = new SetTraceMaskCommand(pwriter, traceMask);
setMask.start();
return true;
} else if (len == enviCmd) {
EnvCommand env = new EnvCommand(pwriter);
env.start();
return true;
} else if (len == confCmd) {
ConfCommand ccmd = new ConfCommand(pwriter);
ccmd.start();
return true;
} else if (len == srstCmd) {
StatResetCommand strst = new StatResetCommand(pwriter);
strst.start();
return true;
} else if (len == crstCmd) {
CnxnStatResetCommand crst = new CnxnStatResetCommand(pwriter);
crst.start();
return true;
} else if (len == dumpCmd) {
DumpCommand dump = new DumpCommand(pwriter);
dump.start();
return true;
} else if (len == statCmd || len == srvrCmd) {
StatCommand stat = new StatCommand(pwriter, len);
stat.start();
return true;
} else if (len == consCmd) {
ConsCommand cons = new ConsCommand(pwriter);
cons.start();
return true;
} else if (len == wchpCmd || len == wchcCmd || len == wchsCmd) {
WatchCommand wcmd = new WatchCommand(pwriter, len);
wcmd.start();
return true;
} else if (len == mntrCmd) {
MonitorCommand mntr = new MonitorCommand(pwriter);
mntr.start();
return true;
} else if (len == isroCmd) {
IsroCommand isro = new IsroCommand(pwriter);
isro.start();
return true;
}
return false;
}
private void readPayload() throws IOException, InterruptedException {
if (incomingBuffer.remaining() != 0) { // have we read length bytes?
int rc = sock.read(incomingBuffer); // sock is non-blocking, so ok
if (rc < 0) {
throw new EndOfStreamException(
"Unable to read additional data from client sessionid 0x"
+ Long.toHexString(sessionId)
+ ", likely client has closed socket");
}
}
if (incomingBuffer.remaining() == 0) { // have we read length bytes?
packetReceived();
incomingBuffer.flip();
if (!initialized) {
readConnectRequest();
} else {
readRequest();
}
lenBuffer.clear();
incomingBuffer = lenBuffer;
}
}
NIOServerCnxn寫數據的過程如下：
1. 創建一個LinkedBlockingQueue<ByteBuffer>類型的outgoingBuffers來優化寫，可以一次寫多個ByteBuffer
2. 如果SelectionKey是因爲寫消息被Selector選中 的，先判斷outgoingBuffers的長度是否大於0，如果大於0，就把outgoingBuffers中的ByteBuffer的數據都複製到factory.directBuffer這個直接內存的緩衝區中，如果directBuffer滿了或者outgoingBuffers都已經複製到directBuffer了，就調用它的flip方法把它切換到讀模式，然後把它的數據寫入到SocketChannel中去。
由此可見，每次寫的時候，都是從directBuffer寫到SocketChannel中去的，利用直接內存優化了寫操作。
寫完後清理一下outgoingBuffers，把已經寫完的ByteBuffer清理掉
3. 如果outgoingBuffers都寫完了，就把SocketChannel切換到讀模式中，關閉對寫標誌位的監聽。如果沒寫完，繼續監聽寫請求。
void doIO(SelectionKey k) throws InterruptedException {
try {
if (isSocketOpen() == false) {
LOG.warn("trying to do i/o on a null socket for session:0x"
+ Long.toHexString(sessionId));
return;
}
.......
if (k.isWritable()) {
if (outgoingBuffers.size() > 0) {
ByteBuffer directBuffer = factory.directBuffer;
directBuffer.clear();
for (ByteBuffer b : outgoingBuffers) {
if (directBuffer.remaining() < b.remaining()) {
b = (ByteBuffer) b.slice().limit(
directBuffer.remaining());
}
int p = b.position();
directBuffer.put(b);
b.position(p);
if (directBuffer.remaining() == 0) {
break;
}
}
directBuffer.flip();
int sent = sock.write(directBuffer);
ByteBuffer bb;
// Remove the buffers that we have sent
while (outgoingBuffers.size() > 0) {
bb = outgoingBuffers.peek();
if (bb == ServerCnxnFactory.closeConn) {
throw new CloseRequestException("close requested");
}
int left = bb.remaining() - sent;
if (left > 0) {
bb.position(bb.position() + sent);
break;
}
packetSent();
sent -= bb.remaining();
outgoingBuffers.remove();
}
// ZooLog.logTraceMessage(LOG,
// ZooLog.CLIENT_DATA_PACKET_TRACE_MASK, "after send,
// outgoingBuffers.size() = " + outgoingBuffers.size());
}
synchronized(this.factory){
if (outgoingBuffers.size() == 0) {
if (!initialized
&& (sk.interestOps() & SelectionKey.OP_READ) == 0) {
throw new CloseRequestException("responded to info probe");
}
sk.interestOps(sk.interestOps()
& (~SelectionKey.OP_WRITE));
} else {
sk.interestOps(sk.interestOps()
| SelectionKey.OP_WRITE);
}
}
}
} catch (CancelledKeyException e) {
LOG.warn("Exception causing close of session 0x"
+ Long.toHexString(sessionId)
+ " due to " + e);
if (LOG.isDebugEnabled()) {
LOG.debug("CancelledKeyException stack trace", e);
}
close();
} catch (CloseRequestException e) {
// expecting close to log session closure
close();
} catch (EndOfStreamException e) {
LOG.warn("caught end of stream exception",e); // tell user why
// expecting close to log session closure
close();
} catch (IOException e) {
LOG.warn("Exception causing close of session 0x"
+ Long.toHexString(sessionId)
+ " due to " + e);
if (LOG.isDebugEnabled()) {
LOG.debug("IOException stack trace", e);
}
close();
}
}NIOServerCnxn寫操作的入口方法有兩個，一個是同步IO的sendBufferSync， 一個是NIO的sendBuffer。
1.基於同步IO的 sendBufferSync方法直接把SocketChannel設置爲阻塞模式，然後直接寫到Socket中去。上面提到的相應4字符命令的場景，就是使用了sendBufferSync的方法，直接寫。
2. sendBuffer方法使用了NIO，它主要是因爲使用了outgoingBuffers隊列來優化寫操作，可以一次寫多個ByteBuffer。寫的時候，先加入到outgoingBuffers，然後設置SelectionKey的寫標誌位，這樣在下次Selector執行select方法時，可以進行寫的動作
void sendBufferSync(ByteBuffer bb) {
try {
/* configure socket to be blocking
* so that we dont have to do write in
* a tight while loop
*/
sock.configureBlocking(true);
if (bb != ServerCnxnFactory.closeConn) {
if (sock.isOpen()) {
sock.write(bb);
}
packetSent();
}
} catch (IOException ie) {
LOG.error("Error sending data synchronously ", ie);
}
}
public void sendBuffer(ByteBuffer bb) {
try {
if (bb != ServerCnxnFactory.closeConn) {
// We check if write interest here because if it is NOT set,
// nothing is queued, so we can try to send the buffer right
// away without waking up the selector
if ((sk.interestOps() & SelectionKey.OP_WRITE) == 0) {
try {
sock.write(bb);
} catch (IOException e) {
// we are just doing best effort right now
}
}
// if there is nothing left to send, we are done
if (bb.remaining() == 0) {
packetSent();
return;
}
}
synchronized(this.factory){
sk.selector().wakeup();
if (LOG.isTraceEnabled()) {
LOG.trace("Add a buffer to outgoingBuffers, sk " + sk
+ " is valid: " + sk.isValid());
}
outgoingBuffers.add(bb);
if (sk.isValid()) {
sk.interestOps(sk.interestOps() | SelectionKey.OP_WRITE);
}
}
} catch(Exception e) {
LOG.error("Unexpected Exception: ", e);
}
}協議（編解碼）
ZooKeeper使用Apache jute來序列化和反序列化Java對象，把Java對象序列化成二進制數據在網絡中傳播。在上一篇從ZooKeeper源代碼看如何實現分佈式系統（二）數據的高可用存儲 中已經介紹了Apache Jute，這裏不再贅述，簡單看一下ZooKeeperServer是如何處理收到的數據包的，可以看到如何把二進制的請求序列化成Java對象來使用。
1. 先用ByteBufferInputStream來將incomingBuffer封裝成流，然後用Jute的接口讀到RequestHeader對象，這個對象實現了Jute的Record接口
2. RequestHeader只有兩個屬性，xid表示事務id，type表示請求的類型
3. 如果是auth類型的請求，從incomingBuffer中讀取數據，反序列化到AuthPacket中，然後調用AuthenticationProvider來進行認證
4.如果是sasl的請求，執行相應的代碼
5. 對於其他的事務請求，構造一個Request對象，進入到submitRequest方法去執行相應的事務請求。
// ZooKeeperServer
public void processPacket(ServerCnxn cnxn, ByteBuffer incomingBuffer) throws IOException {
// We have the request, now process and setup for next
InputStream bais = new ByteBufferInputStream(incomingBuffer);
BinaryInputArchive bia = BinaryInputArchive.getArchive(bais);
RequestHeader h = new RequestHeader();
h.deserialize(bia, "header");
// Through the magic of byte buffers, txn will not be
// pointing
// to the start of the txn
incomingBuffer = incomingBuffer.slice();
if (h.getType() == OpCode.auth) {
LOG.info("got auth packet " + cnxn.getRemoteSocketAddress());
AuthPacket authPacket = new AuthPacket();
ByteBufferInputStream.byteBuffer2Record(incomingBuffer, authPacket);
String scheme = authPacket.getScheme();
AuthenticationProvider ap = ProviderRegistry.getProvider(scheme);
Code authReturn = KeeperException.Code.AUTHFAILED;
if(ap != null) {
try {
authReturn = ap.handleAuthentication(cnxn, authPacket.getAuth());
} catch(RuntimeException e) {
LOG.warn("Caught runtime exception from AuthenticationProvider: " + scheme + " due to " + e);
authReturn = KeeperException.Code.AUTHFAILED;
}
}
if (authReturn!= KeeperException.Code.OK) {
if (ap == null) {
LOG.warn("No authentication provider for scheme: "
+ scheme + " has "
+ ProviderRegistry.listProviders());
} else {
LOG.warn("Authentication failed for scheme: " + scheme);
}
// send a response...
ReplyHeader rh = new ReplyHeader(h.getXid(), 0,
KeeperException.Code.AUTHFAILED.intValue());
cnxn.sendResponse(rh, null, null);
// ... and close connection
cnxn.sendBuffer(ServerCnxnFactory.closeConn);
cnxn.disableRecv();
} else {
if (LOG.isDebugEnabled()) {
LOG.debug("Authentication succeeded for scheme: "
+ scheme);
}
LOG.info("auth success " + cnxn.getRemoteSocketAddress());
ReplyHeader rh = new ReplyHeader(h.getXid(), 0,
KeeperException.Code.OK.intValue());
cnxn.sendResponse(rh, null, null);
}
return;
} else {
if (h.getType() == OpCode.sasl) {
Record rsp = processSasl(incomingBuffer,cnxn);
ReplyHeader rh = new ReplyHeader(h.getXid(), 0, KeeperException.Code.OK.intValue());
cnxn.sendResponse(rh,rsp, "response"); // not sure about 3rd arg..what is it?
}
else {
Request si = new Request(cnxn, cnxn.getSessionId(), h.getXid(),
h.getType(), incomingBuffer, cnxn.getAuthInfo());
si.setOwner(ServerCnxn.me);
submitRequest(si);
}
}
cnxn.incrOutstandingRequests(h);
}可以看到ZooKeeper的請求分爲了兩部分，RequestHeader表示消息頭，剩餘部分表示消息體。消息頭標示了消息的類型。
線程模型
ZooKeeper提供了兩種服務器端的線程模型，一種是基於原生NIO的reactor模型，一種是基於Netty的reactor模型。我們看一下基於NIO的reactor模型。
NIOServerCnxnFactory封裝了Selector對象來做事件分發。NIOServerCnxnFactory本身實現了Runnable接口來作爲一個可運行的線程。它還維護了一個線程，來使它本身作爲一個單獨的線程運行。
1. configure方法創建了一個守護線程，並且創建了ServerSocketChannel，註冊到了Selector上去監聽ACCEPT事件
2. 維護了一個HashMap，由客戶端IP映射到來自該IP的NIOServerCnxn連接對象。
3. start方法啓動線程，開始監聽端口來響應客戶端請求
4. run方法就是reactor模型的EventLoop，Selector每隔1秒執行一次select方法來處理IO請求，並分發到對應的SocketChannel中去。可以看到在分發請求的時候並沒有創建新的線程
所以NIOServerCnxnFactory是一個最簡單的單線程的reactor模型，由一個線程來進行IO事件的分發，以及IO的讀寫
public class NIOServerCnxnFactory extends ServerCnxnFactory implements Runnable {
ServerSocketChannel ss;
final Selector selector = Selector.open();
Thread thread;
public void configure(InetSocketAddress addr, int maxcc) throws IOException {
configureSaslLogin();
thread = new Thread(this, "NIOServerCxn.Factory:" + addr);
thread.setDaemon(true);
maxClientCnxns = maxcc;
this.ss = ServerSocketChannel.open();
ss.socket().setReuseAddress(true);
LOG.info("binding to port " + addr);
ss.socket().bind(addr);
ss.configureBlocking(false);
ss.register(selector, SelectionKey.OP_ACCEPT);
}
final HashMap<InetAddress, Set<NIOServerCnxn>> ipMap =
new HashMap<InetAddress, Set<NIOServerCnxn>>( );
public void start() {
// ensure thread is started once and only once
if (thread.getState() == Thread.State.NEW) {
thread.start();
}
}
private void addCnxn(NIOServerCnxn cnxn) {
synchronized (cnxns) {
cnxns.add(cnxn);
synchronized (ipMap){
InetAddress addr = cnxn.sock.socket().getInetAddress();
Set<NIOServerCnxn> s = ipMap.get(addr);
if (s == null) {
// in general we will see 1 connection from each
// host, setting the initial cap to 2 allows us
// to minimize mem usage in the common case
// of 1 entry -- we need to set the initial cap
// to 2 to avoid rehash when the first entry is added
s = new HashSet<NIOServerCnxn>(2);
s.add(cnxn);
ipMap.put(addr,s);
} else {
s.add(cnxn);
}
}
}
}
public void run() {
while (!ss.socket().isClosed()) {
try {
selector.select(1000);
Set<SelectionKey> selected;
synchronized (this) {
selected = selector.selectedKeys();
}
ArrayList<SelectionKey> selectedList = new ArrayList<SelectionKey>(
selected);
Collections.shuffle(selectedList);
for (SelectionKey k : selectedList) {
if ((k.readyOps() & SelectionKey.OP_ACCEPT) != 0) {
SocketChannel sc = ((ServerSocketChannel) k
.channel()).accept();
InetAddress ia = sc.socket().getInetAddress();
int cnxncount = getClientCnxnCount(ia);
if (maxClientCnxns > 0 && cnxncount >= maxClientCnxns){
LOG.warn("Too many connections from " + ia
+ " - max is " + maxClientCnxns );
sc.close();
} else {
LOG.info("Accepted socket connection from "
+ sc.socket().getRemoteSocketAddress());
sc.configureBlocking(false);
SelectionKey sk = sc.register(selector,
SelectionKey.OP_READ);
NIOServerCnxn cnxn = createConnection(sc, sk);
sk.attach(cnxn);
addCnxn(cnxn);
}
} else if ((k.readyOps() & (SelectionKey.OP_READ | SelectionKey.OP_WRITE)) != 0) {
NIOServerCnxn c = (NIOServerCnxn) k.attachment();
c.doIO(k);
} else {
if (LOG.isDebugEnabled()) {
LOG.debug("Unexpected ops in select "
+ k.readyOps());
}
}
}
selected.clear();
} catch (RuntimeException e) {
LOG.warn("Ignoring unexpected runtime exception", e);
} catch (Exception e) {
LOG.warn("Ignoring exception", e);
}
}
closeAll();
LOG.info("NIOServerCnxn factory exited run method");
}
