事件機制：
Watcher 監聽機制是 Zookeeper 中非常重要的特性，我們基於 zookeeper 上創建的節點，可以對這些節點綁定監聽事件，比如可以監聽節點數據變更、節點刪除、子節點狀態變更等事件，通過這個事件機制，可以基於 zookeeper實現分佈式鎖、集羣管理等功能。
watcher 特性：當數據發生變化的時候， zookeeper 會產生一個 watcher 事件，並且會發送到客戶端。但是客戶端只會收到一次通知。如果後續這個節點再次發生變化，那麼之前設置 watcher 的客戶端不會再次收到消息。（watcher 是一次性的操作）。 可以通過循環監聽去達到永久監聽效果。
如何註冊事件機制：
ZooKeeper 的 Watcher 機制，總的來說可以分爲三個過程：客戶端註冊 Watcher、服務器處理 Watcher 和客戶端回調 Watcher客戶端。註冊 watcher 有 3 種方式，getData、exists、getChildren；以如下代碼爲例
如何觸發事件？ 凡是事務類型的操作，都會觸發監聽事件。create /delete /setData,來看以下代碼簡單實現
public class WatcherDemo {
public static void main(String[] args) throws IOException, InterruptedException, KeeperException {
final CountDownLatch countDownLatch=new CountDownLatch(1);
final ZooKeeper zooKeeper=
new ZooKeeper("192.168.254.135:2181," +
"192.168.254.136:2181,192.168.254.137:2181",
4000, new Watcher() {
@Override
public void process(WatchedEvent event) {
System.out.println("默認事件： "+event.getType());
if(Event.KeeperState.SyncConnected==event.getState()){
//如果收到了服務端的響應事件，連接成功
countDownLatch.countDown();
}
}
});
countDownLatch.await();
zooKeeper.create("/zk-wuzz","1".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE,CreateMode.PERSISTENT);
//exists getdata getchildren
//通過exists綁定事件
Stat stat=zooKeeper.exists("/zk-wuzz", new Watcher() {
@Override
public void process(WatchedEvent event) {
System.out.println(event.getType()+"->"+event.getPath());
try {
//再一次去綁定事件 ,但是這個走的是默認事件
zooKeeper.exists(event.getPath(),true);
} catch (KeeperException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
//通過修改的事務類型操作來觸發監聽事件
stat=zooKeeper.setData("/zk-wuzz","2".getBytes(),stat.getVersion());
Thread.sleep(1000);
zooKeeper.delete("/zk-wuzz",stat.getVersion());
System.in.read();
}
}
以上就是 Watcher 的簡單實現操作。接下來淺析一下這個 Watcher 實現的流程。
watcher 事件類型：
//org.apache.zookeeper.Watcher.Event.EventType
public enum EventType {
None (-1), // 客戶端連接狀態發生變化的時候 會受到none事件 NodeCreated (1), // 節點創建事件 NodeDeleted (2), // 節點刪除事件 NodeDataChanged (3), // 節點數據變化 NodeChildrenChanged (4); // 子節點被創建 刪除觸發該事件 }
事件的實現原理：
client 端連接後會註冊一個事件，然後客戶端會保存這個事件，通過zkWatcherManager 保存客戶端的事件註冊，通知服務端 Watcher 爲 true，然後服務端會通過WahcerManager 會綁定path對應的事件。如下圖：
請求發送：
接下去通過源碼層面去熟悉一下這個 Watcher 的流程。由於我們demo 是通過exists 來註冊事件，那麼我們就通過 exists 來作爲入口。先來看看ZooKeeper API 的初始化過程：
public ZooKeeper(String connectString, int sessionTimeout, Watcher watcher,
boolean canBeReadOnly)
throws IOException
{
LOG.info("Initiating client connection, connectString=" + connectString
+ " sessionTimeout=" + sessionTimeout + " watcher=" + watcher);
//--在這裏將 watcher 設置到ZKWatchManager
watchManager.defaultWatcher = watcher;
ConnectStringParser connectStringParser = new ConnectStringParser(
connectString);
HostProvider hostProvider = new StaticHostProvider(connectStringParser.getServerAddresses());
//初始化了 ClientCnxn，並且調用 cnxn.start()方法
cnxn = new ClientCnxn(connectStringParser.getChrootPath(),hostProvider, sessionTimeout, this, watchManager,getClientCnxnSocket(), canBeReadOnly);
cnxn.start();
}
在創建一個 ZooKeeper 客戶端對象實例時，我們通過 new Watcher()向構造方法中傳入一個默認的 Watcher, 這個 Watcher 將作爲整個 ZooKeeper 會話期間的默認Watcher，會一直被保存在客戶端 ZKWatchManager 的 defaultWatcher 中.其中初始化了 ClientCnxn並且調用了其start 方法：
public ClientCnxn(String chrootPath, HostProvider hostProvider, int sessionTimeout, ZooKeeper zooKeeper,
ClientWatchManager watcher, ClientCnxnSocket clientCnxnSocket,
long sessionId, byte[] sessionPasswd, boolean canBeReadOnly) {
this.zooKeeper = zooKeeper;
this.watcher = watcher;
this.sessionId = sessionId;
this.sessionPasswd = sessionPasswd;
this.sessionTimeout = sessionTimeout;//會話超時
this.hostProvider = hostProvider;
this.chrootPath = chrootPath;
// 連接超時
connectTimeout = sessionTimeout / hostProvider.size();
readTimeout = sessionTimeout * 2 / 3; //超時
readOnly = canBeReadOnly;
// 新建了一個發送線程
sendThread = new SendThread(clientCnxnSocket);
// 處理watcher回調event的線程
eventThread = new EventThread();
}
//啓動兩個線程
public void start() {
sendThread.start();
eventThread.start();
}
ClientCnxn:是 Zookeeper 客戶端和 Zookeeper 服務器端進行通信和事件通知處理的主要類，它內部包含兩個類，
- SendThread ：負責客戶端和服務器端的數據通信, 也包括事件信息的傳輸
- EventThread : 主要在客戶端回調註冊的 Watchers 進行通知處理
接下去就是我們通過getData、exists、getChildren 註冊事件的過程了，以exists爲例：
public Stat exists(final String path, Watcher watcher)
throws KeeperException, InterruptedException
{
final String clientPath = path;
PathUtils.validatePath(clientPath);
// 這個很關鍵，執行回調的時候會用到
WatchRegistration wcb = null;
if (watcher != null) {//不爲空，將進行包裝
wcb = new ExistsWatchRegistration(watcher, clientPath);
}
final String serverPath = prependChroot(clientPath);
//類似手寫RPC中的一個請求類request
//在這裏 requesr就封裝了兩個東西 1.ZooDefs.OpCode.exists
//還有一個是watch ->true
RequestHeader h = new RequestHeader();
h.setType(ZooDefs.OpCode.exists);
ExistsRequest request = new ExistsRequest();
request.setPath(serverPath);
request.setWatch(watcher != null);
SetDataResponse response = new SetDataResponse();
//通過客戶端的網絡處理類去提交請求
ReplyHeader r = cnxn.submitRequest(h, request, response, wcb);
if (r.getErr() != 0) {
if (r.getErr() == KeeperException.Code.NONODE.intValue()) {
return null;
}
throw KeeperException.create(KeeperException.Code.get(r.getErr()),
clientPath);
}
return response.getStat().getCzxid() == -1 ? null : response.getStat();
}
其實這個方法內就做了兩件事，初始化了ExistsWatchRegistration 以及封裝了一個網絡請求參數 ExistsRequest，接着通過 cnxn.submitRequest 發送請求：
public ReplyHeader submitRequest(RequestHeader h, Record request,
Record response, WatchRegistration watchRegistration)
throws InterruptedException {
ReplyHeader r = new ReplyHeader();//應答消息頭
//組裝請求入隊
Packet packet = queuePacket(h, r, request, response, null, null, null,
null, watchRegistration);
//等待請求完成。否則阻塞
synchronized (packet) {
while (!packet.finished) {
packet.wait();
}
}
return r;
}
這裏驗證了我們之前流程圖中對於請求進行封包都過程，緊接着會調用wait進入阻塞，一直的等待整個請求處理完畢，我們跟進 queuePacket：
Packet queuePacket(RequestHeader h, ReplyHeader r, Record request,
Record response, AsyncCallback cb, String clientPath,
String serverPath, Object ctx, WatchRegistration watchRegistration)
{
Packet packet = null;
// 這個隊列就是存放我們請求的隊列，注意，我們還沒有爲包生成Xid。它是在發送時生成，通過實現ClientCnxnSocket::doIO()，數據包實際發送的地方。
synchronized (outgoingQueue) {
packet = new Packet(h, r, request, response, watchRegistration);
packet.cb = cb;
packet.ctx = ctx;
packet.clientPath = clientPath;
packet.serverPath = serverPath;
if (!state.isAlive() || closing) {
conLossPacket(packet);
} else {
// If the client is asking to close the session then
// mark as closing
if (h.getType() == OpCode.closeSession) {
closing = true;
}//請求包入隊
outgoingQueue.add(packet);
}
}
//喚醒selector
sendThread.getClientCnxnSocket().wakeupCnxn();
return packet;
}
這裏加了個同步鎖以避免併發問題，封裝了一個 Packet 並將其加入到一個阻塞隊列 outgoingQueue 中，最後調用 sendThread.getClientCnxnSocket().wakeupCnxn() 喚醒selector。看到這裏，發現只是發送了數據，那哪裏觸發了對 outgoingQueue 隊列的消息進行消費。再把組裝的packeet 放入隊列的時候用到的 cnxn.submitRequest(h, request, response, wcb);這個cnxn 是哪裏來的呢？ 在 zookeeper的構造函數中，我們初始化了一個ClientCnxn並且啓動了兩個線程：
public void start() {
sendThread.start();
eventThread.start();
}
對於當前場景來說，目前是需要將封裝好的數據包發送出去，很顯然走的是 SendThread，我們進入他的 Run 方法：
public void run() {
clientCnxnSocket.introduce(this,sessionId);
clientCnxnSocket.updateNow();
//心跳相關
clientCnxnSocket.updateLastSendAndHeard();
int to;
long lastPingRwServer = Time.currentElapsedTime();
final int MAX_SEND_PING_INTERVAL = 10000; //10 seconds
InetSocketAddress serverAddress = null;
while (state.isAlive()) {
//......七七八八一頓判斷
//發起網絡請求
clientCnxnSocket.doTransport(to, pendingQueue, outgoingQueue, ClientCnxn.this);
}
cleanup();
clientCnxnSocket.close();
if (state.isAlive()) {
eventThread.queueEvent(new WatchedEvent(Event.EventType.None,
Event.KeeperState.Disconnected, null));
}
}
這一步大部分的邏輯是進行校驗判斷連接狀態，以及相關心跳維持得操作，最後會走 clientCnxnSocket.doTransport ：
void doTransport(int waitTimeOut, List<Packet> pendingQueue, LinkedList<Packet> outgoingQueue,
ClientCnxn cnxn)
throws IOException, InterruptedException {
selector.select(waitTimeOut);
Set<SelectionKey> selected;
synchronized (this) {// 獲取 selectKeys
selected = selector.selectedKeys();
}
updateNow();//理解爲時間常量
for (SelectionKey k : selected) {//獲取channel
SocketChannel sc = ((SocketChannel) k.channel());
// readyOps :獲取此鍵上ready操作集合.即在當前通道上已經就緒的事件
// SelectKey.OP_CONNECT 連接就緒事件，表示客戶與服務器的連接已經建立成功
// 兩者的與計算不等於0
if ((k.readyOps() & SelectionKey.OP_CONNECT) != 0) {
if (sc.finishConnect()) {
updateLastSendAndHeard();
sendThread.primeConnection();
}
// 讀或者寫通道準備完畢
} else if ((k.readyOps() & (SelectionKey.OP_READ | SelectionKey.OP_WRITE)) != 0) {
//進行IO傳輸
doIO(pendingQueue, outgoingQueue, cnxn);
}
}
if (sendThread.getZkState().isConnected()) {
synchronized(outgoingQueue) {
if (findSendablePacket(outgoingQueue,
cnxn.sendThread.clientTunneledAuthenticationInProgress()) != null) {
enableWrite();
}
}
}
selected.clear();
}
這裏的代碼相信很多小夥伴都不會很陌生，是 Java NIO相關操作的API，對於當前場景，這裏我們是走 SelectionKey.OP_WRITE ，即 doIO(pendingQueue, outgoingQueue, cnxn) ：
void doIO(List<Packet> pendingQueue, LinkedList<Packet> outgoingQueue, ClientCnxn cnxn)
throws InterruptedException, IOException {
SocketChannel sock = (SocketChannel) sockKey.channel();
if (sock == null) {
throw new IOException("Socket is null!");
}
// 可讀狀態
// ....省略部分代碼，對於目前來說是要將exsits指令發送出去，寫出去
// 可寫狀態
if (sockKey.isWritable()) {
synchronized(outgoingQueue) {//加鎖
// 發現傳輸包
Packet p = findSendablePacket(outgoingQueue,
cnxn.sendThread.clientTunneledAuthenticationInProgress());
if (p != null) {
updateLastSend();//心跳相關操作
// If we already started writing p, p.bb will already exist
if (p.bb == null) {
if ((p.requestHeader != null) &&
(p.requestHeader.getType() != OpCode.ping) &&
(p.requestHeader.getType() != OpCode.auth)) {
p.requestHeader.setXid(cnxn.getXid());
}
p.createBB();
}//將數據寫入channel
sock.write(p.bb);
// .......省略部分代碼
}
}
}
public void createBB() {
try {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
BinaryOutputArchive boa = BinaryOutputArchive.getArchive(baos);
boa.writeInt(-1, "len"); // We'll fill this in later
if (requestHeader != null) {
requestHeader.serialize(boa, "header");
}
if (request instanceof ConnectRequest) {
request.serialize(boa, "connect");
// append "am-I-allowed-to-be-readonly" flag
boa.writeBool(readOnly, "readOnly");
} else if (request != null) {
request.serialize(boa, "request");
}
baos.close();
this.bb = ByteBuffer.wrap(baos.toByteArray());
this.bb.putInt(this.bb.capacity() - 4);
this.bb.rewind();
} catch (IOException e) {
LOG.warn("Ignoring unexpected exception", e);
}
}
序列化框架：jute.至此就將當前都操作發送至服務器端，當服務器端接收到請求進行下一步的處理.
服務端接收請求處理流程：
服務端有一個 NIOServerCnxn 類，在服務器端初始化的時候，在QuorumPeerMain.runFromConfig方法中：
ServerCnxnFactory cnxnFactory = ServerCnxnFactory.createFactory();
這裏創建的 cnxnFactory 就是服務器端的網絡請求處理類工廠對象，即 NIOServerCnxnFactory ，最後會調用 quorumPeer.start();啓動，這裏啓動的就是 NIOServerCnxnFactory 裏面都Run方法，我們跟進去看看：
public void run() {
// socket 不是關閉狀態
while (!ss.socket().isClosed()) {
try {//設置超時時間
selector.select(1000);
Set<SelectionKey> selected;
synchronized (this) {//跟剛剛一樣，獲取事件鍵列表
selected = selector.selectedKeys();
}
ArrayList<SelectionKey> selectedList = new ArrayList<SelectionKey>(
selected);
Collections.shuffle(selectedList);
for (SelectionKey k : selectedList) {//遍歷事件keys
if ((k.readyOps() & SelectionKey.OP_ACCEPT) != 0) {//就緒，等待連接
SocketChannel sc = ((ServerSocketChannel) k
.channel()).accept();
InetAddress ia = sc.socket().getInetAddress();
int cnxncount = getClientCnxnCount(ia);
if (maxClientCnxns > 0 && cnxncount >= maxClientCnxns){
LOG.warn("Too many connections from " + ia
+ " - max is " + maxClientCnxns );
sc.close();
} else {
LOG.info("Accepted socket connection from "
+ sc.socket().getRemoteSocketAddress());
sc.configureBlocking(false);
SelectionKey sk = sc.register(selector,
SelectionKey.OP_READ);
NIOServerCnxn cnxn = createConnection(sc, sk);
sk.attach(cnxn);
addCnxn(cnxn);
}
// 就緒讀寫事件
} else if ((k.readyOps() & (SelectionKey.OP_READ | SelectionKey.OP_WRITE)) != 0) {
NIOServerCnxn c = (NIOServerCnxn) k.attachment();
c.doIO(k);
} else {
......//省略部分代碼
}
看到這裏大家應該都清楚了，這裏就是一個 selector 的循環監聽，由於客戶端發送過來，服務端負責處理，即對於服務器端是達到一個讀事件，所以這裏會走 c.doIO(k); 我們跟進去看看具體做了什麼：
void doIO(SelectionKey k) throws InterruptedException {
try {
if (isSocketOpen() == false) {
LOG.warn("trying to do i/o on a null socket for session:0x"
+ Long.toHexString(sessionId));
return;
}// 可讀
if (k.isReadable()) {
int rc = sock.read(incomingBuffer);
if (rc < 0) {
throw new EndOfStreamException(
"Unable to read additional data from client sessionid 0x"
+ Long.toHexString(sessionId)
+ ", likely client has closed socket");
}// 返回剩餘的可用長度，此長度爲實際讀取的數據長度 如果是0，代表讀完了
if (incomingBuffer.remaining() == 0) {
boolean isPayload;
if (incomingBuffer == lenBuffer) { // start of next request
incomingBuffer.flip();
isPayload = readLength(k);
incomingBuffer.clear();
} else {
// continuation
isPayload = true;
}
if (isPayload) { // not the case for 4letterword
readPayload();
}
else {
// four letter words take care
// need not do anything else
return;
}
}
}
......//省略部分代碼
}
這裏進入依舊會判斷是什麼事件，我們這裏重點看 isReadable，這裏會從channel中讀取請求數據，繼而進入 readPayload();
private void readPayload() throws IOException, InterruptedException {
if (incomingBuffer.remaining() != 0) { // have we read length bytes?
int rc = sock.read(incomingBuffer); // sock is non-blocking, so ok
if (rc < 0) {
throw new EndOfStreamException(
"Unable to read additional data from client sessionid 0x"
+ Long.toHexString(sessionId)
+ ", likely client has closed socket");
}
}
//判斷是否有可讀數據
if (incomingBuffer.remaining() == 0) { // have we read length bytes?
packetReceived();
incomingBuffer.flip();
if (!initialized) {
readConnectRequest();
} else {
readRequest();
}
lenBuffer.clear();
incomingBuffer = lenBuffer;
}
}
這裏會判斷buffer中是否有可讀數據，繼而調用 readRequest() 去處理請求：
private void readRequest() throws IOException {
zkServer.processPacket(this, incomingBuffer);
}
繼而調用 zkServer.processPacket：
public void processPacket(ServerCnxn cnxn, ByteBuffer incomingBuffer) throws IOException {
// We have the request, now process and setup for next
//我們有了請求，現在處理並設置next
InputStream bais = new ByteBufferInputStream(incomingBuffer);
BinaryInputArchive bia = BinaryInputArchive.getArchive(bais);
RequestHeader h = new RequestHeader();
h.deserialize(bia, "header");
incomingBuffer = incomingBuffer.slice();
if (h.getType() == OpCode.auth) {
......
} else {
if (h.getType() == OpCode.sasl) {
......
}
else {// 由於exists方法一開始設置了 h.setType(ZooDefs.OpCode.exists);所以走這個流程
Request si = new Request(cnxn, cnxn.getSessionId(), h.getXid(),
h.getType(), incomingBuffer, cnxn.getAuthInfo());
si.setOwner(ServerCnxn.me);
submitRequest(si);
}
}
cnxn.incrOutstandingRequests(h);
}
根據當前調用鏈會走else裏得else的流程，所以會調到 submitRequest(si) ：
public void submitRequest(Request si) {
if (firstProcessor == null) {
synchronized (this) {
try {
// Since all requests are passed to the request
// processor it should wait for setting up the request
// processor chain. The state will be updated to RUNNING
// after the setup.
while (state == State.INITIAL) {
wait(1000);
}
} catch (InterruptedException e) {
LOG.warn("Unexpected interruption", e);
}
if (firstProcessor == null || state != State.RUNNING) {
throw new RuntimeException("Not started");
}
}
}
try {
touch(si.cnxn);
boolean validpacket = Request.isValid(si.type);
if (validpacket) { // 鏈式處理
firstProcessor.processRequest(si);
if (si.cnxn != null) {
incInProcess();
}
// ...省略部分代碼
}
這裏到了服務端的處理鏈都流程了，首先我們需要知道這個處理鏈是哪裏初始化的呢？我們需要知道在整個調用鏈過程中採用的是責任鏈都設計模式，其中在ZK中每種角色以及部署方式都有其獨特的調用鏈，我們先來看一下他是在哪裏初始化的，在本類(ZookeeperServer)中搜索到如下方法：
protected void setupRequestProcessors() {
RequestProcessor finalProcessor = new FinalRequestProcessor(this);
RequestProcessor syncProcessor = new SyncRequestProcessor(this,
finalProcessor);
((SyncRequestProcessor)syncProcessor).start();
firstProcessor = new PrepRequestProcessor(this, syncProcessor);
((PrepRequestProcessor)firstProcessor).start();
}
public synchronized void startup() {
if (sessionTracker == null) {
createSessionTracker();
}
startSessionTracker();
setupRequestProcessors();
registerJMX();
setState(State.RUNNING);
notifyAll();
}
從代碼中可以看出在 setupRequestProcessors初始化了該鏈路，其中由 startup() 進入初始化，而這個startup在我們跟leader選舉的時候，服務端初始化中在 QuorumPeer 類中的Run方法中有調到，可以跟單機版的流程看一下，針對不同的角色，這裏有4種不同的實現
我們來看看每種不同角色的調用鏈：standalone，單機部署：
protected void setupRequestProcessors() {
// PrepRequestProcessor -> SyncRequestProcessor-> FinalRequestProcessor
RequestProcessor finalProcessor = new FinalRequestProcessor(this);
RequestProcessor syncProcessor = new SyncRequestProcessor(this,
finalProcessor);
((SyncRequestProcessor)syncProcessor).start();
firstProcessor = new PrepRequestProcessor(this, syncProcessor);
((PrepRequestProcessor)firstProcessor).start();
}
集羣部署 Leader ：
protected void setupRequestProcessors() {
// PrepRequestProcessor->ProposalRequestProcessor -> CommitProcessor
// -> ToBeAppliedRequestProcessor ->FinalRequestProcessor
RequestProcessor finalProcessor = new FinalRequestProcessor(this);
RequestProcessor toBeAppliedProcessor = new Leader.ToBeAppliedRequestProcessor(
finalProcessor, getLeader().toBeApplied);
//提交相關
commitProcessor = new CommitProcessor(toBeAppliedProcessor,
Long.toString(getServerId()), false,
getZooKeeperServerListener());
commitProcessor.start();
////事務相關
ProposalRequestProcessor proposalProcessor = new ProposalRequestProcessor(this,
commitProcessor);
proposalProcessor.initialize();
firstProcessor = new PrepRequestProcessor(this, proposalProcessor);
((PrepRequestProcessor)firstProcessor).start();
}
集羣部署 Follower:
protected void setupRequestProcessors() {
// FollowerRequestProcessor->CommitProcessor ->FinalRequestProcessor
RequestProcessor finalProcessor = new FinalRequestProcessor(this);
commitProcessor = new CommitProcessor(finalProcessor,
Long.toString(getServerId()), true,
getZooKeeperServerListener());
commitProcessor.start();
firstProcessor = new FollowerRequestProcessor(this, commitProcessor);
((FollowerRequestProcessor) firstProcessor).start();
//同步應答相關
syncProcessor = new SyncRequestProcessor(this,
new SendAckRequestProcessor((Learner)getFollower()));
syncProcessor.start();
}
集羣部署 Observer：
protected void setupRequestProcessors() {
RequestProcessor finalProcessor = new FinalRequestProcessor(this);
commitProcessor = new CommitProcessor(finalProcessor,
Long.toString(getServerId()), true,
getZooKeeperServerListener());
commitProcessor.start();
firstProcessor = new ObserverRequestProcessor(this, commitProcessor);
((ObserverRequestProcessor) firstProcessor).start();
if (syncRequestProcessorEnabled) {
syncProcessor = new SyncRequestProcessor(this, null);
syncProcessor.start();
}
}
這裏 setupRequestProcessors 方法，對於不同的集羣角色都有相對應都類去重寫該方法，我們這裏以單機部署的流程去處理對應流程：回到剛剛 submitRequest 方法中：
public void submitRequest(Request si) {
//firstProcessor不可能是null
try {
touch(si.cnxn);
boolean validpacket = Request.isValid(si.type);
if (validpacket) {
firstProcessor.processRequest(si);
if (si.cnxn != null) {
incInProcess();
}
//.......
}
我們根據單機版的調用鏈的順序：PrepRequestProcessor -> SyncRequestProcessor-> FinalRequestProcessor。而這3個處理器的主要功能如下：
- PrepRequestProcessor：此請求處理器通常位於RequestProcessor的開頭，等等可以看到，就exsits對應就一個Session的檢查
- SyncRequestProcessor：此RequestProcessor將請求記錄到磁盤。簡單來說就是持久化的處理器
- FinalRequestProcessor：此請求處理程序實際應用與請求關聯的任何事務，併爲任何查詢提供服務
首先進入PrepRequestProcessor.processRequest：
public void processRequest(Request request) {
// request.addRQRec(">prep="+zks.outstandingChanges.size());
submittedRequests.add(request);
}
很奇怪，processRequest 只是把 request 添加到submittedRequests中，根據前面的經驗，很自然的想到這裏又是一個異步操作。而submittedRequests又是一個阻塞隊列LinkedBlockingQueue submittedRequests = new LinkedBlockingQueue();而 PrepRequestProcessor 這個類又繼承了線程類，因此我們直接找到當前類中的方法如下:
public void run() {
try {
while (true) {
Request request = submittedRequests.take();
long traceMask = ZooTrace.CLIENT_REQUEST_TRACE_MASK;
if (request.type == OpCode.ping) {
traceMask = ZooTrace.CLIENT_PING_TRACE_MASK;
}
if (LOG.isTraceEnabled()) {
ZooTrace.logRequest(LOG, traceMask, 'P', request, "");
}
if (Request.requestOfDeath == request) {
break;
}// 調用 pRequest 進行預處理
pRequest(request);
}
} catch (RequestProcessorException e) {
if (e.getCause() instanceof XidRolloverException) {
LOG.info(e.getCause().getMessage());
}
handleException(this.getName(), e);
} catch (Exception e) {
handleException(this.getName(), e);
}
LOG.info("PrepRequestProcessor exited loop!");
}
protected void pRequest(Request request) throws RequestProcessorException {
// LOG.info("Prep>>> cxid = " + request.cxid + " type = " +
// request.type + " id = 0x" + Long.toHexString(request.sessionId));
request.hdr = null;
request.txn = null;
try {
switch (request.type) {
......//省略部分代碼
case OpCode.sync:
case OpCode.exists: //根據我們這個案例會走這個分支
case OpCode.getData:
case OpCode.getACL:
case OpCode.getChildren:
case OpCode.getChildren2:
case OpCode.ping:
case OpCode.setWatches:
zks.sessionTracker.checkSession(request.sessionId,
request.getOwner());
break;
.....//省略部分代碼
request.zxid = zks.getZxid();
nextProcessor.processRequest(request);
}
這裏通過判斷請求的類型進而調用處理，而在本場景中 case OpCode.exists: 會走檢查 Session 而沒有做其他操作，進而進入下一個調用鏈 SyncRequestProcessor.processRequest：
public void processRequest(Request request) {
// request.addRQRec(">sync");
queuedRequests.add(request);
}
又是一樣的套路，進入其 Run方法：
public void run() {
try {
int logCount = 0;
// we do this in an attempt to ensure that not all of the servers
// in the ensemble take a snapshot at the same time
setRandRoll(r.nextInt(snapCount/2));
while (true) {
Request si = null;
if (toFlush.isEmpty()) {
//出隊
si = queuedRequests.take();
} else {
si = queuedRequests.poll();
if (si == null) {
flush(toFlush);
continue;
}
}
if (si == requestOfDeath) {
break;
}
if (si != null) {
//下面這塊代碼，粗略看來是觸發快照操作，啓動一個處理快照的線程
// track the number of records written to the log
if (zks.getZKDatabase().append(si)) {
logCount++;
if (logCount > (snapCount / 2 + randRoll)) {
setRandRoll(r.nextInt(snapCount/2));
// roll the log
zks.getZKDatabase().rollLog();
// take a snapshot
if (snapInProcess != null && snapInProcess.isAlive()) {
LOG.warn("Too busy to snap, skipping");
} else {
snapInProcess = new ZooKeeperThread("Snapshot Thread") {
public void run() {
try {
zks.takeSnapshot();
} catch(Exception e) {
LOG.warn("Unexpected exception", e);
}
}
};
snapInProcess.start();
}
logCount = 0;
}
} else if (toFlush.isEmpty()) {
// optimization for read heavy workloads
// iff this is a read, and there are no pending
// flushes (writes), then just pass this to the next
// processor
if (nextProcessor != null) {//調用下一個處理器
nextProcessor.processRequest(si);
if (nextProcessor instanceof Flushable) {
((Flushable)nextProcessor).flush();
}
}
continue;
}
toFlush.add(si);
if (toFlush.size() > 1000) {
flush(toFlush);
}
}
}
......
}
接着進入下一個調用鏈 FinalRequestProcessor.processRequest：
public void processRequest(Request request) {
//省略部分代碼 校驗 相關
switch (request.type) {
//省略部分代碼
case OpCode.exists: {
lastOp = "EXIS";
// TODO we need to figure out the security requirement for this!
ExistsRequest existsRequest = new ExistsRequest();
// 反序列化 將 ByteBuffer 反序列化成爲 ExitsRequest. 這個就是我們在客戶端發起請求的時候傳遞過來的 Request 對象
ByteBufferInputStream.byteBuffer2Record(request.request,existsRequest);
// 得到請求的路徑
String path = existsRequest.getPath();
if (path.indexOf('\0') != -1) {
throw new KeeperException.BadArgumentsException();
}
// 終於找到一個很關鍵的代碼，判斷請求的 getWatch 是否存在，如果存在，則傳遞 cnxn （ servercnxn ）
// 對於 exists 請求，需要監聽 data 變化事件，添加 watcher
Stat stat = zks.getZKDatabase().statNode(path, existsRequest.getWatch() ? cnxn : null);
// 在服務端內存數據庫中根據路徑得到結果進行組裝，設置爲 ExistsResponse
rsp = new ExistsResponse(stat);
break;
}
.....//省略部分代碼
}
這裏的 cnxn 是 SverCnxn cnxn = request.cnxn在 processRequest(Request request) 方法內，推至前面 c.doIO(k) 的這個c 是通過 NIOServerCnxn c = (NIOServerCnxn) k.attachment() 獲取到的。
最後將這個信息保存在服務器端：
public Stat statNode(String path, Watcher watcher)
throws KeeperException.NoNodeException {
Stat stat = new Stat();
DataNode n = nodes.get(path);
if (watcher != null) {
//保存watch事件
dataWatches.addWatch(path, watcher);
}
if (n == null) {
throw new KeeperException.NoNodeException();
}
synchronized (n) {
n.copyStat(stat);
return stat;
}
}
//保存 watch 事件
public synchronized void addWatch(String path, Watcher watcher) {
HashSet<Watcher> list = watchTable.get(path);
if (list == null) {
// don't waste memory if there are few watches on a node
// rehash when the 4th entry is added, doubling size thereafter
// seems like a good compromise
list = new HashSet<Watcher>(4);
watchTable.put(path, list);
}
list.add(watcher);
HashSet<String> paths = watch2Paths.get(watcher);
if (paths == null) {
// cnxns typically have many watches, so use default cap here
paths = new HashSet<String>();
watch2Paths.put(watcher, paths);
}
paths.add(path);
}
至此，服務端處理完成。
客戶端接收服務端處理完成的響應：
服務端處理完成以後，由於在 發送exsits的時候調用了doTransport ，本身調用這個方法之前的ClientCnxn 的 run方法是一直在輪詢跑着的。所以在不斷的輪詢Selector ，所以這裏不管是客戶端的讀還是寫操作，都會進入ClientCnxnSocketNIO.doIO ，這裏是接收服務端的返回：
void doIO(List<Packet> pendingQueue, LinkedList<Packet> outgoingQueue, ClientCnxn cnxn)
throws InterruptedException, IOException {
SocketChannel sock = (SocketChannel) sockKey.channel();
if (sock == null) {
throw new IOException("Socket is null!");
}
if (sockKey.isReadable()) {
int rc = sock.read(incomingBuffer);
if (rc < 0) {
throw new EndOfStreamException(
"Unable to read additional data from server sessionid 0x"
+ Long.toHexString(sessionId)
+ ", likely server has closed socket");
}//判斷是否有刻度數據
if (!incomingBuffer.hasRemaining()) {
incomingBuffer.flip();
if (incomingBuffer == lenBuffer) {
recvCount++;
readLength();
} else if (!initialized) {
readConnectResult();
enableRead();
if (findSendablePacket(outgoingQueue,
cnxn.sendThread.clientTunneledAuthenticationInProgress()) != null) {
// Since SASL authentication has completed (if client is configured to do so),
// outgoing packets waiting in the outgoingQueue can now be sent.
enableWrite();
}
lenBuffer.clear();
incomingBuffer = lenBuffer;
updateLastHeard();
initialized = true;
} else {//讀取響應
sendThread.readResponse(incomingBuffer);
lenBuffer.clear();
incomingBuffer = lenBuffer;
updateLastHeard();
}
}
}
......//省略部分代碼
}
}
根據當前場景我們現在是接收服務器響應應該走的是 read，最後會調用 sendThread.readResponse(incomingBuffer);來讀取數據：
void readResponse(ByteBuffer incomingBuffer) throws IOException {
ByteBufferInputStream bbis = new ByteBufferInputStream(
incomingBuffer);
BinaryInputArchive bbia = BinaryInputArchive.getArchive(bbis);
ReplyHeader replyHdr = new ReplyHeader();
// 反序列化 header
replyHdr.deserialize(bbia, "header");
if (replyHdr.getXid() == -2) {
// -2 is the xid for pings
if (LOG.isDebugEnabled()) {
LOG.debug("Got ping response for sessionid: 0x"
+ Long.toHexString(sessionId)
+ " after "
+ ((System.nanoTime() - lastPingSentNs) / 1000000)
+ "ms");
}
return;
}
if (replyHdr.getXid() == -4) {
// -4 is the xid for AuthPacket
if(replyHdr.getErr() == KeeperException.Code.AUTHFAILED.intValue()) {
state = States.AUTH_FAILED;
eventThread.queueEvent( new WatchedEvent(Watcher.Event.EventType.None,
Watcher.Event.KeeperState.AuthFailed, null) );
}
if (LOG.isDebugEnabled()) {
LOG.debug("Got auth sessionid:0x"
+ Long.toHexString(sessionId));
}
return;
}
if (replyHdr.getXid() == -1) {
// -1 means notification
// 表示當前的消息類型爲一個 notification( 意味着是服務端的一個響應事件）
if (LOG.isDebugEnabled()) {
LOG.debug("Got notification sessionid:0x"
+ Long.toHexString(sessionId));
}
WatcherEvent event = new WatcherEvent();
// 反序列化響應信息
event.deserialize(bbia, "response");
// convert from a server path to a client path
if (chrootPath != null) {
String serverPath = event.getPath();
if(serverPath.compareTo(chrootPath)==0)
event.setPath("/");
else if (serverPath.length() > chrootPath.length())
event.setPath(serverPath.substring(chrootPath.length()));
else {
LOG.warn("Got server path " + event.getPath()
+ " which is too short for chroot path "
+ chrootPath);
}
}
WatchedEvent we = new WatchedEvent(event);
if (LOG.isDebugEnabled()) {
LOG.debug("Got " + we + " for sessionid 0x"
+ Long.toHexString(sessionId));
}
eventThread.queueEvent( we );
return;
}
// If SASL authentication is currently in progress, construct and
// send a response packet immediately, rather than queuing a
// response as with other packets.
if (clientTunneledAuthenticationInProgress()) {
GetSASLRequest request = new GetSASLRequest();
request.deserialize(bbia,"token");
zooKeeperSaslClient.respondToServer(request.getToken(),
ClientCnxn.this);
return;
}
Packet packet;
synchronized (pendingQueue) {
if (pendingQueue.size() == 0) {
throw new IOException("Nothing in the queue, but got "
+ replyHdr.getXid());
}
// 因爲當前這個數據包已經收到了響應，所以講它從 pendingQueued 中移除
packet = pendingQueue.remove();
}
/*
* Since requests are processed in order, we better get a response
* to the first request!
*/
try {// 校驗數據包信息，校驗成功後講數據包信息進行更新（替換爲服務端的信息）
if (packet.requestHeader.getXid() != replyHdr.getXid()) {
packet.replyHeader.setErr(
KeeperException.Code.CONNECTIONLOSS.intValue());
throw new IOException("Xid out of order. Got Xid "
+ replyHdr.getXid() + " with err " +
+ replyHdr.getErr() +
" expected Xid "
+ packet.requestHeader.getXid()
+ " for a packet with details: "
+ packet );
}
packet.replyHeader.setXid(replyHdr.getXid());
packet.replyHeader.setErr(replyHdr.getErr());
packet.replyHeader.setZxid(replyHdr.getZxid());
if (replyHdr.getZxid() > 0) {
lastZxid = replyHdr.getZxid();
}
if (packet.response != null && replyHdr.getErr() == 0) {
packet.response.deserialize(bbia, "response");
// 獲得服務端的響應，反序列化以後設置到 packet.response 屬性中。
// 所以我們可以在 exists 方法的最後一行通過 packet.response 拿到改請求的返回結果
}
if (LOG.isDebugEnabled()) {
LOG.debug("Reading reply sessionid:0x"
+ Long.toHexString(sessionId) + ", packet:: " + packet);
}
} finally {
// 最後調用 finishPacket 方法完成處理
finishPacket(packet);
}
}
這個方法裏面主要的流程如下首先讀取 header，如果其 xid == -2，表明是一個 ping 的 response，return。如果 xid 是 -4 ，表明是一個 AuthPacket 的 response return。如果 xid 是 -1，表明是一個 notification,此時要繼續讀取並構造一個 enent，通過EventThread.queueEvent 發送，return。其它情況下：從 pendingQueue 拿出一個 Packet，校驗後更新 packet 信息，最後調用 finishPacket 註冊本地事件：主要功能是把從 Packet 中取出對應的 Watcher 並註冊到 ZKWatchManager 中去
private void finishPacket(Packet p) {
// exists中初始化的 ExistsWatchRegistration
if (p.watchRegistration != null) {
// 將事件註冊到 zkwatchemanager 中
p.watchRegistration.register(p.replyHeader.getErr());
}
if (p.cb == null) {
synchronized (p) {
p.finished = true;
p.notifyAll();
}
} else {
p.finished = true;
//處理時間的線程進行處理
eventThread.queuePacket(p);
}
}
其中 watchRegistration 爲 exists 方法中初始化的 ExistsWatchRegistration，調用其註冊事件：
public void register(int rc) {
if (shouldAddWatch(rc)) {
Map<String, Set<Watcher>> watches = getWatches(rc);
synchronized(watches) {
Set<Watcher> watchers = watches.get(clientPath);
if (watchers == null) {
watchers = new HashSet<Watcher>();
watches.put(clientPath, watchers);
}
watchers.add(watcher);//初始化客戶端的時候自己定義的實現Watcher接口的類
}
}
}
//ExistsWatchRegistration.getWatches
protected Map<String, Set<Watcher>> getWatches(int rc) {
return rc == 0 ? watchManager.dataWatches : watchManager.existWatches;
}
而這裏的 ExistsWatchRegistration.getWatches 獲取到的集合在本場景下是獲取到的 dataWatches ：
private static class ZKWatchManager implements ClientWatchManager { private final Map<String, Set<Watcher>> dataWatches = new HashMap<String, Set<Watcher>>(); private final Map<String, Set<Watcher>> existWatches = new HashMap<String, Set<Watcher>>(); private final Map<String, Set<Watcher>> childWatches = new HashMap<String, Set<Watcher>>();
.......
總的來說，當使用 ZooKeeper 構造方法或者使用 getData、exists 和getChildren 三個接口來向 ZooKeeper 服務器註冊 Watcher 的時候，首先將此消息傳遞給服務端，傳遞成功後，服務端會通知客戶端，然後客戶端將該路徑和Watcher 對應關係存儲起來備用。finishPacket 方法最終會調用 eventThread.queuePacket， 將當前的數據包添加到等待事件通知的隊列中.
public void queuePacket(Packet packet) {
if (wasKilled) {
synchronized (waitingEvents) {
if (isRunning) waitingEvents.add(packet);
else processEvent(packet);
}
} else {
waitingEvents.add(packet);
}
}
事件觸發:
前面這麼長的說明，只是爲了清晰的說明事件的註冊流程，最終的觸發，還得需要通過事務型操作來完成在我們最開始的案例中，通過如下代碼去完成了事件的觸發zooKeeper.setData("/zk-wuzz","1".getBytes(),stat.getVersion()); 修改節點的值觸發監聽前面的客戶端和服務端對接的流程就不再重複講解了，交互流程是一樣的，唯一的差別在於事件觸發了.由於調用鏈路最終都會走到FinalRequestProcessor.processRequest：我們回到這個裏面：
public void processRequest(Request request) {
if (LOG.isDebugEnabled()) {
LOG.debug("Processing request:: " + request);
}
// request.addRQRec(">final");
long traceMask = ZooTrace.CLIENT_REQUEST_TRACE_MASK;
if (request.type == OpCode.ping) {
traceMask = ZooTrace.SERVER_PING_TRACE_MASK;
}
if (LOG.isTraceEnabled()) {
ZooTrace.logRequest(LOG, traceMask, 'E', request, "");
}
ProcessTxnResult rc = null;
synchronized (zks.outstandingChanges) {
while (!zks.outstandingChanges.isEmpty()
&& zks.outstandingChanges.get(0).zxid <= request.zxid) {
ChangeRecord cr = zks.outstandingChanges.remove(0);
if (cr.zxid < request.zxid) {
LOG.warn("Zxid outstanding "
+ cr.zxid
+ " is less than current " + request.zxid);
}
if (zks.outstandingChangesForPath.get(cr.path) == cr) {
zks.outstandingChangesForPath.remove(cr.path);
}
}//獲取header 不爲空
if (request.hdr != null) {
TxnHeader hdr = request.hdr;
Record txn = request.txn;
//事務請求會走這裏
rc = zks.processTxn(hdr, txn);
}
// do not add non quorum packets to the queue.
if (Request.isQuorum(request.type)) {
zks.getZKDatabase().addCommittedProposal(request);
}
}
.....//省略部分代碼
}
我們跟進 zks.processTxn(hdr, txn) :
public ProcessTxnResult processTxn(TxnHeader hdr, Record txn) {
ProcessTxnResult rc;
int opCode = hdr.getType();
long sessionId = hdr.getClientId();
//處理
rc = getZKDatabase().processTxn(hdr, txn);
if (opCode == OpCode.createSession) {
if (txn instanceof CreateSessionTxn) {
CreateSessionTxn cst = (CreateSessionTxn) txn;
sessionTracker.addSession(sessionId, cst
.getTimeOut());
} else {
LOG.warn("*****>>>>> Got "
+ txn.getClass() + " "
+ txn.toString());
}
} else if (opCode == OpCode.closeSession) {
sessionTracker.removeSession(sessionId);
}
return rc;
}
通過 getZKDatabase().processTxn(hdr, txn) 鏈路，最終會調用到 DataTree.processTxn(TxnHeader header, Record txn) :
public ProcessTxnResult processTxn(TxnHeader header, Record txn)
{
ProcessTxnResult rc = new ProcessTxnResult();
try {
rc.clientId = header.getClientId();
rc.cxid = header.getCxid();
rc.zxid = header.getZxid();
rc.type = header.getType();
rc.err = 0;
rc.multiResult = null;
switch (header.getType()) {
//省略代碼
case OpCode.setData:
SetDataTxn setDataTxn = (SetDataTxn) txn;
rc.path = setDataTxn.getPath();
rc.stat = setData(setDataTxn.getPath(), setDataTxn
.getData(), setDataTxn.getVersion(), header
.getZxid(), header.getTime());
break;
//省略代碼
}
} return rc;
}
在這裏我們會再進這個分支：
public Stat setData(String path, byte data[], int version, long zxid,
long time) throws KeeperException.NoNodeException {
Stat s = new Stat();
DataNode n = nodes.get(path);
if (n == null) {
throw new KeeperException.NoNodeException();
}
byte lastdata[] = null;
synchronized (n) {
lastdata = n.data;
n.data = data;
n.stat.setMtime(time);
n.stat.setMzxid(zxid);
n.stat.setVersion(version);
n.copyStat(s);
}
// now update if the path is in a quota subtree.
String lastPrefix;
if((lastPrefix = getMaxPrefixWithQuota(path)) != null) {
this.updateBytes(lastPrefix, (data == null ? 0 : data.length)
- (lastdata == null ? 0 : lastdata.length));
}
// 觸發對應節點的 NodeDataChanged 事件
dataWatches.triggerWatch(path, EventType.NodeDataChanged);
return s;
}
在這裏可以看到 ，在服務端的節點是利用 DataNode 來保存的，在保存好數據後會觸發對應節點的 NodeDataChanged 事件：
public Set<Watcher> triggerWatch(String path, EventType type, Set<Watcher> supress) {
// 根據事件類型、連接狀態、節點路徑創建 WatchedEvent
WatchedEvent e = new WatchedEvent(type,
KeeperState.SyncConnected, path);
HashSet<Watcher> watchers;
synchronized (this) {
// 從 watcher 表中移除 path ，並返回其對應的 watcher 集合
//這也是ZK默認事件只通知一次的原因
watchers = watchTable.remove(path);
if (watchers == null || watchers.isEmpty()) {
if (LOG.isTraceEnabled()) {
ZooTrace.logTraceMessage(LOG,
ZooTrace.EVENT_DELIVERY_TRACE_MASK,
"No watchers for " + path);
}
return null;
} // 遍歷 watcher 集合
for (Watcher w : watchers) {
// 根據 watcher 從 watcher 表中取出路徑集合
HashSet<String> paths = watch2Paths.get(w);
if (paths != null) {
paths.remove(path);// 移除路徑
}
}
}// 遍歷 watcher 集合
for (Watcher w : watchers) {
if (supress != null && supress.contains(w)) {
continue;
}//OK ，重點又來了， w.process 是做什麼呢？
w.process(e);
}
return watchers;
}
還記得我們在服務端綁定事件的時候，watcher 綁定是是什麼？是 ServerCnxn，所以 w.process(e)，其實調用的應該是 ServerCnxn 的 process 方法。而servercnxn 又是一個抽象方法，有兩個實現類，分別是：NIOServerCnxn 和 NettyServerCnxn。那接下來我們扒開 NIOServerCnxn 這個類的 process 方法看看究竟:
synchronized public void process(WatchedEvent event) {
ReplyHeader h = new ReplyHeader(-1, -1L, 0);
if (LOG.isTraceEnabled()) {
ZooTrace.logTraceMessage(LOG, ZooTrace.EVENT_DELIVERY_TRACE_MASK,
"Deliver event " + event + " to 0x"
+ Long.toHexString(this.sessionId)
+ " through " + this);
}
// Convert WatchedEvent to a type that can be sent over the wire
WatcherEvent e = event.getWrapper();
sendResponse(h, e, "notification");
}
那接下里，客戶端會收到這個 response，觸發 SendThread.readResponse 方法。
客戶端處理事件響應：
還是在不斷輪詢Selector ，所以這裏不管是客戶端的讀還是寫操作，都會進入ClientCnxnSocketNIO.doIO，然後我們直接進入 SendThread.readResponse：
void readResponse(ByteBuffer incomingBuffer) throws IOException {
ByteBufferInputStream bbis = new ByteBufferInputStream(
incomingBuffer);
BinaryInputArchive bbia = BinaryInputArchive.getArchive(bbis);
ReplyHeader replyHdr = new ReplyHeader();
// 反序列化 header
replyHdr.deserialize(bbia, "header");
//省略代碼
if (replyHdr.getXid() == -1) {
// -1 means notification
// 表示當前的消息類型爲一個 notification( 意味着是服務端的一個響應事件）
if (LOG.isDebugEnabled()) {
LOG.debug("Got notification sessionid:0x"
+ Long.toHexString(sessionId));
}
WatcherEvent event = new WatcherEvent();
// 反序列化響應信息
event.deserialize(bbia, "response");
// convert from a server path to a client path
if (chrootPath != null) {
String serverPath = event.getPath();
if(serverPath.compareTo(chrootPath)==0)
event.setPath("/");
else if (serverPath.length() > chrootPath.length())
event.setPath(serverPath.substring(chrootPath.length()));
else {
LOG.warn("Got server path " + event.getPath()
+ " which is too short for chroot path "
+ chrootPath);
}
}
WatchedEvent we = new WatchedEvent(event);
if (LOG.isDebugEnabled()) {
LOG.debug("Got " + we + " for sessionid 0x"
+ Long.toHexString(sessionId));
}
eventThread.queueEvent( we );
return;
}
.....//省略代碼
} finally {
// 最後調用 finishPacket 方法完成處理
finishPacket(packet);
}
}
這裏是客戶端處理事件回調，這裏傳過來的 xid 是等於 -1。SendThread 接收到服務端的通知事件後，會通過調用 EventThread 類的queueEvent 方法將事件傳給 EventThread 線程，queueEvent 方法根據該通知事件，從 ZKWatchManager 中取出所有相關的 Watcher，如果獲取到相應的 Watcher，就會讓 Watcher 移除失效:
public void queueEvent(WatchedEvent event) {
// 判斷類型
if (event.getType() == EventType.None
&& sessionState == event.getState()) {
return;
}
sessionState = event.getState();
// materialize the watchers based on the event
// 封裝 WatcherSetEventPair 對象，添加到 waitngEvents 隊列中
WatcherSetEventPair pair = new WatcherSetEventPair(
watcher.materialize(event.getState(), event.getType(),
event.getPath()),
event);
// queue the pair (watch set & event) for later processing
waitingEvents.add(pair);
}
其中Meterialize 方法是通過 dataWatches 或者 existWatches 或者 childWatches 的 remove 取出對應的watch，表明客戶端 watch 也是註冊一次就移除同時需要根據 keeperState、eventType 和 path 返回應該被通知的 Watcher 集合
這裏也進一步說明了zookeeper的watcher事件是不復用的，觸發一次就沒了，除非再註冊一次。
public Set<Watcher> materialize(Watcher.Event.KeeperState state,
Watcher.Event.EventType type,
String clientPath)
{
Set<Watcher> result = new HashSet<Watcher>();
switch (type) {
case None:
result.add(defaultWatcher);
boolean clear = ClientCnxn.getDisableAutoResetWatch() &&
state != Watcher.Event.KeeperState.SyncConnected;
synchronized(dataWatches) {
for(Set<Watcher> ws: dataWatches.values()) {
result.addAll(ws);
}
if (clear) {
dataWatches.clear();
}
}
synchronized(existWatches) {
for(Set<Watcher> ws: existWatches.values()) {
result.addAll(ws);
}
if (clear) {
existWatches.clear();
}
}
synchronized(childWatches) {
for(Set<Watcher> ws: childWatches.values()) {
result.addAll(ws);
}
if (clear) {
childWatches.clear();
}
}
return result;
case NodeDataChanged://節點變化
case NodeCreated://節點創建
synchronized (dataWatches) {
addTo(dataWatches.remove(clientPath), result);
}
synchronized (existWatches) {
addTo(existWatches.remove(clientPath), result);
}
break;
case NodeChildrenChanged://子節點變化
synchronized (childWatches) {
addTo(childWatches.remove(clientPath), result);
}
break;
case NodeDeleted://節點刪除
synchronized (dataWatches) {
addTo(dataWatches.remove(clientPath), result);
}
// XXX This shouldn't be needed, but just in case
synchronized (existWatches) {
Set<Watcher> list = existWatches.remove(clientPath);
if (list != null) {
addTo(list, result);
LOG.warn("We are triggering an exists watch for delete! Shouldn't happen!");
}
}
synchronized (childWatches) {
addTo(childWatches.remove(clientPath), result);
}
break;
default://默認
String msg = "Unhandled watch event type " + type
+ " with state " + state + " on path " + clientPath;
LOG.error(msg);
throw new RuntimeException(msg);
}
return result;
}
最後一步，接近真相了，waitingEvents 是 EventThread 這個線程中的阻塞隊列，很明顯，又是在我們第一步操作的時候實例化的一個線程。從名字可以知道，waitingEvents 是一個待處理 Watcher 的隊列，EventThread 的run() 方法會不斷從隊列中取數據，交由 processEvent 方法處理：
public void run() {
try {
isRunning = true;
while (true) {
Object event = waitingEvents.take();
if (event == eventOfDeath) {
wasKilled = true;
} else {
processEvent(event);
}
if (wasKilled)
synchronized (waitingEvents) {
if (waitingEvents.isEmpty()) {
isRunning = false;
break;
}
}
}
} catch (InterruptedException e) {
LOG.error("Event thread exiting due to interruption", e);
}
LOG.info("EventThread shut down for session: 0x{}",
Long.toHexString(getSessionId()));
}
繼而調用 processEvent(event)：
private void processEvent(Object event) {
try {// 判斷事件類型
if (event instanceof WatcherSetEventPair) {
// each watcher will process the event
// 得到 watcherseteventPair
WatcherSetEventPair pair = (WatcherSetEventPair) event;
// 拿到符合觸發機制的所有 watcher 列表，循環進行調用
for (Watcher watcher : pair.watchers) {
try {// 調用客戶端的回
watcher.process(pair.event);
} catch (Throwable t) {
LOG.error("Error while calling watcher ", t);
}
}
} else {
。。。。//省略代碼
}
}
最後調用到自定義的 Watcher 處理類。至此整個Watcher 事件處理完畢。