dubbo服務消費過程(二)

服務的訂閱和通知

Directory

Directory 繼承自 Node 接口，Node 這個接口繼承者比較多，像 Registry、Monitor、Invoker 等均繼承了這個接口。這個接口包含了一個獲取配置信息的方法 getUrl，實現該接口的類可以向外提供配置信息。另外，大家注意看 RegistryDirectory 實現了 NotifyListener 接口，當註冊中心節點信息發生變化後，RegistryDirectory 可以通過此接口方法得到變更信息，並根據變更信息動態調整內部 Invoker 列表

在容錯過程中(如FailoverClusterInvoker)會使用Directory#list來獲取所有的invoke的列表。
模板模式

• Directory頂層接口
• AbstractDirectory封裝了通用實現邏輯:list等
• StaticDirectory Directory的靜態列表實現，提供靜態的Invoke列表，將傳入的Invoker列表封裝成靜態的Directory對象，裏面的列表不會改變，在ReferenceConfig#createProxy使用CLUSTER.join(new StaticDirectory(invokers))
• RegistryDirectoryDirectory的動態列表實現提供動態的Invoke列表。會自動從註冊中心更新Invoker列表，配置信息、路由信息。

RegistryDirectory
RegistryDirectory實現兩點：
(1) 框架與註冊中心的訂閱，並動態更新本地Invoker列表、路由列表、配置信息的邏輯
(2) 實現父類的doList方法

訂閱與動態更新主要方法subscribe、notify、refreshInvoker，輔助方法toConfigurators、toRouter

服務的訂閱subscribe

註冊中心ZK 的節點訂閱和通知
調用鏈
RegistryProtocol#refer
-> RegistryProtocol#doRefer
-> directory#subscribe
-> registry#subscribe -->這個registry 是ZookeeperRegistry

ZookeeperRegistry父類FailbackRegistry#subscribe ->ZookeeperRegistry.doSubscribe()

會去監聽下面的節點的路徑的子節點變動

/dubbo/org.apache.dubbo.demo.DemoService/providers
/dubbo/org.apache.dubbo.demo.DemoService/configurators
/dubbo/org.apache.dubbo.de mo.DemoService/routers

directory#subscribe

訂閱某個URL的更新信息。Dubbo在引用每個需要RPC調用(refer)Bean的時候，會調用directory.subscribe訂閱這個bean的各種URL變化

public void subscribe(URL url) { 
	  // 設置consumerUrl 
      setConsumerUrl(url);
      // 把當前的RegistryDriectory作爲listener，去監聽zk節點的變化
      CONSUMER_CONFIGURATION_LISTENER.addNotifyListener(this);
      serviceConfigurationListener = new ReferenceConfigurationListener(this, url);
      registry.subscribe(url, this);
  }

FailbackRegistry#subscribe

移除失效的listener，調用doSubscribe進行訂閱

// listener爲RegistryDirectory
public void subscribe(URL url, NotifyListener listener) {
    super.subscribe(url, listener);
    // 移除失效的listener
    removeFailedSubscribed(url, listener); 
    try {
        // 調用ZookeeperRegistry.doSubscribe 實現 
        doSubscribe(url, listener);
    } catch (Exception e) {
        Throwable t = e;

        List<URL> urls = getCacheUrls(url);
        if (CollectionUtils.isNotEmpty(urls)) {
            notify(url, listener, urls);
            logger.error("Failed to subscribe " + url + ", Using cached list: " + urls + " from cache file: " + getUrl().getParameter(FILE_KEY, System.getProperty("user.home") + "/dubbo-registry-" + url.getHost() + ".cache") + ", cause: " + t.getMessage(), t);
        } else {
            // If the startup detection is opened, the Exception is thrown directly.
            boolean check = getUrl().getParameter(Constants.CHECK_KEY, true)
                    && url.getParameter(Constants.CHECK_KEY, true);
            boolean skipFailback = t instanceof SkipFailbackWrapperException;
            if (check || skipFailback) {
                if (skipFailback) {
                    t = t.getCause();
                }
                throw new IllegalStateException("Failed to subscribe " + url + ", cause: " + t.getMessage(), t);
            } else {
                logger.error("Failed to subscribe " + url + ", waiting for retry, cause: " + t.getMessage(), t);
            }
        }
        // Record a failed registration request to a failed list, retry regularly
        addFailedSubscribed(url, listener); // 失敗重試
    }
}

ZookeeperRegistry#doSubscribe

主要實現把所有Service層發起的訂閱以及指定的Service層發起的訂閱分開處理。

• 所有Service層類似於監控中心發起的訂閱。
• 指定的Service層發起的訂閱可以看作是服務消費者的訂閱。

    @Override
    public void doSubscribe(final URL url, final NotifyListener listener) {
        try {
            if (ANY_VALUE.equals(url.getServiceInterface())) { // 這裏主要用與服務端和zk連接 註冊服務
               ... // 省略部分代碼
            } else { // 消費端和zk連接 訂閱節點 
                List<URL> urls = new ArrayList<>();
                for (String path : toCategoriesPath(url)) {
                    ConcurrentMap<NotifyListener, ChildListener> listeners = zkListeners.get(url);
                    if (listeners == null) {
                       // 如果之前該路徑沒有添加過listener，則創建一個map來放置listener
                        zkListeners.putIfAbsent(url, new ConcurrentHashMap<>());
                        listeners = zkListeners.get(url);
                    }
                    ChildListener zkListener = listeners.get(listener);
                    if (zkListener == null) {
                       // 如果沒有添加過對於子節點的listener，則創建,通知服務變化 回調NotifyListener
                        listeners.putIfAbsent(listener, (parentPath, currentChilds) -> ZookeeperRegistry.this.notify(url, listener, toUrlsWithEmpty(url, parentPath, currentChilds)));
                        zkListener = listeners.get(listener);
                    }
                    zkClient.create(path, false);
                    //添加path節點的當前節點及子節點監聽，並且獲取子節點信息 
                    //也就是dubbo://ip:port/...
                    List<String> children = zkClient.addChildListener(path, zkListener);
                    if (children != null) {
                        urls.addAll(toUrlsWithEmpty(url, path, children));
                    }
                }
                // 調用notify進行通知，對已經可用的列表進行通知
                notify(url, listener, urls);
            }
        } catch (Throwable e) {
            throw new RpcException("Failed to subscribe " + url + " to zookeeper " + getUrl() + ", cause: " + e.getMessage(), e);
        }
    }

服務通知notify

FailbackRegistry.notify

調用FailbackRegistry.notify， 對參數進行判斷。 然後調用AbstractRegistry.notify方法

 protected void notify(URL url, NotifyListener listener, List<URL> urls) {
        if (url == null) {
            throw new IllegalArgumentException("notify url == null");
        }
        if (listener == null) {
            throw new IllegalArgumentException("notify listener == null");
        }
        try {
            doNotify(url, listener, urls);
        } catch (Exception t) {
            // Record a failed registration request to a failed list, retry regularly
            addFailedNotified(url, listener, urls);
            logger.error("Failed to notify for subscribe " + url + ", waiting for retry, cause: " + t.getMessage(), t);
        }
    }

 protected void doNotify(URL url, NotifyListener listener, List<URL> urls) {
     super.notify(url, listener, urls);
 }

AbstractRegistry.notify

protected void notify(URL url, NotifyListener listener, List<URL> urls) {
      if (url == null) {
          throw new IllegalArgumentException("notify url == null");
      }
      if (listener == null) {
          throw new IllegalArgumentException("notify listener == null");
      }
      if ((CollectionUtils.isEmpty(urls))
              && !ANY_VALUE.equals(url.getServiceInterface())) {
          logger.warn("Ignore empty notify urls for subscribe url " + url);
          return;
      }
      if (logger.isInfoEnabled()) {
          logger.info("Notify urls for subscribe url " + url + ", urls: " + urls);
      }
      // keep every provider's category.
      Map<String, List<URL>> result = new HashMap<>();
      for (URL u : urls) {
          if (UrlUtils.isMatch(url, u)) {
              String category = u.getParameter(CATEGORY_KEY, DEFAULT_CATEGORY);
              List<URL> categoryList = result.computeIfAbsent(category, k -> new ArrayList<>());
              categoryList.add(u);
          }
      }
      if (result.size() == 0) {
          return;
      }
      // 會針對每一個category，調用listener.notify進行通知，然後更新本地的緩存文件
      Map<String, List<URL>> categoryNotified = notified.computeIfAbsent(url, u -> new ConcurrentHashMap<>());
      for (Map.Entry<String, List<URL>> entry : result.entrySet()) {
          String category = entry.getKey();
          List<URL> categoryList = entry.getValue();
          categoryNotified.put(category, categoryList);
          listener.notify(categoryList); // 觸發listener.notify  RegisteryDirectory.notify 
          saveProperties(url);
      }
  }

RegisteryDirectory.notify

接收服務變更通知, 監聽到配置中心對應的URL變化，然後更新本地的配置參數。

(1) 新建三個list，分別用於保存更新的InvokerURL、路由配置URL、配置URL。遍歷監聽返回所有的url，分類放入三個list中
(2) 解析並更新配置參數:

• 對於router類型參數，首先遍歷所有的router類型URL，然後通過router工廠RouterFactory把每個URL包裝成路由規則，最後更新本地的路由信息。這個過程會忽略以empty開頭的URL
• 對於Configurator類的參數，管理員可以在dubbo-admin動態配置功能上修改生產者的參數，這些參數會保存在配置中心的configurator類目錄下。notify監聽到的url配置變化，會解析並更新本地的Configurator配置
• 對於Invoker類型的參數，如果是empty協議的URL,則會禁用該服務，並銷燬本地緩存的Invoker；如果監聽到的Invoker類型URL都是空的，則說明沒有更新，直接使用本地的緩存；如果監聽到的URL不爲空，則把更新的URL和本地老的URL合併，創建新的Invoker，找出差異的老Invoker並銷燬

    public synchronized void notify(List<URL> urls) {
    	//對url列表進行校驗、過濾，然後分成 config、router、provider 3個分組map
        Map<String, List<URL>> categoryUrls = urls.stream()
                .filter(Objects::nonNull)
                .filter(this::isValidCategory)
                .filter(this::isNotCompatibleFor26x)
                .collect(Collectors.groupingBy(url -> {
                    if (UrlUtils.isConfigurator(url)) {
                        return CONFIGURATORS_CATEGORY;
                    } else if (UrlUtils.isRoute(url)) {
                        return ROUTERS_CATEGORY;
                    } else if (UrlUtils.isProvider(url)) {
                        return PROVIDERS_CATEGORY;
                    }
                    return "";
                }));
		// 定義配置器url的list
        List<URL> configuratorURLs = categoryUrls.getOrDefault(CONFIGURATORS_CATEGORY, Collections.emptyList());
        // 將url轉換成configurator
        this.configurators = Configurator.toConfigurators(configuratorURLs).orElse(this.configurators);
		// 路由url
        List<URL> routerURLs = categoryUrls.getOrDefault(ROUTERS_CATEGORY, Collections.emptyList());  
        // 將url轉成rounter     
        toRouters(routerURLs).ifPresent(this::addRouters);
        // provider url
        List<URL> providerURLs = categoryUrls.getOrDefault(PROVIDERS_CATEGORY, Collections.emptyList());
        /**
         * 3.x added for extend URL address
         */
        ExtensionLoader<AddressListener> addressListenerExtensionLoader = ExtensionLoader.getExtensionLoader(AddressListener.class);
        List<AddressListener> supportedListeners = addressListenerExtensionLoader.getActivateExtension(getUrl(), (String[]) null);
        if (supportedListeners != null && !supportedListeners.isEmpty()) {
            for (AddressListener addressListener : supportedListeners) {
                providerURLs = addressListener.notify(providerURLs, getUrl(),this);
            }
        }
        // 刷新和覆蓋 invokerlist
        refreshOverrideAndInvoker(providerURLs);
    }
    /**
     * 逐個調用註冊中心裏面的配置，覆蓋原來的url，組成最新的url 放入overrideDirectoryUrl 存儲
     * 根據 provider urls，重新刷新Invoker
     */
    private void refreshOverrideAndInvoker(List<URL> urls) {
        // mock zookeeper://xxx?mock=return null
        overrideDirectoryUrl();
        refreshInvoker(urls);
    }

服務override和refresh

private volatile URL overrideDirectoryUrl;
private volatile URL registeredConsumerUrl;

overrideDirectoryUrl 的參數可以通過dubbo-admin上的服務參數重寫
規則：override > -D >Consumer > Provider

RegistryDirectory .overrideDirectoryUrl

dubo-admin上更新路由規則或參數是通過override://協議實現的

private void overrideDirectoryUrl() {
      // merge override parameters
      this.overrideDirectoryUrl = directoryUrl;
      // 本地Configurator 重寫
      List<Configurator> localConfigurators = this.configurators; // local reference
      doOverrideUrl(localConfigurators);
      List<Configurator> localAppDynamicConfigurators = CONSUMER_CONFIGURATION_LISTENER.getConfigurators(); // local reference
      // 消費端 重寫
      doOverrideUrl(localAppDynamicConfigurators);
      if (serviceConfigurationListener != null) {
          List<Configurator> localDynamicConfigurators = serviceConfigurationListener.getConfigurators(); // local reference
          // 服務端的重寫
          doOverrideUrl(localDynamicConfigurators);
      }
  }
  private void doOverrideUrl(List<Configurator> configurators) {
     if (CollectionUtils.isNotEmpty(configurators)) {
         for (Configurator configurator : configurators) {
             this.overrideDirectoryUrl = configurator.configure(overrideDirectoryUrl);
         }
  }  

RegistryDirectory.refreshInvoker

refreshInvoker 方法是保證 RegistryDirectory 隨註冊中心變化而變化的關鍵所在

    private void refreshInvoker(List<URL> invokerUrls) {
        Assert.notNull(invokerUrls, "invokerUrls should not be null");
    	// invokerUrls 僅有一個元素，且 url 協議頭爲 empty，此時表示禁用所有服務
        if (invokerUrls.size() == 1
                && invokerUrls.get(0) != null
                && EMPTY_PROTOCOL.equals(invokerUrls.get(0).getProtocol())) {
            // 設置 forbidden 爲 true
            this.forbidden = true; // Forbid to access
            this.invokers = Collections.emptyList();
            routerChain.setInvokers(this.invokers);
            // 銷燬所有 Invoker
            destroyAllInvokers(); // Close all invokers
        } else {
            this.forbidden = false; // Allow to access
            Map<String, Invoker<T>> oldUrlInvokerMap = this.urlInvokerMap; // local reference
            if (invokerUrls == Collections.<URL>emptyList()) {
                invokerUrls = new ArrayList<>();
            }
            if (invokerUrls.isEmpty() && this.cachedInvokerUrls != null) {
            	// 添加緩存 url 到 invokerUrls 中
                invokerUrls.addAll(this.cachedInvokerUrls);
            } else {
            	// 緩存 invokerUrls
                this.cachedInvokerUrls = new HashSet<>();
                this.cachedInvokerUrls.addAll(invokerUrls);//Cached invoker urls, convenient for comparison
            }
            if (invokerUrls.isEmpty()) {
                return;
            }
            // 將 url 轉成 Invoker
            Map<String, Invoker<T>> newUrlInvokerMap = toInvokers(invokerUrls);// Translate url list to Invoker map

            if (CollectionUtils.isEmptyMap(newUrlInvokerMap)) {
                logger.error(new IllegalStateException("urls to invokers error .invokerUrls.size :" + invokerUrls.size() + ", invoker.size :0. urls :" + invokerUrls
                        .toString()));
                return;
            }

            List<Invoker<T>> newInvokers = Collections.unmodifiableList(new ArrayList<>(newUrlInvokerMap.values()));
            // pre-route and build cache, notice that route cache should build on original Invoker list.
            // toMergeMethodInvokerMap() will wrap some invokers having different groups, those wrapped invokers not should be routed.
            // 重置路由invoker
            routerChain.setInvokers(newInvokers);
            // 合併多個組的Invoker
            //如果服務配置了分組，則把分組下的provider包裝成StaticDirectory,組成一個 invoker 
            //實際上就是按照group進行合併
            this.invokers = multiGroup ? toMergeInvokerList(newInvokers) : newInvokers;
            this.urlInvokerMap = newUrlInvokerMap;

            try {
            	// 銷燬無用 Invoker
                destroyUnusedInvokers(oldUrlInvokerMap, newUrlInvokerMap); // Close the unused Invoker
            } catch (Exception e) {
                logger.warn("destroyUnusedInvokers error. ", e);
            }
        }
    }

服務的列舉list

AbstractDirectory#list

 @Override
 public List<Invoker<T>> list(Invocation invocation) throws RpcException {
     if (destroyed) {
         throw new RpcException("Directory already destroyed .url: " + getUrl());
     }
   return doList(invocation);
 }

RegistryDirectory#dolist

列舉Invoker，在AbstractDirectory#list中調用子類模板方法dolist

 @Override
  public List<Invoker<T>> doList(Invocation invocation) {
       if (forbidden) {
          // 服務提供者關閉或禁用了服務，此時拋出 No provider 異常
           // 1. No service provider 2. Service providers are disabled
           throw new RpcException(RpcException.FORBIDDEN_EXCEPTION, "No provider available from registry " +
                   getUrl().getAddress() + " for service " + getConsumerUrl().getServiceKey() + " on consumer " +
                   NetUtils.getLocalHost() + " use dubbo version " + Version.getVersion() +
                   ", please check status of providers(disabled, not registered or in blacklist).");
       }
	  // 判斷是否需要分組路由
       if (multiGroup) {// true  直接返回invokers 
           return this.invokers == null ? Collections.emptyList() : this.invokers;
       }
       List<Invoker<T>> invokers = null;
       try {
           // Get invokers from cache, only runtime routers will be executed.
           // 通過路由規則 選出 invokers
           invokers = routerChain.route(getConsumerUrl(), invocation);
       } catch (Throwable t) {
           logger.error("Failed to execute router: " + getUrl() + ", cause: " + t.getMessage(), t);
       }
       return invokers == null ? Collections.emptyList() : invokers;
   }

服務url->InvokertoInvokers

將服務的url轉成invoker
RegistryDirectory#toInvokers

  private Map<String, Invoker<T>> toInvokers(List<URL> urls) {
        Map<String, Invoker<T>> newUrlInvokerMap = new HashMap<>();
        if (urls == null || urls.isEmpty()) {
            return newUrlInvokerMap;
        }
        Set<String> keys = new HashSet<>();
        // 獲取服務消費端配置的協議
        String queryProtocols = this.queryMap.get(PROTOCOL_KEY);
        for (URL providerUrl : urls) {
            // If protocol is configured at the reference side, only the matching protocol is selected
            if (queryProtocols != null && queryProtocols.length() > 0) {
                boolean accept = false;
                String[] acceptProtocols = queryProtocols.split(",");
                // 檢測服務提供者協議是否被服務消費者所支持
                for (String acceptProtocol : acceptProtocols) {
                    if (providerUrl.getProtocol().equals(acceptProtocol)) {
                        accept = true;
                        break;
                    }
                }
                if (!accept) {
                    // 若服務消費者協議頭不被消費者所支持，則忽略當前 providerUrl
                    continue;
                }
            }
            // 忽略 empty 協議
            if (EMPTY_PROTOCOL.equals(providerUrl.getProtocol())) {
                continue;
            }
            // 通過 SPI 檢測服務端協議是否被消費端支持，不支持則拋出異常
            if (!ExtensionLoader.getExtensionLoader(Protocol.class).hasExtension(providerUrl.getProtocol())) {
                logger.error(new IllegalStateException("Unsupported protocol " + providerUrl.getProtocol() +
                        " in notified url: " + providerUrl + " from registry " + getUrl().getAddress() +
                        " to consumer " + NetUtils.getLocalHost() + ", supported protocol: " +
                        ExtensionLoader.getExtensionLoader(Protocol.class).getSupportedExtensions()));
                continue;
            }
            // 合併url 規則：override > -D >Consumer > Provider
            URL url = mergeUrl(providerUrl);
            String key = url.toFullString(); // The parameter urls are sorted
            if (keys.contains(key)) { // Repeated url
                continue; // 忽略重複 url
            }
            keys.add(key);
            // 將本地 Invoker 緩存賦值給 localUrlInvokerMap
            Map<String, Invoker<T>> localUrlInvokerMap = this.urlInvokerMap; // local reference
            // 獲取與 url 對應的 Invoker
            Invoker<T> invoker = localUrlInvokerMap == null ? null : localUrlInvokerMap.get(key);
            if (invoker == null) { // 緩存未命中
                try {
                    boolean enabled = true;
                    if (url.hasParameter(DISABLED_KEY)) {
                        // 獲取 disable 配置，取反，然後賦值給 enable 變量
                        enabled = !url.getParameter(DISABLED_KEY, false);
                    } else {
                        // 獲取 enable 配置，並賦值給 enable 變量
                        enabled = url.getParameter(ENABLED_KEY, true);
                    }
                    if (enabled) {
                        // 調用 refer 獲取 Invoker protocol-> xxxWrapper(DubboProtocol)
                        // InvokerDelegate(ProtocolFilterWrapper(ListenerInvokerWrapper(DubboInvoker())
                        invoker = new InvokerDelegate<>(protocol.refer(serviceType, url), url, providerUrl);
                    }
                } catch (Throwable t) {
                    logger.error("Failed to refer invoker for interface:" + serviceType + ",url:(" + url + ")" + t.getMessage(), t);
                }
                if (invoker != null) { // Put new invoker in cache
                   // 緩存 Invoker 實例
                    newUrlInvokerMap.put(key, invoker);
                }
            } else { // 緩存命中
               // 將 invoker 存儲到 newUrlInvokerMap 中
                newUrlInvokerMap.put(key, invoker);
            }
        }
        keys.clear();
        return newUrlInvokerMap;
    }

客戶端連接

在將url轉換成invoker時RegistryDirectory#toInvokers中的DubboProtocol建立連接

invoker = new InvokerDelegate<>(protocol.refer(serviceType, url), url, providerUrl);

DubboProtocol.refer
->AbstractProtocol#refer
->DubboProtocol#protocolBindingRefer
->DubboProtocol#getClients

DubboProtocol#getClients

獲得客戶端連接的方法

• 判斷是否爲共享連接，默認是共享同一個連接進行通信
• 是否配置了多個連接通道 connections，默認只有一個

   public <T> Invoker<T> protocolBindingRefer(Class<T> serviceType, URL url) throws RpcException {
        optimizeSerialization(url); //優化序列化
        // create rpc invoker. 構建DbbuoInvoker
        DubboInvoker<T> invoker = new DubboInvoker<T>(serviceType, url, getClients(url), invokers);
        invokers.add(invoker);
        return invoker;
    }
	// 獲取ExchangeClient 服務通信連接
    private ExchangeClient[] getClients(URL url) {
        // whether to share connection
        boolean useShareConnect = false;
        int connections = url.getParameter(CONNECTIONS_KEY, 0);
        List<ReferenceCountExchangeClient> shareClients = null;
        // if not configured, connection is shared, otherwise, one connection for one service
        if (connections == 0) { //如果沒有配置連接數，則默認爲共享連接
            useShareConnect = true;
            String shareConnectionsStr = url.getParameter(SHARE_CONNECTIONS_KEY, (String) null);
            connections = Integer.parseInt(StringUtils.isBlank(shareConnectionsStr) ? ConfigUtils.getProperty(SHARE_CONNECTIONS_KEY,
                    DEFAULT_SHARE_CONNECTIONS) : shareConnectionsStr);
            shareClients = getSharedClient(url, connections);
        }
        ExchangeClient[] clients = new ExchangeClient[connections];
        for (int i = 0; i < clients.length; i++) {
            if (useShareConnect) {
                clients[i] = shareClients.get(i);

            } else {
                clients[i] = initClient(url);
            }
        }
        return clients;
    }

DubboProtocol#getSharedClient

獲得一個共享連接

 private List<ReferenceCountExchangeClient> getSharedClient(URL url, int connectNum) {
        String key = url.getAddress();
        List<ReferenceCountExchangeClient> clients = referenceClientMap.get(key);
        //檢查當前的key檢查連接是否已經創建過並且可用，如果是，則直接返回並且增加連接的個數的統 計
        if (checkClientCanUse(clients)) {
            batchClientRefIncr(clients);
            return clients;
        }
        //如果連接已經關閉或者連接沒有創建過
        locks.putIfAbsent(key, new Object());
        synchronized (locks.get(key)) {
            clients = referenceClientMap.get(key);
            // // 在創建連接之前，在做一次檢查，防止連接併發創建
            if (checkClientCanUse(clients)) {
                batchClientRefIncr(clients);
                return clients;
            }
            // 連接數必須大於等於1
            connectNum = Math.max(connectNum, 1);
			//如果當前消費者還沒有和服務端產生連接，則初始化
            if (CollectionUtils.isEmpty(clients)) {
                clients = buildReferenceCountExchangeClientList(url, connectNum);
                //創建clients之後，保存到map中
                referenceClientMap.put(key, clients);
            } else { //如果clients不爲空，則從clients數組中進行遍歷
                for (int i = 0; i < clients.size(); i++) {
                    ReferenceCountExchangeClient referenceCountExchangeClient = clients.get(i);
                  	// 如果在集合中存在一個連接但是這個連接處於closed狀態，則重新構建一個 進行替換
                    if (referenceCountExchangeClient == null || referenceCountExchangeClient.isClosed()) {
                        clients.set(i, buildReferenceCountExchangeClient(url));
                        continue;
                    }
					//增加個數
                    referenceCountExchangeClient.incrementAndGetCount();
                }
            }
            /**
             * I understand that the purpose of the remove operation here is to avoid the expired url key
             * always occupying this memory space.
             */
            locks.remove(key);
            return clients;
        }
    }

DubboProtocol#buildReferenceCountExchangeClientList

  private List<ReferenceCountExchangeClient> buildReferenceCountExchangeClientList(URL url, int connectNum) {
        List<ReferenceCountExchangeClient> clients = new ArrayList<>();
        for (int i = 0; i < connectNum; i++) {
            clients.add(buildReferenceCountExchangeClient(url));
        }
        return clients;
    }

    private ReferenceCountExchangeClient buildReferenceCountExchangeClient(URL url) {
        ExchangeClient exchangeClient = initClient(url);
        return new ReferenceCountExchangeClient(exchangeClient);
    }

DubboProtocol#initClient

根據url構建連接

 private ExchangeClient initClient(URL url) {
        // 獲得連接類型
        String str = url.getParameter(CLIENT_KEY, url.getParameter(SERVER_KEY, DEFAULT_REMOTING_CLIENT));
        // 序列化協議
        url = url.addParameter(CODEC_KEY, DubboCodec.NAME);
        // 心跳  
        url = url.addParameterIfAbsent(HEARTBEAT_KEY, String.valueOf(DEFAULT_HEARTBEAT));
        // BIO is not allowed since it has severe performance issue.
        // 判斷str是否存在於擴展點中，如果不存在則直接報錯
        if (str != null && str.length() > 0 && !ExtensionLoader.getExtensionLoader(Transporter.class).hasExtension(str)) {
            throw new RpcException("Unsupported client type: " + str + "," +
                    " supported client type is " + StringUtils.join(ExtensionLoader.getExtensionLoader(Transporter.class).getSupportedExtensions(), " "));
        }
        ExchangeClient client;
        try {
            // connection should be lazy
            // 是否需要延遲創建連接， 這裏的requestHandler是一個適配器
            if (url.getParameter(LAZY_CONNECT_KEY, false)) {
                client = new LazyConnectExchangeClient(url, requestHandler);
            } else {
                client = Exchangers.connect(url, requestHandler);
            }
        } catch (RemotingException e) {
            throw new RpcException("Fail to create remoting client for service(" + url + "): " + e.getMessage(), e);
        }
        return client;
    }

Exchangers.connect

通過SPI創建一個客戶端連接,默認HeaderExchanger

 public static ExchangeClient connect(URL url, ExchangeHandler handler) throws RemotingException {
        if (url == null) {
            throw new IllegalArgumentException("url == null");
        }
        if (handler == null) {
            throw new IllegalArgumentException("handler == null");
        }
        url = url.addParameterIfAbsent(Constants.CODEC_KEY, "exchange");
        return getExchanger(url).connect(url, handler);
    }
 public static Exchanger getExchanger(URL url) {
      String type = url.getParameter(Constants.EXCHANGER_KEY, Constants.DEFAULT_EXCHANGER);
      return getExchanger(type);
  }
  public static Exchanger getExchanger(String type) {
      return ExtensionLoader.getExtensionLoader(Exchanger.class).getExtension(type);
  }

HeaderExchanger.connect

@Override
public ExchangeClient connect(URL url, ExchangeHandler handler) throws RemotingException {
   // NettyTransport.connect
   return new HeaderExchangeClient(Transporters.connect(url, new DecodeHandler(new HeaderExchangeHandler(handler))), true);
 }

Transporters.connect

 public static Client connect(URL url, ChannelHandler... handlers) throws RemotingException {
      if (url == null) {
          throw new IllegalArgumentException("url == null");
      }
      ChannelHandler handler;
      if (handlers == null || handlers.length == 0) {
          handler = new ChannelHandlerAdapter();
      } else if (handlers.length == 1) {
          handler = handlers[0];
      } else {
          handler = new ChannelHandlerDispatcher(handlers);
      }
      return getTransporter().connect(url, handler);
  }

   public static Transporter getTransporter() {
       return ExtensionLoader.getExtensionLoader(Transporter.class).getAdaptiveExtension();
   }

NettyTransport.connect

構建一個netty的客戶端連接，終於到這，不容易 ，後面是netty的知識可以參考netty章節的博客

@Override
public Client connect(URL url, ChannelHandler listener) throws RemotingException {
    return new NettyClient(url, listener);
}

NettyClient

    protected void doOpen() throws Throwable {
        final NettyClientHandler nettyClientHandler = new NettyClientHandler(getUrl(), this);
        bootstrap = new Bootstrap();
        bootstrap.group(nioEventLoopGroup)
                .option(ChannelOption.SO_KEEPALIVE, true)
                .option(ChannelOption.TCP_NODELAY, true)
                .option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT)
                //.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, getTimeout())
                .channel(NioSocketChannel.class);

        bootstrap.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, Math.max(3000, getConnectTimeout()));
        bootstrap.handler(new ChannelInitializer() {

            @Override
            protected void initChannel(Channel ch) throws Exception {
                int heartbeatInterval = UrlUtils.getHeartbeat(getUrl());

                if (getUrl().getParameter(SSL_ENABLED_KEY, false)) {
                    ch.pipeline().addLast("negotiation", SslHandlerInitializer.sslClientHandler(getUrl(), nettyClientHandler));
                }

                NettyCodecAdapter adapter = new NettyCodecAdapter(getCodec(), getUrl(), NettyClient.this);
                ch.pipeline()//.addLast("logging",new LoggingHandler(LogLevel.INFO))//for debug
                        .addLast("decoder", adapter.getDecoder())
                        .addLast("encoder", adapter.getEncoder())
                        .addLast("client-idle-handler", new IdleStateHandler(heartbeatInterval, 0, 0, MILLISECONDS))
                        .addLast("handler", nettyClientHandler);

                String socksProxyHost = ConfigUtils.getProperty(SOCKS_PROXY_HOST);
                if(socksProxyHost != null) {
                    int socksProxyPort = Integer.parseInt(ConfigUtils.getProperty(SOCKS_PROXY_PORT, DEFAULT_SOCKS_PROXY_PORT));
                    Socks5ProxyHandler socks5ProxyHandler = new Socks5ProxyHandler(new InetSocketAddress(socksProxyHost, socksProxyPort));
                    ch.pipeline().addFirst(socks5ProxyHandler);
                }
            }
        });
    }

    @Override
    protected void doConnect() throws Throwable {
        long start = System.currentTimeMillis();
        ChannelFuture future = bootstrap.connect(getConnectAddress());
        try {
            boolean ret = future.awaitUninterruptibly(getConnectTimeout(), MILLISECONDS);

            if (ret && future.isSuccess()) {
                Channel newChannel = future.channel();
                try {
                    // Close old channel
                    // copy reference
                    Channel oldChannel = NettyClient.this.channel;
                    if (oldChannel != null) {
                        try {
                            if (logger.isInfoEnabled()) {
                                logger.info("Close old netty channel " + oldChannel + " on create new netty channel " + newChannel);
                            }
                            oldChannel.close();
                        } finally {
                            NettyChannel.removeChannelIfDisconnected(oldChannel);
                        }
                    }
                } finally {
                    if (NettyClient.this.isClosed()) {
                        try {
                            if (logger.isInfoEnabled()) {
                                logger.info("Close new netty channel " + newChannel + ", because the client closed.");
                            }
                            newChannel.close();
                        } finally {
                            NettyClient.this.channel = null;
                            NettyChannel.removeChannelIfDisconnected(newChannel);
                        }
                    } else {
                        NettyClient.this.channel = newChannel;
                    }
                }
            } else if (future.cause() != null) {
                throw new RemotingException(this, "client(url: " + getUrl() + ") failed to connect to server "
                        + getRemoteAddress() + ", error message is:" + future.cause().getMessage(), future.cause());
            } else {
                throw new RemotingException(this, "client(url: " + getUrl() + ") failed to connect to server "
                        + getRemoteAddress() + " client-side timeout "
                        + getConnectTimeout() + "ms (elapsed: " + (System.currentTimeMillis() - start) + "ms) from netty client "
                        + NetUtils.getLocalHost() + " using dubbo version " + Version.getVersion());
            }
        } finally {
            // just add new valid channel to NettyChannel's cache
            if (!isConnected()) {
                //future.cancel(true);
            }
        }
    }

RegistryProtocol.refer 過程中有一個關鍵步驟，即在監聽到服務提供者url時觸發RegistryDirectory.notify() 方法。
RegistryDirectory.notify() 方法調用 refreshInvoker() 方法將服務提供者urls轉換爲對應的 遠 程invoker ，最終調用到 DubboProtocol.refer()方法生成對應的 DubboInvoker 。 DubboInvoker 的構造方法中有一項入參 ExchangeClient[] clients ，即對應本文要講的網絡客戶
端 Client 。DubboInvoker就是通過調用 client.request() 方法完成網絡通信的請求發送和響應接收功能。Client 的具體生成過程就是通過 DubboProtocol 的 initClient(URL url) 方法創建了一個HeaderExchangeClient 。