Dubbo源碼學習09

RegistryProtocol.export服務導出流程：導出服務ExporterChangeableWrapper->註冊服務到註冊中心->訂閱註冊中心overrideSubscribeUrl數據;篇幅有限，本篇幅主要分析註冊服務到註冊中心的實現

RegistryProtocol.export(final Invoker<T> invoker)

public <T> Exporter<T> export(final Invoker<T> originInvoker) throws RpcException {
        //export invoker
        //導出服務
        final ExporterChangeableWrapper<T> exporter = doLocalExport(originInvoker);
        // 獲取註冊中心 URL，以 zookeeper 註冊中心爲例，得到的示例 URL 如下：
        // zookeeper://127.0.0.1:2181/com.alibaba.dubbo.registry.RegistryService?application=demo-provider&dubbo=2.0.2&export=dubbo%3A%2F%2F172.17.48.52%3A20880%2Fcom.alibaba.dubbo.demo.DemoService%3Fanyhost%3Dtrue%26application%3Ddemo-provider
        URL registryUrl = getRegistryUrl(originInvoker);

        //registry provider
        // 根據 URL 加載 Registry 實現類，比如 ZookeeperRegistry
        final Registry registry = getRegistry(originInvoker);
        // 獲取已註冊的服務提供者 URL，比如：
        // dubbo://172.17.48.52:20880/com.alibaba.dubbo.demo.DemoService?anyhost=true&application=demo-provider&dubbo=2.0.2&generic=false&interface=com.alibaba.dubbo.demo.DemoService&methods=sayHello
        final URL registeredProviderUrl = getRegisteredProviderUrl(originInvoker);

        //to judge to delay publish whether or not
        //獲取register參數;register表示是否註冊到註冊中心
        boolean register = registeredProviderUrl.getParameter("register", true);
        //緩存到ProviderConsumerRegTable的表中
        ProviderConsumerRegTable.registerProvider(originInvoker, registryUrl, registeredProviderUrl);
        //註冊服務到zookeeper
        if (register) {
            register(registryUrl, registeredProviderUrl);
            //找到該originInvoker對應的ProviderInvokerWrapper設置reg屬性爲true
            ProviderConsumerRegTable.getProviderWrapper(originInvoker).setReg(true);
        }

        // Subscribe the override data
        // FIXME When the provider subscribes, it will affect the scene : a certain JVM exposes the service and call the same service. Because the subscribed is cached key with the name of the service, it causes the subscription information to cover.
        // FIXME 提供者訂閱時，會影響同一JVM即暴露服務，又引用同一服務的的場景，因爲subscribed以服務名爲緩存的key，導致訂閱信息覆蓋
        final URL overrideSubscribeUrl = getSubscribedOverrideUrl(registeredProviderUrl);
        //創建監聽器
        final OverrideListener overrideSubscribeListener = new OverrideListener(overrideSubscribeUrl, originInvoker);
        //放入overrideSubscribeUrl對應的OverrideListener
        overrideListeners.put(overrideSubscribeUrl, overrideSubscribeListener);
        // 向註冊中心進行訂閱 override 數據
        registry.subscribe(overrideSubscribeUrl, overrideSubscribeListener);
        //Ensure that a new exporter instance is returned every time export
        //創建並返回DestroyableExporter
        return new DestroyableExporter<T>(exporter, originInvoker, overrideSubscribeUrl, registeredProviderUrl);
    }

getRegistryUrl(originInvoler)

private URL getRegistryUrl(Invoker<?> originInvoker) {
        URL registryUrl = originInvoker.getUrl();
        if (Constants.REGISTRY_PROTOCOL.equals(registryUrl.getProtocol())) {
            //zookeeper
            String protocol = registryUrl.getParameter(Constants.REGISTRY_KEY, Constants.DEFAULT_DIRECTORY);
            registryUrl = registryUrl.setProtocol(protocol).removeParameter(Constants.REGISTRY_KEY);
        }
        return registryUrl;
    }

該方法獲取註冊中心url，假如使用zookeeper作爲註冊中心，得到的示例

zookeeper://127.0.0.1:2181/com.alibaba.dubbo.registry.RegistryService?application=demo-provider&dubbo=2.0.2&export=dubbo%3A%2F%2F172.17.48.52%3A20880%2Fcom.alibaba.dubbo.demo.DemoService%3Fanyhost%3Dtrue%26application%3Ddemo-provider

getRegistry(originInvoker)

/**
     *
     * 根據調用者的地址獲取註冊表的實例
     *
     * @param originInvoker
     * @return
     */
    private Registry getRegistry(final Invoker<?> originInvoker) {
        URL registryUrl = getRegistryUrl(originInvoker);
        return registryFactory.getRegistry(registryUrl);
    }

RegistryFactory是dubbo的spi接口，由dubbo的spi機制可知，這裏的registryFactory類型爲RegistryFactory$Adaptvie代碼如下

package com.alibaba.dubbo.registry;

import com.alibaba.dubbo.common.extension.ExtensionLoader;

public class RegistryFactory$Adaptive implements com.alibaba.dubbo.registry.RegistryFactory {
    public com.alibaba.dubbo.registry.Registry getRegistry(com.alibaba.dubbo.common.URL arg0) {
        if (arg0 == null) throw new IllegalArgumentException("url == null");
        com.alibaba.dubbo.common.URL url = arg0;
        String extName = (url.getProtocol() == null ? "dubbo" : url.getProtocol());
        if (extName == null)
            throw new IllegalStateException("Fail to get extension(com.alibaba.dubbo.registry.RegistryFactory) name from url(" + url.toString() + ") use keys([protocol])");
        com.alibaba.dubbo.registry.RegistryFactory extension = (com.alibaba.dubbo.registry.RegistryFactory) ExtensionLoader.getExtensionLoader(com.alibaba.dubbo.registry.RegistryFactory.class).getExtension(extName);
        return extension.getRegistry(arg0);
    }
}

通過debug得知registryFactory類型爲ZookeeperRegistryFactory

AbstractRegistryFactory.getRegistry(URL url)

@Override
    public Registry getRegistry(URL url) {
        url = url.setPath(RegistryService.class.getName())
                .addParameter(Constants.INTERFACE_KEY, RegistryService.class.getName())
                .removeParameters(Constants.EXPORT_KEY, Constants.REFER_KEY);
        //zookeeper://127.0.0.1:2181/com.alibaba.dubbo.registry.RegistryService
        String key = url.toServiceStringWithoutResolving();
        // Lock the registry access process to ensure a single instance of the registry
        LOCK.lock();
        try {
            //訪問已經緩存的Registry
            Registry registry = REGISTRIES.get(key);
            if (registry != null) {
                return registry;
            }
            //模板方法，委託給子類創建
            registry = createRegistry(url);
            if (registry == null) {
                throw new IllegalStateException("Can not create registry " + url);
            }
            //加入緩存
            REGISTRIES.put(key, registry);
            return registry;
        } finally {
            // Release the lock
            LOCK.unlock();
        }
    }

上述方法首先從緩存REGISTRIES中取，如果取出失敗，通過子類覆蓋createRegistry(url)創建相應類型的註冊中心ZookeeperRegistry然後加入緩存中

ZookeeperRegistryFactory.createRegistry(URL url)

@Override
    public Registry createRegistry(URL url) {
        return new ZookeeperRegistry(url, zookeeperTransporter);
    }

ZookeeperRegistry.java

RegistryService:註冊中心服務接口

public interface RegistryService {

    /**
     * 註冊數據，例如：提供者服務，使用者地址，路由規則，覆蓋規則和其他數據。
     * <p>
     * 註冊需要支持如下規則<br>
     * 1. URL設置check = false參數時。註冊失敗，不會拋出異常而會在後臺重試。否則將會拋出異常<br>
     * 2. URL設置dynamic = false參數時，他需要被永久存儲，否則當註冊着異常退出，他應該被刪除<br>
     * 3. URL設置category=routers，這意味分類存儲，默認類型爲providers，數據將會被分類部分通知<br>
     * 4. 當註冊中心被重啓了，比如網絡抖動，數據不會丟失，包括自動從broken line處刪除數據<br>
     * 5. 允許具有相同URL但不同參數的URL共存，它們不能相互覆蓋。<br>
     *
     * @param url  註冊信息，不允許爲空， e.g: dubbo://10.20.153.10/com.alibaba.foo.BarService?version=1.0.0&application=kylin
     */
    void register(URL url);

    /**取消註冊
     *
     * <p>
     * 取消註冊被要求支持如下規則<br>
     * 1. 由於設置了dynamic = false存儲的屬於，當找不到註冊信息數據，將會拋出異常，其他情況將會忽略<br>
     * 2. 根據完整的網址匹配取消註冊。<br>
     *
     * @param url 註冊信息，不允許爲空， e.g: dubbo://10.20.153.10/com.alibaba.foo.BarService?version=1.0.0&application=kylin
     */
    void unregister(URL url);

    /**
     * 訂閱合適的註冊數據，當註冊過的數據修改時，自動推送
     * <p>
     * 訂閱需要遵循的規則<br>
     * 1. URL設置check = false參數時。 註冊失敗時，不會在後臺引發異常並重試該異常。<br>
     * 2. 當URL設置了 category=routers，將會通知指定類型的數據。多個分類用逗號分隔，並允許星號匹配，這表示已訂閱所有分類數據。<br>
     * 3. 允許將接口，組，版本和分類器作爲條件查詢，例如：interface = com.alibaba.foo.BarService＆version = 1.0.0<br>
     * 4. 查詢條件允許星號匹配，訂閱所有接口的所有數據包的所有版本，例如 ：interface = *＆group = *＆version = *＆classifier = *<br>
     * 5. 當註冊表重新啓動並且出現網絡抖動時，有必要自動恢復訂閱請求。<br>
     * 6. 允許具有相同URL但不同參數的URL共存，它們不能相互覆蓋。<br>
     * 7. 當第一個通知完成並且返回後，訂閱程序必須被阻塞<br>
     *
     * @param url      訂閱條件，不允許爲空， e.g. consumer://10.20.153.10/com.alibaba.foo.BarService?version=1.0.0&application=kylin
     * @param listener 事件變化的監聽器，不允許爲空
     */
    void subscribe(URL url, NotifyListener listener);

    /**
     * 取消訂閱
     * <p>
     * 取消訂閱要遵循的規則<br>
     * 1. 如果沒有訂閱，則直接忽略它。<br>
     * 2. 取消訂閱完整的URL匹配。<br>
     *
     * @param url      Subscription condition, not allowed to be empty, e.g. consumer://10.20.153.10/com.alibaba.foo.BarService?version=1.0.0&application=kylin
     * @param listener A listener of the change event, not allowed to be empty
     */
    void unsubscribe(URL url, NotifyListener listener);

    /**
     * Query the registered data that matches the conditions. Corresponding to the push mode of the subscription, this is the pull mode and returns only one result.
     *
     * 查找匹配條件的註冊過的數據.對於訂閱的推送模式，這是請求模式將會返回一個結果
     * @param url Query condition, is not allowed to be empty, e.g. consumer://10.20.153.10/com.alibaba.foo.BarService?version=1.0.0&application=kylin
     * @return  註冊信息列表，可以爲空，含義與參數相同{@link com.alibaba.dubbo.registry.NotifyListener#notify(List<URL>)}.
     * @see com.alibaba.dubbo.registry.NotifyListener#notify(List)
     */
    List<URL> lookup(URL url);

}

Registry:繼承了Node和RegistryService的接口，實現該接口的類的應該是註冊中心

AbstractRegistry:用來實現服務與訂閱url的緩存文件的創建和生成。

• 成員變量

// URL address separator, used in file cache, service provider URL separation
    //URL地址分隔符，用於文件緩存，服務提供商URL分隔
    private static final char URL_SEPARATOR = ' ';
    // URL address separated regular expression for parsing the service provider URL list in the file cache
    private static final String URL_SPLIT = "\\s+";
    // Log output
    protected final Logger logger = LoggerFactory.getLogger(getClass());
    // Local disk cache, where the special key value.registies records the list of registry centers, and the others are the list of notified service providers
    //本地磁盤緩存，其中特殊鍵value.registies記錄註冊表中心列表，其他是已通知服務提供者的列表
    private final Properties properties = new Properties();
    // 文件緩存定時寫入
    private final ExecutorService registryCacheExecutor = Executors.newFixedThreadPool(1, new NamedThreadFactory("DubboSaveRegistryCache", true));
    // 是否同步保存文件
    private final boolean syncSaveFile;
    // 上次文件緩存變更版本
    private final AtomicLong lastCacheChanged = new AtomicLong();
    // 已註冊服務URL集合
    private final Set<URL> registered = new ConcurrentHashSet<URL>();
    //已經訂閱的<URL, Set<NotifyListener>>
    private final ConcurrentMap<URL, Set<NotifyListener>> subscribed = new ConcurrentHashMap<URL, Set<NotifyListener>>();
    //已經通知的<URL, Map<String, List<URL>>>
    private final ConcurrentMap<URL, Map<String, List<URL>>> notified = new ConcurrentHashMap<URL, Map<String, List<URL>>>();
    //zookeeper://127.0.0.1:2181/com.alibaba.dubbo.registry.RegistryService?application=demo-provider&
    // client=curator&dubbo=2.0.0&interface=com.alibaba.dubbo.registry.RegistryService&pid=4685&timestamp=1507286468150
    private URL registryUrl;
    // 本地磁盤緩存文件
    private File file;

• 構造函數

public AbstractRegistry(URL url) {
        setUrl(url);
        /**
         * 獲取URL對象的save.file屬性默認爲false代表不異步保存文件
         */
        syncSaveFile = url.getParameter(Constants.REGISTRY_FILESAVE_SYNC_KEY, false);
        /**
         * 獲取URL對象的file屬性，如果沒有則dubbo幫我們指定默認的文件配置：像這樣子
         * C:\Users\Administrator\.dubbo\dubbo-registry-echo-provider-192.168.1.233:2181.cache
         */
        String filename = url.getParameter(Constants.FILE_KEY, System.getProperty("user.home") + "/.dubbo/dubbo-registry-" + url.getParameter(Constants.APPLICATION_KEY) + "-" + url.getAddress() + ".cache");
        File file = null;
        if (ConfigUtils.isNotEmpty(filename)) {
            file = new File(filename);
            if (!file.exists() && file.getParentFile() != null && !file.getParentFile().exists()) {
                if (!file.getParentFile().mkdirs()) {
                    throw new IllegalArgumentException("Invalid registry store file " + file + ", cause: Failed to create directory " + file.getParentFile() + "!");
                }
            }
        }
        this.file = file;
        /**
         * 加載文件C:\Users\Administrator\.dubbo\dubbo-registry-echo-provider-192.168.1.233:2181.cache內容保存類似下面這樣子的
         *  com.alibaba.dubbo.demo.DemoService=empty\://10.10.10.10\:20880/com.alibaba.dubbo.demo.DemoService?anyhost\=true&application\=demo-provider&category
         *  \=configurators&check\=false&dubbo\=2.0.0&generic\=false&interface\=com.alibaba.dubbo.demo.DemoService&methods\=sayHello&pid\=5259&side\=provider&timestamp\=1507294508053
         * 到成員變Properties properties = new Properties()
         */
        loadProperties();
       /**
         * zookeeper://127.0.0.1:2181/com.alibaba.dubbo.registry.RegistryService?application=echo-provider&backup=10.20.153.11:2181,10.20.153.12:2181&dubbo=2.0.2&interface=com.alibaba.dubbo.registry.RegistryService&pid=8640&timestamp=1572932516092
         * 通過調用getBackUpUrls最終變成了
         * zookeeper://127.0.0.1:2181/com.alibaba.dubbo.registry.RegistryService?application=echo-provider&backup=10.20.153.11:2181,10.20.153.12:2181&dubbo=2.0.2&interface=com.alibaba.dubbo.registry.RegistryService&pid=8640&timestamp=1572932516092
         * zookeeper://10.20.153.11:2181/com.alibaba.dubbo.registry.RegistryService?application=echo-provider&backup=10.20.153.11:2181,10.20.153.12:2181&dubbo=2.0.2&interface=com.alibaba.dubbo.registry.RegistryService&pid=8640&timestamp=1572932516092
         * zookeeper://10.20.153.12:2181/com.alibaba.dubbo.registry.RegistryService?application=echo-provider&backup=10.20.153.11:2181,10.20.153.12:2181&dubbo=2.0.2&interface=com.alibaba.dubbo.registry.RegistryService&pid=8640&timestamp=1572932516092
         *通知監聽器，URL 變化結果
         */
        notify(url.getBackupUrls());
    }

​
protected void notify(List<URL> urls) {
        if (urls == null || urls.isEmpty()) return;
        //遍歷URL對應的所有NotifyListener
        for (Map.Entry<URL, Set<NotifyListener>> entry : getSubscribed().entrySet()) {
            URL url = entry.getKey();
            //如果urls中的任意一個url與當subscribed的key對應的url匹配
            if (!UrlUtils.isMatch(url, urls.get(0))) {
                continue;
            }
            //通知
            Set<NotifyListener> listeners = entry.getValue();
            if (listeners != null) {
                for (NotifyListener listener : listeners) {
                    try {
                        notify(url, listener, filterEmpty(url, urls));
                    } catch (Throwable t) {
                        logger.error("Failed to notify registry event, urls: " + urls + ", cause: " + t.getMessage(), t);
                    }
                }
            }
        }
    }

​

FailbackRegistry：通過任務調度線程池用來做失敗重試操作（包括：註冊失敗／取消註冊失敗／訂閱失敗／取消訂閱失敗／通知失敗）的重試

• 成員變量

 // 定時調度線程池，用於對註冊失敗、訂閱失敗的重試
    private final ScheduledExecutorService retryExecutor = Executors.newScheduledThreadPool(1, new NamedThreadFactory("DubboRegistryFailedRetryTimer", true));

    // Timer for failure retry, regular check if there is a request for failure, and if there is, an unlimited retry
    // 重試失敗計時器，定期檢查是否有失敗請求，是否有無限次重試，用於取消重試的調度任務
    private final ScheduledFuture<?> retryFuture;
    // 註冊失敗的Set<URL>
    private final Set<URL> failedRegistered = new ConcurrentHashSet<URL>();
    // 取消註冊失敗的Set<URL>
    private final Set<URL> failedUnregistered = new ConcurrentHashSet<URL>();
    //訂閱失敗的url
    private final ConcurrentMap<URL, Set<NotifyListener>> failedSubscribed = new ConcurrentHashMap<URL, Set<NotifyListener>>();
    //取消訂閱失敗的url
    private final ConcurrentMap<URL, Set<NotifyListener>> failedUnsubscribed = new ConcurrentHashMap<URL, Set<NotifyListener>>();
    //通知失敗的url
    private final ConcurrentMap<URL, Map<NotifyListener, List<URL>>> failedNotified = new ConcurrentHashMap<URL, Map<NotifyListener, List<URL>>>();

• 構造函數：

public FailbackRegistry(URL url) {
        //調用父類的構造函數
        super(url);
        //從URL對象中獲取屬性retry.period 如果沒指定默認爲5000毫秒
        this.retryPeriod = url.getParameter(Constants.REGISTRY_RETRY_PERIOD_KEY, Constants.DEFAULT_REGISTRY_RETRY_PERIOD);
        //使用retryExecutor定時調度retry()方法
        //retry()方法主要是
        this.retryFuture = retryExecutor.scheduleWithFixedDelay(new Runnable() {
            @Override
            public void run() {
                // Check and connect to the registry
                try {
                    /**
                     * 註冊失敗的重新註冊url
                     * 取消註冊失敗的url重新取消註冊
                     * 訂閱失敗的url重新訂閱
                     * 取消訂閱失敗的url重新取消訂閱
                     */
                    retry();
                } catch (Throwable t) { // Defensive fault tolerance
                    logger.error("Unexpected error occur at failed retry, cause: " + t.getMessage(), t);
                }
            }
        }, retryPeriod, retryPeriod, TimeUnit.MILLISECONDS);
    }

// Retry the failed actions
    protected void retry() {
        if (!failedRegistered.isEmpty()) {
            Set<URL> failed = new HashSet<URL>(failedRegistered);
            if (failed.size() > 0) {
                if (logger.isInfoEnabled()) {
                    logger.info("Retry register " + failed);
                }
                try {
                    for (URL url : failed) {
                        try {
                            doRegister(url);
                            failedRegistered.remove(url);
                        } catch (Throwable t) { // Ignore all the exceptions and wait for the next retry
                            logger.warn("Failed to retry register " + failed + ", waiting for again, cause: " + t.getMessage(), t);
                        }
                    }
                } catch (Throwable t) { // Ignore all the exceptions and wait for the next retry
                    logger.warn("Failed to retry register " + failed + ", waiting for again, cause: " + t.getMessage(), t);
                }
            }
        }
        if (!failedUnregistered.isEmpty()) {
            Set<URL> failed = new HashSet<URL>(failedUnregistered);
            if (!failed.isEmpty()) {
                if (logger.isInfoEnabled()) {
                    logger.info("Retry unregister " + failed);
                }
                try {
                    for (URL url : failed) {
                        try {
                            doUnregister(url);
                            failedUnregistered.remove(url);
                        } catch (Throwable t) { // Ignore all the exceptions and wait for the next retry
                            logger.warn("Failed to retry unregister  " + failed + ", waiting for again, cause: " + t.getMessage(), t);
                        }
                    }
                } catch (Throwable t) { // Ignore all the exceptions and wait for the next retry
                    logger.warn("Failed to retry unregister  " + failed + ", waiting for again, cause: " + t.getMessage(), t);
                }
            }
        }
        if (!failedSubscribed.isEmpty()) {
            Map<URL, Set<NotifyListener>> failed = new HashMap<URL, Set<NotifyListener>>(failedSubscribed);
            for (Map.Entry<URL, Set<NotifyListener>> entry : new HashMap<URL, Set<NotifyListener>>(failed).entrySet()) {
                if (entry.getValue() == null || entry.getValue().size() == 0) {
                    failed.remove(entry.getKey());
                }
            }
            if (failed.size() > 0) {
                if (logger.isInfoEnabled()) {
                    logger.info("Retry subscribe " + failed);
                }
                try {
                    for (Map.Entry<URL, Set<NotifyListener>> entry : failed.entrySet()) {
                        URL url = entry.getKey();
                        Set<NotifyListener> listeners = entry.getValue();
                        for (NotifyListener listener : listeners) {
                            try {
                                doSubscribe(url, listener);
                                listeners.remove(listener);
                            } catch (Throwable t) { // Ignore all the exceptions and wait for the next retry
                                logger.warn("Failed to retry subscribe " + failed + ", waiting for again, cause: " + t.getMessage(), t);
                            }
                        }
                    }
                } catch (Throwable t) { // Ignore all the exceptions and wait for the next retry
                    logger.warn("Failed to retry subscribe " + failed + ", waiting for again, cause: " + t.getMessage(), t);
                }
            }
        }
        if (!failedUnsubscribed.isEmpty()) {
            Map<URL, Set<NotifyListener>> failed = new HashMap<URL, Set<NotifyListener>>(failedUnsubscribed);
            for (Map.Entry<URL, Set<NotifyListener>> entry : new HashMap<URL, Set<NotifyListener>>(failed).entrySet()) {
                if (entry.getValue() == null || entry.getValue().isEmpty()) {
                    failed.remove(entry.getKey());
                }
            }
            if (failed.size() > 0) {
                if (logger.isInfoEnabled()) {
                    logger.info("Retry unsubscribe " + failed);
                }
                try {
                    for (Map.Entry<URL, Set<NotifyListener>> entry : failed.entrySet()) {
                        URL url = entry.getKey();
                        Set<NotifyListener> listeners = entry.getValue();
                        for (NotifyListener listener : listeners) {
                            try {
                                doUnsubscribe(url, listener);
                                listeners.remove(listener);
                            } catch (Throwable t) { // Ignore all the exceptions and wait for the next retry
                                logger.warn("Failed to retry unsubscribe " + failed + ", waiting for again, cause: " + t.getMessage(), t);
                            }
                        }
                    }
                } catch (Throwable t) { // Ignore all the exceptions and wait for the next retry
                    logger.warn("Failed to retry unsubscribe " + failed + ", waiting for again, cause: " + t.getMessage(), t);
                }
            }
        }
        if (!failedNotified.isEmpty()) {
            Map<URL, Map<NotifyListener, List<URL>>> failed = new HashMap<URL, Map<NotifyListener, List<URL>>>(failedNotified);
            for (Map.Entry<URL, Map<NotifyListener, List<URL>>> entry : new HashMap<URL, Map<NotifyListener, List<URL>>>(failed).entrySet()) {
                if (entry.getValue() == null || entry.getValue().size() == 0) {
                    failed.remove(entry.getKey());
                }
            }
            if (failed.size() > 0) {
                if (logger.isInfoEnabled()) {
                    logger.info("Retry notify " + failed);
                }
                try {
                    for (Map<NotifyListener, List<URL>> values : failed.values()) {
                        for (Map.Entry<NotifyListener, List<URL>> entry : values.entrySet()) {
                            try {
                                NotifyListener listener = entry.getKey();
                                List<URL> urls = entry.getValue();
                                listener.notify(urls);
                                values.remove(listener);
                            } catch (Throwable t) { // Ignore all the exceptions and wait for the next retry
                                logger.warn("Failed to retry notify " + failed + ", waiting for again, cause: " + t.getMessage(), t);
                            }
                        }
                    }
                } catch (Throwable t) { // Ignore all the exceptions and wait for the next retry
                    logger.warn("Failed to retry notify " + failed + ", waiting for again, cause: " + t.getMessage(), t);
                }
            }
        }
    }

ZookeeperRegistry:Zookeeper實現dubbo的註冊中心

• 成員變量

 /**
     * 默認zookeeper的根節點
     */
    private final static String DEFAULT_ROOT = "dubbo";
    /**
     * zookeeper的根節點
     */
    private final String root;
    /**
     * 服務接口集合
     */
    private final Set<String> anyServices = new ConcurrentHashSet<String>();
    /**
     * 監聽器
     */
    private final ConcurrentMap<URL, ConcurrentMap<NotifyListener, ChildListener>> zkListeners = new ConcurrentHashMap<URL, ConcurrentMap<NotifyListener, ChildListener>>();
    /**
     * 操作zookeeper的客戶端實例
     */
    private final ZookeeperClient zkClient;

• 構造函數

public ZookeeperRegistry(URL url, ZookeeperTransporter zookeeperTransporter) {
        super(url);
        if (url.isAnyHost()) {
            throw new IllegalStateException("registry address == null");
        }
        //獲取url中的group屬性，不存在使用dubbo作爲默認分組
        String group = url.getParameter(Constants.GROUP_KEY, DEFAULT_ROOT);
        //如果不以"/"開頭，添加文件分隔符
        if (!group.startsWith(Constants.PATH_SEPARATOR)) {
            group = Constants.PATH_SEPARATOR + group;
        }
        //zookeeper的根節點爲group
        this.root = group;
        //基於dubbo的spi機制會根據url中攜帶的參數去選擇用哪個實現類。
        //目前提供了ZkclientZookeeperTransporter這時候的zkClient爲ZkclientZookeeperClient
        //CuratorZookeeperTransporter這時候的zkClient爲CuratorZookeeperClient
        zkClient = zookeeperTransporter.connect(url);
        //添加狀態監聽器
        zkClient.addStateListener(new StateListener() {
            @Override
            public void stateChanged(int state) {
                //重連後，調動recover方法
                if (state == RECONNECTED) {
                    try {
                        recover();
                    } catch (Exception e) {
                        logger.error(e.getMessage(), e);
                    }
                }
            }
        });
    }

調用父類構造函數，構造函數通過讀取url中的group參數初始化zookeeper的根節點；通過zookeeperTransporter獲取ZookeeperClient，ZookeeperTransporter爲dubbo的spi接口，根據dubbo的spi實例化的過程這裏的zookeeperTransporter類型爲ZookeeperTransporter$Adaptive其代碼如下

package com.alibaba.dubbo.remoting.zookeeper;

import com.alibaba.dubbo.common.extension.ExtensionLoader;

public class ZookeeperTransporter$Adaptive implements com.alibaba.dubbo.remoting.zookeeper.ZookeeperTransporter {
    public com.alibaba.dubbo.remoting.zookeeper.ZookeeperClient connect(com.alibaba.dubbo.common.URL arg0) {
        if (arg0 == null) throw new IllegalArgumentException("url == null");
        com.alibaba.dubbo.common.URL url = arg0;
        String extName = url.getParameter("client", url.getParameter("transporter", "curator"));
        if (extName == null)
            throw new IllegalStateException("Fail to get extension(com.alibaba.dubbo.remoting.zookeeper.ZookeeperTransporter) name from url(" + url.toString() + ") use keys([client, transporter])");
        com.alibaba.dubbo.remoting.zookeeper.ZookeeperTransporter extension = (com.alibaba.dubbo.remoting.zookeeper.ZookeeperTransporter) ExtensionLoader.getExtensionLoader(com.alibaba.dubbo.remoting.zookeeper.ZookeeperTransporter.class).getExtension(extName);
        return extension.connect(arg0);
    }
}

所以不難得知dubbo默認情況下使用的zookeeper的客戶端類型爲CuratorZookeeperTransporter，代碼如下

public class CuratorZookeeperTransporter implements ZookeeperTransporter {

    @Override
    public ZookeeperClient connect(URL url) {
        return new CuratorZookeeperClient(url);
    }

}

最後給zkClient添加StateListener;該StateListener監聽到zk客戶端的重連事件調用recover()方法,添加到failedRegistered和failedSubscribed通過FailbackRegistry定時任務重新註冊、重新訂閱，代碼如下。

@Override
    protected void recover() throws Exception {
        // 已經註冊的url添加到failedRegistered，通過FailbackRegistry的定時任務
        //進行重新註冊
        Set<URL> recoverRegistered = new HashSet<URL>(getRegistered());
        if (!recoverRegistered.isEmpty()) {
            if (logger.isInfoEnabled()) {
                logger.info("Recover register url " + recoverRegistered);
            }
            for (URL url : recoverRegistered) {
                failedRegistered.add(url);
            }
        }
        // 已經訂閱的URL添加到failedSubscribed,通過FailbackRegistry的定時任務
        // 進行重新訂閱
        Map<URL, Set<NotifyListener>> recoverSubscribed = new HashMap<URL, Set<NotifyListener>>(getSubscribed());
        if (!recoverSubscribed.isEmpty()) {
            if (logger.isInfoEnabled()) {
                logger.info("Recover subscribe url " + recoverSubscribed.keySet());
            }
            for (Map.Entry<URL, Set<NotifyListener>> entry : recoverSubscribed.entrySet()) {
                URL url = entry.getKey();
                for (NotifyListener listener : entry.getValue()) {
                    addFailedSubscribed(url, listener);
                }
            }
        }
    }

getRegisteredProviderUrl(originInvoker)

private URL getRegisteredProviderUrl(final Invoker<?> originInvoker) {
        URL providerUrl = getProviderUrl(originInvoker);
        //The address you see at the registry
        return providerUrl.removeParameters(getFilteredKeys(providerUrl))
                .removeParameter(Constants.MONITOR_KEY)
                .removeParameter(Constants.BIND_IP_KEY)
                .removeParameter(Constants.BIND_PORT_KEY)
                .removeParameter(QOS_ENABLE)
                .removeParameter(QOS_PORT)
                .removeParameter(ACCEPT_FOREIGN_IP)
                .removeParameter(VALIDATION_KEY);
    }

該方法返回的key類似

dubbo://169.254.22.149:20880/com.alibaba.dubbo.study.day01.xml.service.EchoService?addListener.1.callback=true&addListener.retries=2&anyhost=true&application=echo-provider&bean.name=com.alibaba.dubbo.study.day01.xml.service.EchoService&dubbo=2.0.2&generic=false&interface=com.alibaba.dubbo.study.day01.xml.service.EchoService&methods=echo,addListener&pid=6688&side=provider&timestamp=1572936570702

ProviderConsumerRegTable.registerProvider(originInvoker, registryUrl, registeredProviderUrl);

public static void registerProvider(Invoker invoker, URL registryUrl, URL providerUrl) {
        ProviderInvokerWrapper wrapperInvoker = new ProviderInvokerWrapper(invoker, registryUrl, providerUrl);
        // group/interface:version
        // 比如group2/com.alibaba.dubbo.study.day01.xml.service.EchoService:1.0.0
        String serviceUniqueName = providerUrl.getServiceKey();
        // 獲取group/interface:version對應的ProviderInvokerWrapper列表
        Set<ProviderInvokerWrapper> invokers = providerInvokers.get(serviceUniqueName);
        //不存在是,緩存進去謝謝
        if (invokers == null) {
            providerInvokers.putIfAbsent(serviceUniqueName, new ConcurrentHashSet<ProviderInvokerWrapper>());
            invokers = providerInvokers.get(serviceUniqueName);
        }

        invokers.add(wrapperInvoker);
    }

將registryUrl,providerUrl,invoker封裝成ProvidereInvokerWrapper對象，然後根據providerUrl生成一個服務的名稱保存到內存中，也就是說我們可以通過ProviderConsumerRegTable類拿到已經註冊過的服務的相關信息！！！

register(registryUrl, registeredProviderUrl);

public void register(URL registryUrl, URL registedProviderUrl) {
        Registry registry = registryFactory.getRegistry(registryUrl);
        registry.register(registedProviderUrl);
    }

獲取註冊中心，交給註冊中心註冊服務(由上文分析可知此時registry爲ZookeeperRegistry)

FailbackRegistry.registry(URL url)

@Override
    public void register(URL url) {
        //父類方法實現將該url添加到registered集合中
        super.register(url);
        //從failedRegistered和failedUnregistered集合一處url
        failedRegistered.remove(url);
        failedUnregistered.remove(url);
        try {
            // Sending a registration request to the server side
            // 委託給子類實現
            doRegister(url);
        } catch (Exception e) {
            Throwable t = e;

            // If the startup detection is opened, the Exception is thrown directly.
            // 如果啓動檢測已打開，則直接引發Exception。如果check是true直接拋出異常
            boolean check = getUrl().getParameter(Constants.CHECK_KEY, true)
                    && url.getParameter(Constants.CHECK_KEY, true)
                    && !Constants.CONSUMER_PROTOCOL.equals(url.getProtocol());
            boolean skipFailback = t instanceof SkipFailbackWrapperException;
            if (check || skipFailback) {
                if (skipFailback) {
                    t = t.getCause();
                }
                throw new IllegalStateException("Failed to register " + url + " to registry " + getUrl().getAddress() + ", cause: " + t.getMessage(), t);
            } else {
                logger.error("Failed to register " + url + ", waiting for retry, cause: " + t.getMessage(), t);
            }

            // Record a failed registration request to a failed list, retry regularly
            //記錄註冊失敗的url
            failedRegistered.add(url);
        }
    }

ZookeeperRegistry.doRegistry(URL url)

@Override
    protected void doRegister(URL url) {
        try {
            //創建URL節點
            //通過 Zookeeper 客戶端創建節點，節點路徑由 toUrlPath 方法生成，路徑格式如下:
            ///${group}/${serviceInterface}/providers/${url}
            zkClient.create(toUrlPath(url), url.getParameter(Constants.DYNAMIC_KEY, true));
        } catch (Throwable e) {
            throw new RpcException("Failed to register " + url + " to zookeeper " + getUrl() + ", cause: " + e.getMessage(), e);
        }
    }

委託給zkClient創建zk的目錄

AbstractZookeeperClient.create(String path, boolean ephemeral)

@Override
    public void create(String path, boolean ephemeral) {
        if (!ephemeral) {
            // 如果要創建的節點類型非臨時節點，那麼這裏要檢測節點是否存在
            if (checkExists(path)) {
                return;
            }
        }
        int i = path.lastIndexOf('/');
        if (i > 0) {
            //遞歸創建父節點
            create(path.substring(0, i), false);
        }
        if (ephemeral) {
            //創建臨時節點
            createEphemeral(path);
        } else {
            //創建持久節點
            createPersistent(path);
        }
    }

經過該方法會在在zookeeper上創建了一個類似文件夾目錄結構的樹：

從上圖中可以看到EchoService 這個服務對應的配置信息（存儲在 URL 中）最終被註冊到了 /dubbo/com.alibaba.dubbo.study.day01.xml.service.EchoService/providers/ 節點下。搞懂了服務註冊的本質，那麼接下來我們就可以去閱讀服務註冊的代碼了。

Dubbo屏蔽了curator-framework、zkclient實現zookeeper客戶端對zookeeper操作細節的不同

• ZookeeperClient.java：統一定義dubbo對zk需要的操作接口

/**
 * dubbo對zkClient的操作封裝的接口
 */
public interface ZookeeperClient {

    void create(String path, boolean ephemeral);

    void delete(String path);

    List<String> getChildren(String path);

    List<String> addChildListener(String path, ChildListener listener);

    void removeChildListener(String path, ChildListener listener);

    void addStateListener(StateListener listener);

    void removeStateListener(StateListener listener);

    boolean isConnected();

    void close();

    URL getUrl();

}

• AbstractZookeeperClient:zk狀態監聽器的維護，創建path方法的遞歸實現，最終還會委託給子類

 /**
     * 服務註冊中心的url
     *zookeeper://127.0.0.1:2181/com.alibaba.dubbo.registry.RegistryService?application=echo-provider&
     * backup=10.20.153.11:2181,10.20.153.12:2181&dubbo=2.0.2&interface=com.alibaba.dubbo.registry.RegistryService&pid=17464&timestamp=1573182003753
     */
    private final URL url;
    /**
     * zookeeper狀態監聽器
     */
    private final Set<StateListener> stateListeners = new CopyOnWriteArraySet<StateListener>();
    /**
     * path路徑監聽器
     */
    private final ConcurrentMap<String, ConcurrentMap<ChildListener, TargetChildListener>> childListeners = new ConcurrentHashMap<String, ConcurrentMap<ChildListener, TargetChildListener>>();
    
    private volatile boolean closed = false;

• CuratorZookeeperClient：通過組合org.apache.curator.framework.CuratorFramework實現對zookeeper的操作

private final CuratorFramework client;

public CuratorZookeeperClient(URL url) {
        super(url);
        try {
            int timeout = url.getParameter(Constants.TIMEOUT_KEY, 5000);
            // 創建 CuratorFramework 構造器
            CuratorFrameworkFactory.Builder builder = CuratorFrameworkFactory.builder()
                    .connectString(url.getBackupAddress())
                    .retryPolicy(new RetryNTimes(1, 1000))
                    .connectionTimeoutMs(timeout);
            //zookeeper的權限新
            String authority = url.getAuthority();
            if (authority != null && authority.length() > 0) {
                builder = builder.authorization("digest", authority.getBytes());
            }
            // 構建 CuratorFramework 實例
            client = builder.build();
            // 添加client監聽器
            client.getConnectionStateListenable().addListener(new ConnectionStateListener() {
                @Override
                public void stateChanged(CuratorFramework client, ConnectionState state) {
                    if (state == ConnectionState.LOST) {
                        CuratorZookeeperClient.this.stateChanged(StateListener.DISCONNECTED);
                    } else if (state == ConnectionState.CONNECTED) {
                        CuratorZookeeperClient.this.stateChanged(StateListener.CONNECTED);
                    } else if (state == ConnectionState.RECONNECTED) {
                        CuratorZookeeperClient.this.stateChanged(StateListener.RECONNECTED);
                    }
                }
            });
            client.start();
        } catch (Exception e) {
            throw new IllegalStateException(e.getMessage(), e);
        }
    }

• ZkClientWrapper：Zkclient包裝器類，可以在連接超時後自動監視連接狀態,爲了保持與curator的使用一致對ZkClient做的封裝

/**
     * 超時時間
     */
    private long timeout;
    /**
     * 第三方包的zk操作客戶端
     */
    private ZkClient client;
    /**
     * 連接狀態
     */
    private volatile KeeperState state;
    private ListenableFutureTask<ZkClient> listenableFutureTask;
    private volatile boolean started = false;

• ZkclientZookeeperClient：通過組合ZkClientWrapper的client實現對zookeeper的操作

private final ZkClientWrapper client;

private volatile KeeperState state = KeeperState.SyncConnected;

 public ZkclientZookeeperClient(URL url) {
        super(url);
        long timeout = url.getParameter(Constants.TIMEOUT_KEY, 30000L);
        client = new ZkClientWrapper(url.getBackupAddress(), timeout);
        client.addListener(new IZkStateListener() {
            @Override
            public void handleStateChanged(KeeperState state) throws Exception {
                ZkclientZookeeperClient.this.state = state;
                if (state == KeeperState.Disconnected) {
                    stateChanged(StateListener.DISCONNECTED);
                } else if (state == KeeperState.SyncConnected) {
                    stateChanged(StateListener.CONNECTED);
                }
            }

            @Override
            public void handleNewSession() throws Exception {
                stateChanged(StateListener.RECONNECTED);
            }
        });
        client.start();
    }

Zookeeper註冊中心實現原理參考《深入理解apache dubbo與實戰》一書

Dubbo使用ZK作爲註冊中心時，只會創建臨時節點和持久節點兩種，對創建順序並沒有要求。

/dubbo/com.foo.BarService/providers是服務提供者在Zookeeper註冊中心的路徑示例，是一種屬性結構，該結構分爲四層：root（根節點，對應示例總的dubbo）、service（接口名稱，對應示例中的com.foo.BarService）、四種服務目錄（對應示例中的prividers，其他目錄還有consumers、routers、configurators）。在服務分類節點下是具體的Dubbo服務URL。屬性結構示例如下：

+ /dubbo
+-- service
            +-- providers
            +-- consumers
            +-- routers
            +--configurators

樹形結構的關係：

• a.樹的根節點是註冊中心分組，下面有多個服務接口，分組的值來自用戶<dubbo:registry配置的group屬性，默認爲/dubbo
• b.服務接口下包含4類子目錄，分別是providers,consumers,routers,configurators,這些路徑是持久節點
• c.服務提供者目錄(/dubbo/service/providers)下面包含的接口有多個服務者URL元數據信息。
• d.服務消費者目錄(/dubbo/service/consumers)下面包含的接口有多個消費者URL元數據信息。
• e.路由配置目錄(/dubbo/service/routers)下面包含多用用於消費者路由策略URL元數據信息。
• f.動態配置目錄(/dubbo/service/configurators)下面包含多個用於服務者動態配置URL元數據信息。

在dubbo框架啓動時，會根據用戶配置的服務，在註冊中心創建4個目錄，在providers和consumers目錄中分別存儲服務提供方、消費方元數據信息。

在dubbo框架進行調用時，用戶可以通過服務治理平臺(dubbo-admin)下發配路由配置，如果要改變服務參數，用戶可以通過(dubbo-admin)下發動態配置。服務端會通過訂閱收到屬性變更，並重新暴露服務！！！

目錄名稱	存儲值示例
/dubbo/service/providers	dubbo://127.0.0.1:20880/com.xxx.xxService?key=value&...
/dubbo/service/consumers	dubbo://127.0.0.1:5002/com.xxx.xxService?key=value&...
/dubbo/service/routers	condition://0.0.0/com.xxx.xxxService?category=router&key...
/dubbo/service/configurators	override://0.0.0.0/com.xxx.xxxService?category=configurators&key