Java - zookeeper 服務註冊發現

網上實際一些的案例嚴重缺乏,真心不易出此稿~~

1、zookeeper 服務註冊發現模型

流程:1)註冊,2)發現:監聽、負載均衡、故障檢測、變更通知,3)調用。


From http://blog.cloudera.com/blog/2014/03/zookeeper-resilience-at-pinterest/



From http://www.techweb.com.cn/network/hardware/2015-12-25/2246973.shtml



From http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_hadoop-ha/content/ha-hs2-service-discovery.html


2、單機代碼
虛擬機準備:1)啓動zookeeper,2)啓動多個服務(如ElasticSearch, Tomcat之類的),3)一個服務(如tomcat、elasticsearch)對應多個instance(如節點1、節點2)

流程:

1)準備:CuratorFramework(可選擇getClient或newClient方法),ServiceInstance;
2)註冊:ServiceDiscovery (註冊有兩種方式:registerService/thisInstance);
3.1)負載均衡:ServiceProvider providerStrategy()
3.2)啓動監聽:ServiceProvider start(),一個provider對應一個服務;
4)通知:ServiceProvider getInstance()
5)調用: 有ip、port,想怎麼用就怎麼用吧~

import java.io.Closeable;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.UUID;

import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.apache.curator.retry.RetryNTimes;
import org.apache.curator.utils.CloseableUtils;
import org.apache.curator.x.discovery.ServiceDiscovery;
import org.apache.curator.x.discovery.ServiceDiscoveryBuilder;
import org.apache.curator.x.discovery.ServiceInstance;
import org.apache.curator.x.discovery.ServiceProvider;
import org.apache.curator.x.discovery.UriSpec;
import org.apache.curator.x.discovery.details.JsonInstanceSerializer;
import org.apache.curator.x.discovery.strategies.RandomStrategy;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;

import com.google.common.collect.Lists;
import com.google.common.collect.Maps;

/*
 * http://www.techweb.com.cn/network/hardware/2015-12-25/2246973.shtml
 * http://blog.csdn.net/hxpjava1/article/details/8612228
 * http://www.tuicool.com/articles/F3Avue
 * http://www.bkjia.com/Javabc/879495.html
 * http://curator.apache.org/apidocs/index.html
 * http://blog.csdn.net/john_f_lau/article/details/50660195  *
 * http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.curator%22
 * share one CuratorFramework per ZooKeeper cluster
 * By default ServiceProvider uses Round-robin ProviderStrategy
 */
public class ZkTest01 {
	
	private static String basePath = "/es_path";
	private static List<Closeable> closeableList = Lists.newArrayList();
	
	// Register worker 1
	private static ServiceDiscovery<Void> registerInZookeeper(CuratorFramework client, 
			ServiceInstance<Void> instance) throws Exception {
		ServiceDiscovery<Void> serviceDiscovery;

		JsonInstanceSerializer<Void> serializer = new JsonInstanceSerializer<Void>(
				Void.class);
		//(3)
		serviceDiscovery = ServiceDiscoveryBuilder
				.builder(Void.class)
				.client(client)
				.serializer(serializer)
				.basePath(basePath)
				//.thisInstance(instance)  //註冊方式2
				.build();
		serviceDiscovery.start();
		//(4) cd <ZK_ROOT>/bin, ./zkCli.sh
		return serviceDiscovery;
	}
	
	// Register worker 2
		private static ServiceDiscovery<Void> registerInZookeeper2(CuratorFramework client, 
				ServiceInstance<Void> instance) throws Exception {
			ServiceDiscovery<Void> serviceDiscovery;

			JsonInstanceSerializer<Void> serializer = new JsonInstanceSerializer<Void>(
					Void.class);
			//(3)
			serviceDiscovery = ServiceDiscoveryBuilder
					.builder(Void.class)
					.client(client)
					.serializer(serializer)
					.basePath(basePath)
					//.thisInstance(instance)  //註冊方式2
					.build();
			serviceDiscovery.start();
			//(4) cd <ZK_ROOT>/bin, ./zkCli.sh
			return serviceDiscovery;
		}
	
	public static ServiceInstance<Void> getInstanceByName(ServiceDiscovery<Void> serviceDiscovery, 
			String serviceName) throws Exception {

		//(5) 非註冊方式,do not register in ZooKeeper, get instance from ServiceProvider
		ServiceProvider<Void> provider = serviceDiscovery.serviceProviderBuilder().
                serviceName(serviceName).
                providerStrategy(new RandomStrategy<Void>())
                .build();
        provider.start();
        closeableList.add(provider);
        return provider.getInstance();
    }
	
	public static synchronized void close(){
       for (Closeable closeable : closeableList) {
           CloseableUtils.closeQuietly(closeable);
       }
	}
	
	public static CuratorFramework getClient() {
		RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000, 3);  
        CuratorFramework client = CuratorFrameworkFactory.builder()  
                .connectString("127.0.0.1:2181")  
                .sessionTimeoutMs(5000)  
                .connectionTimeoutMs(10000)  
                .retryPolicy(retryPolicy)  
                .namespace("text").build();
        client.start();
        return client;
	}
	
	public static void main(String[] args) throws Exception {
		// (1)
		CuratorFramework curatorFramework = CuratorFrameworkFactory.newClient("centos1:2182", new ExponentialBackoffRetry(1000, 1));
		curatorFramework.start();
		// (2)
		ServiceInstance<Void> instance1 = ServiceInstance.<Void>builder()
                .name("es1")
                .port(8080)
                .address("centos1")   //address不寫的話,會取本地ip
                //.payload(new Void(UUID.ra ndomUUID().toString(),"centos1",9200))
                .uriSpec(new UriSpec("{scheme}://{address}:{port}"))
                .build();
		ServiceDiscovery<Void> serviceDiscovery = registerInZookeeper2(curatorFramework, instance1);
		Iterator<String> ItrServices = serviceDiscovery.queryForNames().iterator();
		while(ItrServices.hasNext()) {
			System.out.println("ItrServices~~~" + ItrServices.next());
		}
		serviceDiscovery.registerService(instance1);  //註冊方式1
		Iterator ItrInstances = serviceDiscovery.queryForInstances("es1").iterator();
		while(ItrInstances.hasNext()) {
			System.out.println("ItrInstances~~~" + ItrInstances.next());
		}
		ServiceInstance<Void> instance_loopup = getInstanceByName(serviceDiscovery,"es1");
		System.out.println("address~~~" + instance_loopup.buildUriSpec());
		System.out.println("payload~~~" + instance_loopup.getPayload());
		//System.out.println("listenAddr~~~" + instance_loopup.getPayload().getListenAddress());
		
		close();
		CloseableUtils.closeQuietly(curatorFramework);
	}
	
	/*
	//register
	ServiceDiscovery = ServiceDiscoveryBuilder.builder.thisInstance(serviceInstance)
	start()

	//lookup (by load balancer)
	ServiceProvider = serviceDiscovery.serviceProviderBuilder().providerStrategy
	start()

	//invoke service

	-------
	CuratorFramework = CuratorFrameworkFactory.newClient / builder
	start()
	ServiceInstance = ServiceInstance.builder
	 */
}

3、單機負載均衡測試
多個服務是否按照負載均衡策略進行訪問,需要利用REST http方式進行測試。單機測試如下:

ZkTest02.java

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.util.concurrent.Executors;

import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.apache.curator.x.discovery.ServiceDiscovery;
import org.apache.curator.x.discovery.ServiceDiscoveryBuilder;
import org.apache.curator.x.discovery.ServiceInstance;
import org.apache.curator.x.discovery.ServiceProvider;
import org.apache.curator.x.discovery.UriSpec;
import org.apache.curator.x.discovery.details.JsonInstanceSerializer;
import org.apache.curator.x.discovery.strategies.RandomStrategy;
import org.apache.curator.x.discovery.strategies.RoundRobinStrategy;
import org.slf4j.LoggerFactory;

import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
import com.sun.net.httpserver.spi.HttpServerProvider;


public class ZkTest02 {
	
	private static String basePath = "/es_path";
	private static ServiceDiscovery<Void> serviceDiscovery;
	private static ServiceProvider<Void> provider;
	
	public static void main(String[] args) throws IOException {
        httpserverService();
    }
	
	public static CuratorFramework getClient() {
		CuratorFramework curatorFramework = CuratorFrameworkFactory.newClient("centos1:2182", new ExponentialBackoffRetry(1000, 1));
		curatorFramework.start();
		return curatorFramework;
	}
	public static ServiceDiscovery<Void> getServiceDiscovery(CuratorFramework client) throws Exception {
		JsonInstanceSerializer<Void> serializer = new JsonInstanceSerializer<Void>(
				Void.class);
		ServiceDiscovery<Void> serviceDiscovery = ServiceDiscoveryBuilder
				.builder(Void.class)
				.client(client)
				.serializer(serializer)
				.basePath(basePath)
				.build();
		serviceDiscovery.start();
		return serviceDiscovery;
	}
	public static ServiceInstance<Void> getInstance1() throws Exception {
		ServiceInstance<Void> instance1 = ServiceInstance.<Void>builder()
                .name("es1")
                .port(8080)
                .address("centos1")
                .uriSpec(new UriSpec("http://centos1:8080"))
                .build();
		return instance1;
	}
	public static ServiceInstance<Void> getInstance2() throws Exception {
		ServiceInstance<Void> instance1 = ServiceInstance.<Void>builder()
                .name("es1")
                .port(9200)
                .address("centos1")
                .uriSpec(new UriSpec("http://centos1:9200"))
                .build();
		return instance1;
	}
	public static ServiceProvider<Void> serviceProvider(ServiceDiscovery<Void> serviceDiscovery, 
			String serviceName) throws Exception {
		ServiceProvider<Void> provider = serviceDiscovery.serviceProviderBuilder().
                serviceName(serviceName).
                providerStrategy(new RandomStrategy<Void>())
                .build();
        provider.start();
		return provider;
	}
	public static void process(String param) throws Exception {
		if (null == serviceDiscovery) {
    		LoggerFactory.getLogger("");
	    	serviceDiscovery = getServiceDiscovery(getClient());
	    	serviceDiscovery.registerService(getInstance1());
	    	serviceDiscovery.registerService(getInstance2());
    	}
		
    	if (null == provider) {
    		provider = serviceProvider(serviceDiscovery, param);
    	}
    	if (null != provider) {
	    	System.out.println("address~~~" + provider.getInstance().getAddress() + 
	    		", port~~~" + provider.getInstance().getPort());
//		    provider.close();
    	}
	}
	
	//啓動服務,監聽來自客戶端的請求
    public static void httpserverService() throws IOException {
        HttpServerProvider provider = HttpServerProvider.provider();
        HttpServer httpserver =provider.createHttpServer(new InetSocketAddress(8080), 100);//監聽端口6666,能同時接 受100個請求
        httpserver.createContext("/sc/es", new MyHttpHandler());
        //httpserver.setExecutor(null); //使用單線程
        httpserver.setExecutor(Executors.newFixedThreadPool(5));
        httpserver.start();
        System.out.println("server started");
    }
    //Http請求處理類
    static class MyHttpHandler implements HttpHandler {
        public void handle(HttpExchange httpExchange) throws IOException {
            String responseMsg = "ok";   //響應信息
            
            InputStream in = httpExchange.getRequestBody(); //獲得輸入流
            BufferedReader reader = new BufferedReader(new InputStreamReader(in));
            String temp = null;
            while((temp = reader.readLine()) != null) {
                System.out.println("client request:"+temp);
                try {
                	process(temp);
                } catch (Exception e) {
                	e.printStackTrace();
                }
            }
            
            httpExchange.sendResponseHeaders(200, responseMsg.length()); //設置響應頭屬性及響應信息的長度
            OutputStream out = httpExchange.getResponseBody();  //獲得輸出流
            out.write(responseMsg.getBytes());
            out.flush();
            httpExchange.close();
        }
    }
}

SpringRestTestClient.java (測試類)

import java.net.URI;

import org.springframework.web.client.RestClientException;
import org.springframework.web.client.RestTemplate;


public class SpringRestTestClient {
	public static final String REST_SERVICE_URI = "http://localhost:8080/sc";
//	public static final String REST_SERVICE_URI = "http://localhost:6666/myApp";
	
	private static void listAllUsers(){
		RestTemplate restTemplate = new RestTemplate();
		String param = "es1";
		try {
//			URI uri = restTemplate.postForLocation(REST_SERVICE_URI+"/es", jsonParam, String.class);
//			if (null != uri){
//				System.out.println("Location : "+uri.toASCIIString());
//			} else {
//				System.out.println("uri is null~");
//			}
			String result = restTemplate.postForObject(REST_SERVICE_URI+"/es", param, String.class);
			System.out.println("es-result~" + result);
		} catch (RestClientException e) {
			e.printStackTrace();
		}
	}
	
	public static void main(String[] args) {
		listAllUsers();
	}
}

先啓動ZkTest02,然後連續運行SpringRestTestClient 幾次。ZkTest02那邊的結果(sys out):

client request:es1
address~~~centos1, port~~~8080
client request:es1
address~~~centos1, port~~~9200
client request:es1
address~~~centos1, port~~~9200
client request:es1
address~~~centos1, port~~~8080
client request:es1
address~~~centos1, port~~~9200

4、Spring下負載均衡測試代碼

另外,也可以用Spring controller來做測試,不過遇到一些引用包的問題,應該與spring相關,由於前面的方法已經滿足PoC需求,暫時沒有再試下去,有興趣也可以參考以下代碼:

package com.test;

import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.apache.curator.x.discovery.ServiceDiscovery;
import org.apache.curator.x.discovery.ServiceDiscoveryBuilder;
import org.apache.curator.x.discovery.ServiceInstance;
import org.apache.curator.x.discovery.ServiceProvider;
import org.apache.curator.x.discovery.UriSpec;
import org.apache.curator.x.discovery.details.JsonInstanceSerializer;
import org.apache.curator.x.discovery.strategies.RoundRobinStrategy;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.util.UriComponentsBuilder;

@RestController
@RequestMapping("/zk")
public class ZkController {
	
	private static String basePath = "/es_path";
	private ServiceDiscovery<Void> serviceDiscovery;
	
	@RequestMapping(value = "/es", method = RequestMethod.POST)
    public String postUser(@RequestBody String param, UriComponentsBuilder ucBuilder) throws Exception {
    	if (null == serviceDiscovery) {
    		LoggerFactory.getLogger("");
	    	serviceDiscovery = getServiceDiscovery(getClient());
	    	serviceDiscovery.registerService(getInstance1());
	    	serviceDiscovery.registerService(getInstance2());
    	}
    	System.out.println("param~~~" + param);
    	ServiceProvider<Void> provider = serviceProvider(serviceDiscovery, param);
    	if (null != provider) {
	    	System.out.println("address~~~" + provider.getInstance().getAddress() + 
	    			", port~~~" + provider.getInstance().getPort());
	    	provider.close();
    	}
    	return param;
    }
	
	public CuratorFramework getClient() {
		CuratorFramework curatorFramework = CuratorFrameworkFactory.newClient("centos1:2182", new ExponentialBackoffRetry(1000, 1));
		curatorFramework.start();
		return curatorFramework;
	}
	public ServiceDiscovery<Void> getServiceDiscovery(CuratorFramework client) throws Exception {
		JsonInstanceSerializer<Void> serializer = new JsonInstanceSerializer<Void>(
				Void.class);
		ServiceDiscovery<Void> serviceDiscovery = ServiceDiscoveryBuilder
				.builder(Void.class)
				.client(client)
				.serializer(serializer)
				.basePath(basePath)
				.build();
		serviceDiscovery.start();
		return serviceDiscovery;
	}
	public ServiceInstance<Void> getInstance1() throws Exception {
		ServiceInstance<Void> instance1 = ServiceInstance.<Void>builder()
                .name("es1")
                .port(8080)
                .address("centos1")
                .uriSpec(new UriSpec("http://centos1:8080"))
                .build();
		return instance1;
	}
	public ServiceInstance<Void> getInstance2() throws Exception {
		ServiceInstance<Void> instance1 = ServiceInstance.<Void>builder()
                .name("es1")
                .port(9200)
                .address("centos1")
                .uriSpec(new UriSpec("http://centos1:9200"))
                .build();
		return instance1;
	}
	public ServiceProvider<Void> serviceProvider(ServiceDiscovery<Void> serviceDiscovery, 
			String serviceName) throws Exception {
		ServiceProvider<Void> provider = serviceDiscovery.serviceProviderBuilder().
                serviceName(serviceName).
                providerStrategy(new RoundRobinStrategy<Void>())
                .build();
        provider.start();
		return provider;
	}
}

上面的代碼只是基於zookeeer znode操作,對服務的監測還是需要自己實現的,比如一個provider - war程序服務,可以做一個REST接口出來進行監聽。



5、Q&A

Q: curator沒有現成的jar包
A: 上http://search.maven.org/
Q: Connection Timeout
A: 檢查zookeeper、服務的端口是否打開(如在centos,打開端口命令爲/sbin/iptables -I INPUT -p tcp --dport 2181 -j ACCEPT, /etc/init.d/iptables save, service iptables restart
Q: base path找不到
A: 啓動zookeeper, 運行./zkCli.sh,使用zookeeer命令:ls / 看有沒有base_path,沒有就新建,如上面的例子,put /es_path/es1,ls /es_path/es1
Q: 在瀏覽器http訪問測試時,報不支持post錯誤或訪問爲get方式
A: post方式不能用瀏覽器http直接訪問測試,用代碼發送post請求或用postman來測試;
Q: 除了Curator,還有其它實現手段嗎
A: (1)zoologist(http://npm.taobao.org/package/zoologist,基於Curator REST接口以Node.js實現的包);(2)Spring Cloud Zookeeper(http://cloud.spring.io/spring-cloud-zookeeper/spring-cloud-zookeeper.html,基於ZK和curator,spring boot上的實現);(3)dubbo(http://dubbo.io/User+Guide-zh.htm,阿里框架,可以基於zookeeper和curator,並支持可視化交互monitor)。
Q: 有沒有其它Sample
A: (1)GitHub - curator-playground-master, curator-disco-master, disco-java;(2)Link: http://www.jianshu.com/p/c0ba6d62dd2e

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章