網上實際一些的案例嚴重缺乏,真心不易出此稿~~
1、zookeeper 服務註冊發現模型
流程:1)註冊,2)發現:監聽、負載均衡、故障檢測、變更通知,3)調用。
From http://blog.cloudera.com/blog/2014/03/zookeeper-resilience-at-pinterest/
From http://www.techweb.com.cn/network/hardware/2015-12-25/2246973.shtml
From http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_hadoop-ha/content/ha-hs2-service-discovery.html
2、單機代碼
虛擬機準備:1)啓動zookeeper,2)啓動多個服務(如ElasticSearch, Tomcat之類的),3)一個服務(如tomcat、elasticsearch)對應多個instance(如節點1、節點2)
流程:
1)準備:CuratorFramework(可選擇getClient或newClient方法),ServiceInstance;
2)註冊:ServiceDiscovery (註冊有兩種方式:registerService/thisInstance);
3.1)負載均衡:ServiceProvider providerStrategy()
3.2)啓動監聽:ServiceProvider start(),一個provider對應一個服務;
4)通知:ServiceProvider getInstance()
5)調用: 有ip、port,想怎麼用就怎麼用吧~
import java.io.Closeable;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.apache.curator.retry.RetryNTimes;
import org.apache.curator.utils.CloseableUtils;
import org.apache.curator.x.discovery.ServiceDiscovery;
import org.apache.curator.x.discovery.ServiceDiscoveryBuilder;
import org.apache.curator.x.discovery.ServiceInstance;
import org.apache.curator.x.discovery.ServiceProvider;
import org.apache.curator.x.discovery.UriSpec;
import org.apache.curator.x.discovery.details.JsonInstanceSerializer;
import org.apache.curator.x.discovery.strategies.RandomStrategy;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
/*
* http://www.techweb.com.cn/network/hardware/2015-12-25/2246973.shtml
* http://blog.csdn.net/hxpjava1/article/details/8612228
* http://www.tuicool.com/articles/F3Avue
* http://www.bkjia.com/Javabc/879495.html
* http://curator.apache.org/apidocs/index.html
* http://blog.csdn.net/john_f_lau/article/details/50660195 *
* http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.curator%22
* share one CuratorFramework per ZooKeeper cluster
* By default ServiceProvider uses Round-robin ProviderStrategy
*/
public class ZkTest01 {
private static String basePath = "/es_path";
private static List<Closeable> closeableList = Lists.newArrayList();
// Register worker 1
private static ServiceDiscovery<Void> registerInZookeeper(CuratorFramework client,
ServiceInstance<Void> instance) throws Exception {
ServiceDiscovery<Void> serviceDiscovery;
JsonInstanceSerializer<Void> serializer = new JsonInstanceSerializer<Void>(
Void.class);
//(3)
serviceDiscovery = ServiceDiscoveryBuilder
.builder(Void.class)
.client(client)
.serializer(serializer)
.basePath(basePath)
//.thisInstance(instance) //註冊方式2
.build();
serviceDiscovery.start();
//(4) cd <ZK_ROOT>/bin, ./zkCli.sh
return serviceDiscovery;
}
// Register worker 2
private static ServiceDiscovery<Void> registerInZookeeper2(CuratorFramework client,
ServiceInstance<Void> instance) throws Exception {
ServiceDiscovery<Void> serviceDiscovery;
JsonInstanceSerializer<Void> serializer = new JsonInstanceSerializer<Void>(
Void.class);
//(3)
serviceDiscovery = ServiceDiscoveryBuilder
.builder(Void.class)
.client(client)
.serializer(serializer)
.basePath(basePath)
//.thisInstance(instance) //註冊方式2
.build();
serviceDiscovery.start();
//(4) cd <ZK_ROOT>/bin, ./zkCli.sh
return serviceDiscovery;
}
public static ServiceInstance<Void> getInstanceByName(ServiceDiscovery<Void> serviceDiscovery,
String serviceName) throws Exception {
//(5) 非註冊方式,do not register in ZooKeeper, get instance from ServiceProvider
ServiceProvider<Void> provider = serviceDiscovery.serviceProviderBuilder().
serviceName(serviceName).
providerStrategy(new RandomStrategy<Void>())
.build();
provider.start();
closeableList.add(provider);
return provider.getInstance();
}
public static synchronized void close(){
for (Closeable closeable : closeableList) {
CloseableUtils.closeQuietly(closeable);
}
}
public static CuratorFramework getClient() {
RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000, 3);
CuratorFramework client = CuratorFrameworkFactory.builder()
.connectString("127.0.0.1:2181")
.sessionTimeoutMs(5000)
.connectionTimeoutMs(10000)
.retryPolicy(retryPolicy)
.namespace("text").build();
client.start();
return client;
}
public static void main(String[] args) throws Exception {
// (1)
CuratorFramework curatorFramework = CuratorFrameworkFactory.newClient("centos1:2182", new ExponentialBackoffRetry(1000, 1));
curatorFramework.start();
// (2)
ServiceInstance<Void> instance1 = ServiceInstance.<Void>builder()
.name("es1")
.port(8080)
.address("centos1") //address不寫的話,會取本地ip
//.payload(new Void(UUID.ra ndomUUID().toString(),"centos1",9200))
.uriSpec(new UriSpec("{scheme}://{address}:{port}"))
.build();
ServiceDiscovery<Void> serviceDiscovery = registerInZookeeper2(curatorFramework, instance1);
Iterator<String> ItrServices = serviceDiscovery.queryForNames().iterator();
while(ItrServices.hasNext()) {
System.out.println("ItrServices~~~" + ItrServices.next());
}
serviceDiscovery.registerService(instance1); //註冊方式1
Iterator ItrInstances = serviceDiscovery.queryForInstances("es1").iterator();
while(ItrInstances.hasNext()) {
System.out.println("ItrInstances~~~" + ItrInstances.next());
}
ServiceInstance<Void> instance_loopup = getInstanceByName(serviceDiscovery,"es1");
System.out.println("address~~~" + instance_loopup.buildUriSpec());
System.out.println("payload~~~" + instance_loopup.getPayload());
//System.out.println("listenAddr~~~" + instance_loopup.getPayload().getListenAddress());
close();
CloseableUtils.closeQuietly(curatorFramework);
}
/*
//register
ServiceDiscovery = ServiceDiscoveryBuilder.builder.thisInstance(serviceInstance)
start()
//lookup (by load balancer)
ServiceProvider = serviceDiscovery.serviceProviderBuilder().providerStrategy
start()
//invoke service
-------
CuratorFramework = CuratorFrameworkFactory.newClient / builder
start()
ServiceInstance = ServiceInstance.builder
*/
}
3、單機負載均衡測試
多個服務是否按照負載均衡策略進行訪問,需要利用REST http方式進行測試。單機測試如下:
ZkTest02.java
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.util.concurrent.Executors;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.apache.curator.x.discovery.ServiceDiscovery;
import org.apache.curator.x.discovery.ServiceDiscoveryBuilder;
import org.apache.curator.x.discovery.ServiceInstance;
import org.apache.curator.x.discovery.ServiceProvider;
import org.apache.curator.x.discovery.UriSpec;
import org.apache.curator.x.discovery.details.JsonInstanceSerializer;
import org.apache.curator.x.discovery.strategies.RandomStrategy;
import org.apache.curator.x.discovery.strategies.RoundRobinStrategy;
import org.slf4j.LoggerFactory;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
import com.sun.net.httpserver.spi.HttpServerProvider;
public class ZkTest02 {
private static String basePath = "/es_path";
private static ServiceDiscovery<Void> serviceDiscovery;
private static ServiceProvider<Void> provider;
public static void main(String[] args) throws IOException {
httpserverService();
}
public static CuratorFramework getClient() {
CuratorFramework curatorFramework = CuratorFrameworkFactory.newClient("centos1:2182", new ExponentialBackoffRetry(1000, 1));
curatorFramework.start();
return curatorFramework;
}
public static ServiceDiscovery<Void> getServiceDiscovery(CuratorFramework client) throws Exception {
JsonInstanceSerializer<Void> serializer = new JsonInstanceSerializer<Void>(
Void.class);
ServiceDiscovery<Void> serviceDiscovery = ServiceDiscoveryBuilder
.builder(Void.class)
.client(client)
.serializer(serializer)
.basePath(basePath)
.build();
serviceDiscovery.start();
return serviceDiscovery;
}
public static ServiceInstance<Void> getInstance1() throws Exception {
ServiceInstance<Void> instance1 = ServiceInstance.<Void>builder()
.name("es1")
.port(8080)
.address("centos1")
.uriSpec(new UriSpec("http://centos1:8080"))
.build();
return instance1;
}
public static ServiceInstance<Void> getInstance2() throws Exception {
ServiceInstance<Void> instance1 = ServiceInstance.<Void>builder()
.name("es1")
.port(9200)
.address("centos1")
.uriSpec(new UriSpec("http://centos1:9200"))
.build();
return instance1;
}
public static ServiceProvider<Void> serviceProvider(ServiceDiscovery<Void> serviceDiscovery,
String serviceName) throws Exception {
ServiceProvider<Void> provider = serviceDiscovery.serviceProviderBuilder().
serviceName(serviceName).
providerStrategy(new RandomStrategy<Void>())
.build();
provider.start();
return provider;
}
public static void process(String param) throws Exception {
if (null == serviceDiscovery) {
LoggerFactory.getLogger("");
serviceDiscovery = getServiceDiscovery(getClient());
serviceDiscovery.registerService(getInstance1());
serviceDiscovery.registerService(getInstance2());
}
if (null == provider) {
provider = serviceProvider(serviceDiscovery, param);
}
if (null != provider) {
System.out.println("address~~~" + provider.getInstance().getAddress() +
", port~~~" + provider.getInstance().getPort());
// provider.close();
}
}
//啓動服務,監聽來自客戶端的請求
public static void httpserverService() throws IOException {
HttpServerProvider provider = HttpServerProvider.provider();
HttpServer httpserver =provider.createHttpServer(new InetSocketAddress(8080), 100);//監聽端口6666,能同時接 受100個請求
httpserver.createContext("/sc/es", new MyHttpHandler());
//httpserver.setExecutor(null); //使用單線程
httpserver.setExecutor(Executors.newFixedThreadPool(5));
httpserver.start();
System.out.println("server started");
}
//Http請求處理類
static class MyHttpHandler implements HttpHandler {
public void handle(HttpExchange httpExchange) throws IOException {
String responseMsg = "ok"; //響應信息
InputStream in = httpExchange.getRequestBody(); //獲得輸入流
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
String temp = null;
while((temp = reader.readLine()) != null) {
System.out.println("client request:"+temp);
try {
process(temp);
} catch (Exception e) {
e.printStackTrace();
}
}
httpExchange.sendResponseHeaders(200, responseMsg.length()); //設置響應頭屬性及響應信息的長度
OutputStream out = httpExchange.getResponseBody(); //獲得輸出流
out.write(responseMsg.getBytes());
out.flush();
httpExchange.close();
}
}
}
SpringRestTestClient.java (測試類)
import java.net.URI;
import org.springframework.web.client.RestClientException;
import org.springframework.web.client.RestTemplate;
public class SpringRestTestClient {
public static final String REST_SERVICE_URI = "http://localhost:8080/sc";
// public static final String REST_SERVICE_URI = "http://localhost:6666/myApp";
private static void listAllUsers(){
RestTemplate restTemplate = new RestTemplate();
String param = "es1";
try {
// URI uri = restTemplate.postForLocation(REST_SERVICE_URI+"/es", jsonParam, String.class);
// if (null != uri){
// System.out.println("Location : "+uri.toASCIIString());
// } else {
// System.out.println("uri is null~");
// }
String result = restTemplate.postForObject(REST_SERVICE_URI+"/es", param, String.class);
System.out.println("es-result~" + result);
} catch (RestClientException e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
listAllUsers();
}
}
先啓動ZkTest02,然後連續運行SpringRestTestClient 幾次。ZkTest02那邊的結果(sys out):
client request:es1
address~~~centos1, port~~~8080
client request:es1
address~~~centos1, port~~~9200
client request:es1
address~~~centos1, port~~~9200
client request:es1
address~~~centos1, port~~~8080
client request:es1
address~~~centos1, port~~~9200
4、Spring下負載均衡測試代碼
另外,也可以用Spring controller來做測試,不過遇到一些引用包的問題,應該與spring相關,由於前面的方法已經滿足PoC需求,暫時沒有再試下去,有興趣也可以參考以下代碼:
package com.test;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.apache.curator.x.discovery.ServiceDiscovery;
import org.apache.curator.x.discovery.ServiceDiscoveryBuilder;
import org.apache.curator.x.discovery.ServiceInstance;
import org.apache.curator.x.discovery.ServiceProvider;
import org.apache.curator.x.discovery.UriSpec;
import org.apache.curator.x.discovery.details.JsonInstanceSerializer;
import org.apache.curator.x.discovery.strategies.RoundRobinStrategy;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.util.UriComponentsBuilder;
@RestController
@RequestMapping("/zk")
public class ZkController {
private static String basePath = "/es_path";
private ServiceDiscovery<Void> serviceDiscovery;
@RequestMapping(value = "/es", method = RequestMethod.POST)
public String postUser(@RequestBody String param, UriComponentsBuilder ucBuilder) throws Exception {
if (null == serviceDiscovery) {
LoggerFactory.getLogger("");
serviceDiscovery = getServiceDiscovery(getClient());
serviceDiscovery.registerService(getInstance1());
serviceDiscovery.registerService(getInstance2());
}
System.out.println("param~~~" + param);
ServiceProvider<Void> provider = serviceProvider(serviceDiscovery, param);
if (null != provider) {
System.out.println("address~~~" + provider.getInstance().getAddress() +
", port~~~" + provider.getInstance().getPort());
provider.close();
}
return param;
}
public CuratorFramework getClient() {
CuratorFramework curatorFramework = CuratorFrameworkFactory.newClient("centos1:2182", new ExponentialBackoffRetry(1000, 1));
curatorFramework.start();
return curatorFramework;
}
public ServiceDiscovery<Void> getServiceDiscovery(CuratorFramework client) throws Exception {
JsonInstanceSerializer<Void> serializer = new JsonInstanceSerializer<Void>(
Void.class);
ServiceDiscovery<Void> serviceDiscovery = ServiceDiscoveryBuilder
.builder(Void.class)
.client(client)
.serializer(serializer)
.basePath(basePath)
.build();
serviceDiscovery.start();
return serviceDiscovery;
}
public ServiceInstance<Void> getInstance1() throws Exception {
ServiceInstance<Void> instance1 = ServiceInstance.<Void>builder()
.name("es1")
.port(8080)
.address("centos1")
.uriSpec(new UriSpec("http://centos1:8080"))
.build();
return instance1;
}
public ServiceInstance<Void> getInstance2() throws Exception {
ServiceInstance<Void> instance1 = ServiceInstance.<Void>builder()
.name("es1")
.port(9200)
.address("centos1")
.uriSpec(new UriSpec("http://centos1:9200"))
.build();
return instance1;
}
public ServiceProvider<Void> serviceProvider(ServiceDiscovery<Void> serviceDiscovery,
String serviceName) throws Exception {
ServiceProvider<Void> provider = serviceDiscovery.serviceProviderBuilder().
serviceName(serviceName).
providerStrategy(new RoundRobinStrategy<Void>())
.build();
provider.start();
return provider;
}
}
上面的代碼只是基於zookeeer znode操作,對服務的監測還是需要自己實現的,比如一個provider - war程序服務,可以做一個REST接口出來進行監聽。
5、Q&A
Q: curator沒有現成的jar包
A: 上http://search.maven.org/
Q: Connection Timeout
A: 檢查zookeeper、服務的端口是否打開(如在centos,打開端口命令爲/sbin/iptables -I INPUT -p tcp --dport 2181 -j ACCEPT, /etc/init.d/iptables save, service iptables restart)
Q: base path找不到
A: 啓動zookeeper, 運行./zkCli.sh,使用zookeeer命令:ls / 看有沒有base_path,沒有就新建,如上面的例子,put /es_path/es1,ls /es_path/es1
Q: 在瀏覽器http訪問測試時,報不支持post錯誤或訪問爲get方式
A: post方式不能用瀏覽器http直接訪問測試,用代碼發送post請求或用postman來測試;
Q: 除了Curator,還有其它實現手段嗎
A: (1)zoologist(http://npm.taobao.org/package/zoologist,基於Curator REST接口以Node.js實現的包);(2)Spring Cloud Zookeeper(http://cloud.spring.io/spring-cloud-zookeeper/spring-cloud-zookeeper.html,基於ZK和curator,spring boot上的實現);(3)dubbo(http://dubbo.io/User+Guide-zh.htm,阿里框架,可以基於zookeeper和curator,並支持可視化交互monitor)。
Q: 有沒有其它Sample
A: (1)GitHub - curator-playground-master, curator-disco-master, disco-java;(2)Link: http://www.jianshu.com/p/c0ba6d62dd2e