前言
分佈式協調服務，我們主要講四個方面
- 初步認識Zookeeper
- 瞭解Zookeeper的核心原理
- Zookeeper實踐及與原理分析
- Zookeeper實踐之配合註冊中心完成RPC手寫
本節我們就講最後一個部分 Zookeeper實踐之配合註冊中心完成RPC手寫
使用zookeeper原生API實現分佈式鎖
我們在面試專題講過，實現分佈式鎖的方式有很多種，《JAVA多線程面試總結之分佈式鎖的實現原理》
比如使用
- 數據庫
- redis
- zookeeper
本節，我們通過zookeeper實現分佈式鎖
使用zookeeper實現，我們發現一個弊端：同一時間，只有一個節點可以獲取到鎖，而其他的客戶端需要通過watcher來不斷的訂閱 ，以監聽lock節點下的變化，這就會造成驚羣效應
驚羣效應：如果當佔用的節點釋放了鎖，其他節點就會同時去watcher這個鎖，這樣就會在短時間內產生大量變更，如果訪問節點比較多，我們不建議採取這樣的形式。
針對上面的情況，我們可以採用zookeeper的有序節點的特性來實現分佈式鎖。
我們對每個客戶端都在zookeeper註冊一個帶有seq的節點，獲得鎖時，讓每個節點去監聽比它小1的節點，只需要在lock下獲取一個最小值獲得鎖。
這裏我們寫一個demo，通過客戶端註冊有序節點，實現分佈式鎖：
public class DistributedLock implements Lock,Watcher {
private ZooKeeper zk=null;
/**
* 定義根節點
*/
private String ROOT_LOCK="/locks";
/**
* 等待前一個鎖
*/
private String WAIT_LOCK;
/**
* 表示當前的鎖
*/
private String CURRENT_LOCK;
//控制
private CountDownLatch countDownLatch;
public DistributedLock() {
try {
zk=new ZooKeeper("192.168.200.111:2181",
4000,this);
//判斷根節點是否存在
Stat stat=zk.exists(ROOT_LOCK,false);
if(stat==null){
zk.create(ROOT_LOCK,"0".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE,CreateMode.PERSISTENT);
}
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (KeeperException e) {
e.printStackTrace();
}
}
@Override
public boolean tryLock() {
try {
//創建臨時有序節點
CURRENT_LOCK=zk.create(ROOT_LOCK+"/","0".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE,CreateMode.EPHEMERAL_SEQUENTIAL);
System.out.println(Thread.currentThread().getName()+"->"+
CURRENT_LOCK+"，嘗試競爭鎖");
//獲取根節點下的所有子節點
List<String> childrens=zk.getChildren(ROOT_LOCK,false);
//定義一個集合進行排序
SortedSet<String> sortedSet=new TreeSet();
for(String children:childrens){
sortedSet.add(ROOT_LOCK+"/"+children);
}
//獲得當前所有子節點中最小的節點
String firstNode=sortedSet.first();
SortedSet<String> lessThenMe=(sortedSet).headSet(CURRENT_LOCK);
//通過當前的節點和子節點中最小的節點進行比較，如果相等，表示獲得鎖成功
if(CURRENT_LOCK.equals(firstNode)){
return true;
}
if(!lessThenMe.isEmpty()){
//獲得比當前節點更小的最後一個節點，設置給WAIT_LOCK
WAIT_LOCK=lessThenMe.last();
}
} catch (KeeperException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
return false;
}
@Override
public void lock() {
//如果獲得鎖成功
if(this.tryLock()){
System.out.println(Thread.currentThread().getName()+"->"+CURRENT_LOCK+"->獲得鎖成功");
return;
}
try {
//沒有獲得鎖，繼續等待獲得鎖
waitForLock(WAIT_LOCK);
} catch (KeeperException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private boolean waitForLock(String prev) throws KeeperException, InterruptedException {
//監聽當前節點的上一個節點
Stat stat=zk.exists(prev,true);
if(stat!=null){
System.out.println(Thread.currentThread().getName()+"->等待鎖"+ROOT_LOCK+"/"+prev+"釋放");
countDownLatch=new CountDownLatch(1);
countDownLatch.await();
//TODO watcher觸發以後，還需要再次判斷當前等待的節點是不是最小的
System.out.println(Thread.currentThread().getName()+"->獲得鎖成功");
}
return true;
}
@Override
public void lockInterruptibly() throws InterruptedException {
}
@Override
public boolean tryLock(long time, TimeUnit unit) throws InterruptedException {
return false;
}
@Override
public void unlock() {
System.out.println(Thread.currentThread().getName()+"->釋放鎖"+CURRENT_LOCK);
try {
zk.delete(CURRENT_LOCK,-1);
CURRENT_LOCK=null;
zk.close();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (KeeperException e) {
e.printStackTrace();
}
}
@Override
public Condition newCondition() {
return null;
}
@Override
public void process(WatchedEvent event) {
if(this.countDownLatch!=null){
this.countDownLatch.countDown();
}
}
}
測試類：
public static void main( String[] args ) throws IOException {
CountDownLatch countDownLatch=new CountDownLatch(10);
for(int i=0;i<10;i++){
new Thread(()->{
try {
countDownLatch.await();
DistributedLock distributedLock=new DistributedLock();
//獲得鎖
distributedLock.lock();
} catch (InterruptedException e) {
e.printStackTrace();
}
},"Thread-"+i).start();
countDownLatch.countDown();
}
System.in.read();
}
開始測試：
先看看節點：
啓動：
我們看控制檯輸出：
Thread-6->等待鎖/locks//locks/0000000010釋放
現在我們釋放/locks//locks/0000000010（刪掉此節點）
這就是分佈式鎖！
分析Curator實現分佈式鎖的原理
上述內容，我們通過zookeeper的API實現了分佈式鎖，實際上，zookeeper通過Curator封裝了這個過程，只需要調用幾個類即可完成
public static void main(String[] args) {
CuratorFramework curatorFramework=CuratorFrameworkFactory.builder().build();
InterProcessMutex interProcessMutex=new InterProcessMutex(curatorFramework,"/locks");
try {
interProcessMutex.acquire();
} catch (Exception e) {
e.printStackTrace();
}
}
所以，Curator源碼究竟是怎麼實現的呢？
我們先看new InterProcessMutex都做了什麼事情？
InterProcessMutex(CuratorFramework client, String path, String lockName, int maxLeases, LockInternalsDriver driver)
{
basePath = PathUtils.validatePath(path);
internals = new LockInternals(client, driver, path, lockName, maxLeases);
}
首先在初始化的過程中構造了LockInternals，回過頭來看我們調用的acquir()方法：
@Override
public void acquire() throws Exception
{
if ( !internalLock(-1, null) )
{
throw new IOException("Lost connection while trying to acquire lock: " + basePath);
}
}
通過internalLock獲得鎖：
這是一個可重入鎖，
private boolean internalLock(long time, TimeUnit unit) throws Exception{
Thread currentThread = Thread.currentThread();
LockData lockData = threadData.get(currentThread);
if ( lockData != null ){
// 獲得鎖了就遞增
lockData.lockCount.incrementAndGet();
return true;
}
//否則嘗試獲得鎖
String lockPath = internals.attemptLock(time, unit, getLockNodeBytes());
if ( lockPath != null ){
LockData newLockData = new LockData(currentThread, lockPath);
threadData.put(currentThread, newLockData);
return true;
}
return false;
}
看獲得鎖代碼塊：attemptLock
String attemptLock(long time, TimeUnit unit, byte[] lockNodeBytes) throws Exception
{
final long startMillis = System.currentTimeMillis();
final Long millisToWait = (unit != null) ? unit.toMillis(time) : null;
final byte[] localLockNodeBytes = (revocable.get() != null) ? new byte[0] : lockNodeBytes;
int retryCount = 0;
String ourPath = null;
boolean hasTheLock = false;
boolean isDone = false;
while ( !isDone )
{
isDone = true;
try
{
//創建鎖
ourPath = driver.createsTheLock(client, path, localLockNodeBytes);
//
hasTheLock = internalLockLoop(startMillis, millisToWait, ourPath);
}
catch ( KeeperException.NoNodeException e )
{
if ( client.getZookeeperClient().getRetryPolicy().allowRetry(retryCount++, System.currentTimeMillis() - startMillis, RetryLoop.getDefaultRetrySleeper()) )
{
isDone = false;
}
else
{
throw e;
}
}
}
if ( hasTheLock )
{
return ourPath;
}
return null;
}
看一下這個創建鎖的過程createsTheLock
@Override
public String createsTheLock(CuratorFramework client, String path, byte[] lockNodeBytes) throws Exception
{
String ourPath;
if ( lockNodeBytes != null )
{
//創建一個臨時有序節點
ourPath = client.create().creatingParentContainersIfNeeded().withProtection().withMode(CreateMode.EPHEMERAL_SEQUENTIAL).forPath(path, lockNodeBytes);
}
else
{
ourPath = client.create().creatingParentContainersIfNeeded().withProtection().withMode(CreateMode.EPHEMERAL_SEQUENTIAL).forPath(path);
}
return ourPath;
}
退出此方法，繼續看獲得鎖代碼塊：attemptLock，在創建鎖後internalLockLoop：
獲得鎖的超時機制
private boolean internalLockLoop(long startMillis, Long millisToWait, String ourPath) throws Exception
{
boolean haveTheLock = false;
boolean doDelete = false;
try
{
if ( revocable.get() != null )
{
client.getData().usingWatcher(revocableWatcher).forPath(ourPath);
}
//如果獲得鎖
while ( (client.getState() == CuratorFrameworkState.STARTED) && !haveTheLock )
{
//獲得當前子節點下所有的鎖getSortedChildren
List<String> children = getSortedChildren();
String sequenceNodeName = ourPath.substring(basePath.length() + 1); // +1 to include the slash
PredicateResults predicateResults = driver.getsTheLock(client, children, sequenceNodeName, maxLeases);
if ( predicateResults.getsTheLock() )
{
haveTheLock = true;
}
else
{
String previousSequencePath = basePath + "/" + predicateResults.getPathToWatch();
synchronized(this)
{
try
{
client.getData().usingWatcher(watcher).forPath(previousSequencePath);
if ( millisToWait != null )
{
millisToWait -= (System.currentTimeMillis() - startMillis);
startMillis = System.currentTimeMillis();
if ( millisToWait <= 0 )
{
doDelete = true; // timed out - delete our node
break;
}
wait(millisToWait);
}
else
{
wait();
}
}
catch ( KeeperException.NoNodeException e )
{
// it has been deleted (i.e. lock released). Try to acquire again
}
}
}
}
}
catch ( Exception e )
{
ThreadUtils.checkInterrupted(e);
doDelete = true;
throw e;
}
finally
{
if ( doDelete )
{
deleteOurPath(ourPath);
}
}
return haveTheLock;
}
這裏面有個方法：獲得當前子節點下所有的鎖getSortedChildren
public static List<String> getSortedChildren(CuratorFramework client, String basePath, final String lockName, final LockInternalsSorter sorter) throws Exception
{
List<String> children = client.getChildren().forPath(basePath);
List<String> sortedList = Lists.newArrayList(children);
//排序
Collections.sort
(
sortedList,
new Comparator<String>()
{
@Override
public int compare(String lhs, String rhs)
{
return sorter.fixForSorting(lhs, lockName).compareTo(sorter.fixForSorting(rhs, lockName));
}
}
);
//返回有序的節點列表
return sortedList;
}
退出此方法，接下來看getsTheLock
@Override
public PredicateResults getsTheLock(CuratorFramework client, List<String> children, String sequenceNodeName, int maxLeases) throws Exception
{
//獲取到在children裏面的索引
int ourIndex = children.indexOf(sequenceNodeName);
validateOurIndex(sequenceNodeName, ourIndex);
//是不是當前最小的序號
boolean getsTheLock = ourIndex < maxLeases;
//監控上一個節點
String pathToWatch = getsTheLock ? null : children.get(ourIndex - maxLeases);
return new PredicateResults(pathToWatch, getsTheLock);
}
層層遞進，拿得到最小的節點返回，實現分佈式鎖。
實現帶註冊中心的RPC框架
我們在《分佈式專題-分佈式通信框架RMI原理分析》實現了基於RMI的遠程調用，接下來，我們通過以zookeeper註冊中心的形式，實現帶註冊中心的RPC框架
首先在服務端：
我們先創建一個註冊中心的接口：
public interface IRegisterCenter {
/**
* 註冊服務名稱和服務地址
* @param serviceName
* @param serviceAddress
*/
void register(String serviceName,String serviceAddress);
}
註冊中心實現類：
public class RegisterCenterImpl implements IRegisterCenter{
private CuratorFramework curatorFramework;
{
curatorFramework=CuratorFrameworkFactory.builder().
connectString(ZkConfig.CONNNECTION_STR).
sessionTimeoutMs(4000).
retryPolicy(new ExponentialBackoffRetry(1000,
10)).build();
curatorFramework.start();
}
@Override
public void register(String serviceName, String serviceAddress) {
//註冊相應的服務
String servicePath=ZkConfig.ZK_REGISTER_PATH+"/"+serviceName;
try {
//判斷 /registrys/product-service是否存在，不存在則創建
if(curatorFramework.checkExists().forPath(servicePath)==null){
curatorFramework.create().creatingParentsIfNeeded().
withMode(CreateMode.PERSISTENT).forPath(servicePath,"0".getBytes());
}
String addressPath=servicePath+"/"+serviceAddress;
String rsNode=curatorFramework.create().withMode(CreateMode.EPHEMERAL).
forPath(addressPath,"0".getBytes());
System.out.println("服務註冊成功："+rsNode);
} catch (Exception e) {
e.printStackTrace();
}
}
}
zkConfig：
public class ZkConfig {
public final static String CONNNECTION_STR="192.168.200.111:2181,192.168.200.112:2181,192.168.200.113:2181";
public final static String ZK_REGISTER_PATH="/registrys";
}
用於發佈一個遠程服務：
public class RpcServer {
//創建一個線程池
private static final ExecutorService executorService=Executors.newCachedThreadPool();
//註冊中心
private IRegisterCenter registerCenter;
//服務發佈地址
private String serviceAddress;
// 存放服務名稱和服務對象之間的關係
Map<String,Object> handlerMap=new HashMap<>();
public RpcServer(IRegisterCenter registerCenter, String serviceAddress) {
this.registerCenter = registerCenter;
this.serviceAddress = serviceAddress;
}
/**
* 綁定服務名稱和服務對象
* @param services
*/
public void bind(Object... services){
for(Object service:services){
RpcAnnotation annotation=service.getClass().getAnnotation(RpcAnnotation.class);
String serviceName=annotation.value().getName();
String version=annotation.version();
if(version!=null&&!version.equals("")){
serviceName=serviceName+"-"+version;
}
//綁定服務接口名稱對應的服務
handlerMap.put(serviceName,service);
}
}
public void publisher(){
ServerSocket serverSocket=null;
try{
String[] addrs=serviceAddress.split(":");
//啓動一個服務監聽
serverSocket=new ServerSocket(Integer.parseInt(addrs[1]));
for(String interfaceName:handlerMap.keySet()){
registerCenter.register(interfaceName,serviceAddress);
System.out.println("註冊服務成功："+interfaceName+"->"+serviceAddress);
}
//循環監聽
while(true){
//監聽服務
Socket socket=serverSocket.accept();
//通過線程池去處理請求
executorService.execute(new ProcessorHandler(socket,handlerMap));
}
} catch (IOException e) {
e.printStackTrace();
}finally {
if(serverSocket!=null){
try {
serverSocket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
註解：
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
public @interface RpcAnnotation {
/**
* 對外發布的服務的接口地址
* @return
*/
Class<?> value();
String version() default "";
}
註解注入：
首先是接口：
public interface IGpHello {
String sayHello(String msg);
}
再來兩個實現類：
@RpcAnnotation(IGpHello.class)
public class GpHelloImpl implements IGpHello{
@Override
public String sayHello(String msg) {
return "I'm 8080 Node , "+msg;
}
}
@RpcAnnotation(value = IGpHello.class)
public class GpHelloImpl2 implements IGpHello{
@Override
public String sayHello(String msg) {
return "I'm 8081 node ："+msg;
}
}
測試類：
public class ServerDemo {
public static void main(String[] args) throws IOException {
IGpHello iGpHello=new GpHelloImpl();
IGpHello iGpHello1=new GpHelloImpl2();
IRegisterCenter registerCenter=new RegisterCenterImpl();
RpcServer rpcServer=new RpcServer(registerCenter,"127.0.0.1:8080");
rpcServer.bind(iGpHello,iGpHello1);
rpcServer.publisher();
System.in.read();
}
}
走你！
註冊成功，接下來我們寫客戶端：
首先寫一個發現服務的接口：
public interface IServiceDiscovery {
/**
* 根據請求的服務地址，獲得對應的調用地址
* @param serviceName
* @return
*/
String discover(String serviceName);
}
發現服務實現類：
public class ServiceDiscoveryImpl implements IServiceDiscovery{
List<String> repos=new ArrayList<>();
private String address;
private CuratorFramework curatorFramework;
public ServiceDiscoveryImpl(String address) {
this.address = address;
curatorFramework=CuratorFrameworkFactory.builder().
connectString(address).
sessionTimeoutMs(4000).
retryPolicy(new ExponentialBackoffRetry(1000,
10)).build();
curatorFramework.start();
}
@Override
public String discover(String serviceName) {
String path=ZkConfig.ZK_REGISTER_PATH+"/"+serviceName;
try {
repos=curatorFramework.getChildren().forPath(path);
} catch (Exception e) {
throw new RuntimeException("獲取子節點異常："+e);
}
//動態發現服務節點的變化
registerWatcher(path);
//負載均衡機制
LoadBanalce loadBanalce=new RandomLoadBanalce();
return loadBanalce.selectHost(repos); //返回調用的服務地址
}
private void registerWatcher(final String path){
PathChildrenCache childrenCache=new PathChildrenCache
(curatorFramework,path,true);
PathChildrenCacheListener pathChildrenCacheListener=new PathChildrenCacheListener() {
@Override
public void childEvent(CuratorFramework curatorFramework, PathChildrenCacheEvent pathChildrenCacheEvent) throws Exception {
repos=curatorFramework.getChildren().forPath(path);
}
};
childrenCache.getListenable().addListener(pathChildrenCacheListener);
try {
childrenCache.start();
} catch (Exception e) {
throw new RuntimeException("註冊PatchChild Watcher 異常"+e);
}
}
}
Zkconfig同server端，這裏我就不寫了
然後在客戶端實現負載均衡機制
public interface LoadBanalce {
String selectHost(List<String> repos);
}
抽象工廠：
public abstract class AbstractLoadBanance implements LoadBanalce{
@Override
public String selectHost(List<String> repos) {
if(repos==null||repos.size()==0){
return null;
}
if(repos.size()==1){
return repos.get(0);
}
return doSelect(repos);
}
protected abstract String doSelect(List<String> repos);
}
隨機數算法-負載均衡實現類，當然還有很多算法，這裏爲了演示註冊中心負載均衡，暫時不加：
public class RandomLoadBanalce extends AbstractLoadBanance{
@Override
protected String doSelect(List<String> repos) {
int len=repos.size();
Random random=new Random();
return repos.get(random.nextInt(len));
}
}
RemoteInvocationHandler發現地址：
public class RemoteInvocationHandler implements InvocationHandler {
private IServiceDiscovery serviceDiscovery;
private String version;
public RemoteInvocationHandler(IServiceDiscovery serviceDiscovery,String version) {
this.serviceDiscovery=serviceDiscovery;
this.version=version;
}
@Override
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
//組裝請求
RpcRequest request=new RpcRequest();
request.setClassName(method.getDeclaringClass().getName());
request.setMethodName(method.getName());
request.setParameters(args);
request.setVersion(version);
//根據接口名稱得到對應的服務地址
String serviceAddress=serviceDiscovery.discover(request.getClassName());
//通過tcp傳輸協議進行傳輸
TCPTransport tcpTransport=new TCPTransport(serviceAddress);
//發送請求
return tcpTransport.send(request);
}
}
測試類：
public static void main(String[] args) throws InterruptedException {
IServiceDiscovery serviceDiscovery=new
ServiceDiscoveryImpl(ZkConfig.CONNNECTION_STR);
RpcClientProxy rpcClientProxy=new RpcClientProxy(serviceDiscovery);
for(int i=0;i<10;i++) {
IGpHello hello = rpcClientProxy.clientProxy(IGpHello.class, null);
System.out.println(hello.sayHello("mic"));
Thread.sleep(1000);
}
}
走你！
那我們現在停到服務端試一下：
看客戶端：
建立連接失敗，說明註冊中心沒有問題！
我們看一下基於zookeeper的註冊中心時序圖
後記
本小節代碼地址：手寫RPC框架
下一小節預告：
分佈式服務治理
- 解開Dubbo的神祕面紗
- 分佈式服務治理Dubbo常用配置
- 分佈式服務治理Dubbo源碼分析
