這篇文章主要寫 Hadoop RPC Client 的設計 與實現 . 在講解的時候, 以 ProtobufRpcEngine爲實例, 然後分步進行敘述.
一.Client端架構
Client類只有一個入口， 就是call()方法。 代理類會調用Client.call()方法將RPC請求發送到遠程服務器， 然後等待遠程服務器的響應。 如果遠程服務器響應請求時出現異常， 則在call()方法中拋出異常。
在 call 方法中先將遠程調用信息封裝成一個 Client.Call 對象（保存了完成標誌、返回信息、異常信息等），然後得到 connection 對象用於管理 Client 與 Server 的 Socket 連接。
getConnection 方法中通過 setupIOstreams 建立與 Server 的 socket 連接，啓動 Connection 線程，監聽 socket 讀取 server 響應。
call() 方法發送 RCP 請求。
call() 方法調用 Call.wait() 在 Call 對象上等待 Server 響應信息。
Connection 線程收到響應信息設置 Call 對象返回信息字段，並調用 Call.notify() 喚醒 call() 方法線程讀取 Call 對象返回值。
二.Client端創建流程
下面是創建Client端的代碼.協議採用proto, 所以產生的RpcEngine是 ProtobufRpcEngine. 所以接下來的文章是以ProtobufRpcEngine爲藍本進行源碼分析.
我只放了Server端, 詳細的代碼請查看:
Hadoop3.2.1 【 HDFS 】源碼分析 : RPC原理 [六] ProtobufRpcEngine 使用
public static void main(String[] args) throws Exception {
//1. 構建配置對象
Configuration conf = new Configuration();
//2. 設置協議的RpcEngine爲ProtobufRpcEngine .
RPC.setProtocolEngine(conf, Server.MetaInfoProtocol.class,
ProtobufRpcEngine.class);
//3. 拿到代理對象
Server.MetaInfoProtocol proxy = RPC.getProxy(Server.MetaInfoProtocol.class, 1L,
new InetSocketAddress("localhost", 7777), conf);
//4. 構建發送請求對象
CustomProtos.GetMetaInfoRequestProto obj = CustomProtos.GetMetaInfoRequestProto.newBuilder().setPath("/meta").build();
//5. 將請求對象傳入, 獲取響應信息
CustomProtos.GetMetaInfoResponseProto metaData = proxy.getMetaInfo(null, obj);
//6. 輸出數據
System.out.println(metaData.getInfo());
}
上面的代碼, 主要是分三部分.
1.構建配置對象&設置RpcEngine引擎
2.獲取代理對象
3.設置請求參數&通過代理對象請求.
4.處理結果
第一條和第二條,我就不細說了. 這個很簡單,就是使用proto定義一個協議, 綁定到RPC.Builder的實現對象裏面.
我們直接看這段, 獲取代理對象.
Server.MetaInfoProtocol proxy = RPC.getProxy(Server.MetaInfoProtocol.class, 1L,
new InetSocketAddress("localhost", 7777), conf);
也是就是通過RPC.getProxy方法獲取協議的代理對象.
/**
* Construct a client-side proxy object with the default SocketFactory
* @param <T>
*
* @param protocol 協議
* @param clientVersion 客戶端的版本
* @param addr 請求地址
* @param conf 配置文件
* @return a proxy instance
* @throws IOException
*/
public static <T> T getProxy(Class<T> protocol,
long clientVersion,
InetSocketAddress addr, Configuration conf)
throws IOException {
return getProtocolProxy(protocol, clientVersion, addr, conf).getProxy();
}
接續看,這裏面就一句getProtocolProxy,加斷點一直跟進
/**
* Get a protocol proxy that contains a proxy connection to a remote server
* and a set of methods that are supported by the server
*
* @param protocol protocol
* @param clientVersion client's version
* @param addr server address
* @param ticket security ticket
* @param conf configuration
* @param factory socket factory
* @param rpcTimeout max time for each rpc; 0 means no timeout
* @param connectionRetryPolicy retry policy
* @param fallbackToSimpleAuth set to true or false during calls to indicate if
* a secure client falls back to simple auth
* @return the proxy
* @throws IOException if any error occurs
*/
public static <T> ProtocolProxy<T> getProtocolProxy(Class<T> protocol,
long clientVersion,
InetSocketAddress addr,
UserGroupInformation ticket,
Configuration conf,
SocketFactory factory,
int rpcTimeout,
RetryPolicy connectionRetryPolicy,
AtomicBoolean fallbackToSimpleAuth)
throws IOException {
if (UserGroupInformation.isSecurityEnabled()) {
SaslRpcServer.init(conf);
}
return getProtocolEngine(protocol, conf).getProxy(protocol, clientVersion,
addr, ticket, conf, factory, rpcTimeout, connectionRetryPolicy,
fallbackToSimpleAuth, null);
}
debug界面是這樣的:
核心的是這句
return getProtocolEngine(protocol, conf).getProxy(protocol, clientVersion,
addr, ticket, conf, factory, rpcTimeout, connectionRetryPolicy,
fallbackToSimpleAuth, null);
首先通過 getProtocolEngine 獲取RPC Engine [ ProtobufRpcEngine] ,
// return the RpcEngine configured to handle a protocol
static synchronized RpcEngine getProtocolEngine(Class<?> protocol,
Configuration conf) {
//從緩存中獲取RpcEngine ,
// 這個是提前設置的
// 通過 RPC.setProtocolEngine(conf, MetaInfoProtocol.class,ProtobufRpcEngine.class);
RpcEngine engine = PROTOCOL_ENGINES.get(protocol);
if (engine == null) {
//通過這裏 獲取RpcEngine的實現類 , 這裏我們獲取的是 ProtobufRpcEngine.class
Class<?> impl = conf.getClass(ENGINE_PROP+"."+protocol.getName(),
WritableRpcEngine.class);
// impl : org.apache.hadoop.ipc.ProtobufRpcEngine
engine = (RpcEngine)ReflectionUtils.newInstance(impl, conf);
PROTOCOL_ENGINES.put(protocol, engine);
}
return engine;
}
然後再調用 ProtobufRpcEngine的 getProxy方法.將協議,客戶端的版本號. socket地址, ticket , 配置文件, socket 的創建工廠對象[ StandardSocketFactory ] , PRC 服務的超時時間, connetion的重試策略,以及權限等信息,傳入.
@Override
@SuppressWarnings("unchecked")
public <T> ProtocolProxy<T> getProxy(Class<T> protocol, long clientVersion,
InetSocketAddress addr, UserGroupInformation ticket, Configuration conf,
SocketFactory factory, int rpcTimeout, RetryPolicy connectionRetryPolicy,
AtomicBoolean fallbackToSimpleAuth, AlignmentContext alignmentContext)
throws IOException {
//構造一個實現了InvocationHandler接口的invoker 對象
// (動態代理機制中的InvocationHandler對象會在invoke()方法中代理所有目標接口上的 調用，
// 用戶可以在invoke()方法中添加代理操作)
final Invoker invoker = new Invoker(protocol, addr, ticket, conf, factory,
rpcTimeout, connectionRetryPolicy, fallbackToSimpleAuth,
alignmentContext);
//然後調用Proxy.newProxylnstance()獲取動態代理對象，並通過ProtocolProxy返回
return new ProtocolProxy<T>(protocol, (T) Proxy.newProxyInstance(
protocol.getClassLoader(), new Class[]{protocol}, invoker), false);
}
在getProxy 這個方法中. 主要是分兩步,
1. 構造一個實現了InvocationHandler接口的invoker 對象 (動態代理機制中的InvocationHandler對象會在invoke()方法中代理所有目標接口上的 調用， 用戶可以在invoke()方法中添加代理操作
2.調用Proxy.newProxylnstance()獲取動態代理對象，並通過ProtocolProxy返回
我們先看Invoker的創建.
private Invoker(Class<?> protocol, InetSocketAddress addr,
UserGroupInformation ticket, Configuration conf, SocketFactory factory,
int rpcTimeout, RetryPolicy connectionRetryPolicy,
AtomicBoolean fallbackToSimpleAuth, AlignmentContext alignmentContext)
throws IOException {
this(protocol,
Client.ConnectionId.getConnectionId(addr, protocol, ticket, rpcTimeout, connectionRetryPolicy, conf), conf, factory);
this.fallbackToSimpleAuth = fallbackToSimpleAuth;
this.alignmentContext = alignmentContext;
}
主要是:
this(protocol, Client.ConnectionId.getConnectionId(addr, protocol, ticket, rpcTimeout, connectionRetryPolicy, conf), conf, factory);
我們先看
Client.ConnectionId.getConnectionId(addr, protocol, ticket, rpcTimeout, connectionRetryPolicy, conf), conf, factory)
這裏會調用getConnectionId方法 構建一個Client.ConnectionId對象.
ConnectionId : 這個類 持有 請求地址 和 用戶的ticketclient 連接 server 的唯一憑證 : [remoteAddress, protocol, ticket]
/**
* Returns a ConnectionId object.
* @param addr Remote address for the connection.
* @param protocol Protocol for RPC.
* @param ticket UGI
* @param rpcTimeout timeout
* @param conf Configuration object
* @return A ConnectionId instance
* @throws IOException
*/
static ConnectionId getConnectionId(InetSocketAddress addr,
Class<?> protocol, UserGroupInformation ticket, int rpcTimeout,
RetryPolicy connectionRetryPolicy, Configuration conf) throws IOException {
//構建重試策略
if (connectionRetryPolicy == null) {
//設置最大重試次數 默認值: 10
final int max = conf.getInt(
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_MAX_RETRIES_KEY,
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_MAX_RETRIES_DEFAULT);
// 設置重試間隔: 1 秒
final int retryInterval = conf.getInt(
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_RETRY_INTERVAL_KEY,
CommonConfigurationKeysPublic
.IPC_CLIENT_CONNECT_RETRY_INTERVAL_DEFAULT);
//創建重試策略實例 RetryUpToMaximumCountWithFixedSleep
// 重試10次, 每次間隔1秒
connectionRetryPolicy = RetryPolicies.retryUpToMaximumCountWithFixedSleep(
max, retryInterval, TimeUnit.MILLISECONDS);
}
//創建ConnectionId :
// 這個類 持有 請求地址 和 用戶的ticket
// client 連接 server 的唯一憑證 : [remoteAddress, protocol, ticket]
return new ConnectionId(addr, protocol, ticket, rpcTimeout,
connectionRetryPolicy, conf);
}
在getConnectionId這個方法裏面會幹兩個事, 創一個重試策略 [RetryUpToMaximumCountWithFixedSleep]. 然後構建一個ConnectionId對象.
ConnectionId(InetSocketAddress address, Class<?> protocol,
UserGroupInformation ticket, int rpcTimeout,
RetryPolicy connectionRetryPolicy, Configuration conf) {
// 協議
this.protocol = protocol;
// 請求地址
this.address = address;
//用戶 ticket
this.ticket = ticket;
//設置超時時間
this.rpcTimeout = rpcTimeout;
//設置重試策略 默認: 重試10次, 每次間隔1秒
this.connectionRetryPolicy = connectionRetryPolicy;
// 單位 10秒
this.maxIdleTime = conf.getInt(
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECTION_MAXIDLETIME_KEY,
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECTION_MAXIDLETIME_DEFAULT);
// sasl client最大重試次數 5 次
this.maxRetriesOnSasl = conf.getInt(
CommonConfigurationKeys.IPC_CLIENT_CONNECT_MAX_RETRIES_ON_SASL_KEY,
CommonConfigurationKeys.IPC_CLIENT_CONNECT_MAX_RETRIES_ON_SASL_DEFAULT);
//指示客戶端將在套接字超時時進行重試的次數，以建立服務器連接。 默認值: 45
this.maxRetriesOnSocketTimeouts = conf.getInt(
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_MAX_RETRIES_ON_SOCKET_TIMEOUTS_KEY,
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_MAX_RETRIES_ON_SOCKET_TIMEOUTS_DEFAULT);
//使用TCP_NODELAY標誌繞過Nagle的算法傳輸延遲。 默認值: true
this.tcpNoDelay = conf.getBoolean(
CommonConfigurationKeysPublic.IPC_CLIENT_TCPNODELAY_KEY,
CommonConfigurationKeysPublic.IPC_CLIENT_TCPNODELAY_DEFAULT);
// 從客戶端啓用低延遲連接 默認 false
this.tcpLowLatency = conf.getBoolean(
CommonConfigurationKeysPublic.IPC_CLIENT_LOW_LATENCY,
CommonConfigurationKeysPublic.IPC_CLIENT_LOW_LATENCY_DEFAULT
);
// 啓用從RPC客戶端到服務器的ping操作 默認值: true
this.doPing = conf.getBoolean(
CommonConfigurationKeys.IPC_CLIENT_PING_KEY,
CommonConfigurationKeys.IPC_CLIENT_PING_DEFAULT);
// 設置ping 操作的間隔, 默認值 : 1分鐘
this.pingInterval = (doPing ? Client.getPingInterval(conf) : 0);
this.conf = conf;
}
回到之前的調用Invoker 另一個構造方法,但是入參會變.
/**
* This constructor takes a connectionId, instead of creating a new one.
*/
private Invoker(Class<?> protocol, Client.ConnectionId connId,
Configuration conf, SocketFactory factory) {
this.remoteId = connId;
this.client = CLIENTS.getClient(conf, factory, RpcWritable.Buffer.class);
this.protocolName = RPC.getProtocolName(protocol);
this.clientProtocolVersion = RPC
.getProtocolVersion(protocol);
}
這個是Invoker真正的構建方法,這裏面會將剛剛構建好的ConnectionId 賦值給remoteId 字段.
並且創建一個Client 對象.
// 獲取/創建 客戶端
this.client = CLIENTS.getClient(conf, factory, RpcWritable.Buffer.class);
我們看下getClient 方法. 這裏面會先嚐試從緩存中獲取client對象, 如果沒有的話,在自己創建一個,並且加到緩存中.
爲什麼會放到緩存中呢??
當client和server再次通訊的時候,可以複用這個client .
/**
* 如果沒有緩存的client存在的話
* 根據用戶提供的SocketFactory 構造 或者 緩存一個IPC 客戶端
*
* Construct & cache an IPC client with the user-provided SocketFactory
* if no cached client exists.
*
* @param conf Configuration
* @param factory SocketFactory for client socket
* @param valueClass Class of the expected response
* @return an IPC client
*/
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class<? extends Writable> valueClass) {
// Construct & cache client.
//
// The configuration is only used for timeout,
// and Clients have connection pools. So we can either
// (a) lose some connection pooling and leak sockets, or
// (b) use the same timeout for all configurations.
//
// Since the IPC is usually intended globally, notper-job, we choose (a).
//從緩存中獲取Client
Client client = clients.get(factory);
if (client == null) {
//client在緩存中不存在, 創建一個.
client = new Client(valueClass, conf, factory);
//緩存創建的client
clients.put(factory, client);
} else {
//client的引用計數+1
client.incCount();
}
if (Client.LOG.isDebugEnabled()) {
Client.LOG.debug("getting client out of cache: " + client);
}
// 返回client
return client;
}
到這裏 Invoker 對象就創建完了.
回到 ProtobufRpcEngine 的getProxy 方法 .
//然後調用Proxy.newProxylnstance()獲取動態代理對象，並通過ProtocolProxy返回
return new ProtocolProxy<T>(protocol, (T) Proxy.newProxyInstance(
protocol.getClassLoader(), new Class[]{protocol}, invoker), false);
構建一個ProtocolProxy 對象返回
/**
* Constructor
*
* @param protocol protocol class
* @param proxy its proxy
* @param supportServerMethodCheck If false proxy will never fetch server
* methods and isMethodSupported will always return true. If true,
* server methods will be fetched for the first call to
* isMethodSupported.
*/
public ProtocolProxy(Class<T> protocol, T proxy,
boolean supportServerMethodCheck) {
this.protocol = protocol;
this.proxy = proxy;
this.supportServerMethodCheck = supportServerMethodCheck;
}
然後我們看Client端的第四步 , 根據proto協議,構建一個請求對象.
這個沒啥可說的.是proto自動生成的,我們只是創建了一下而已.
//4. 構建發送請求對象
CustomProtos.GetMetaInfoRequestProto obj = CustomProtos.GetMetaInfoRequestProto.newBuilder().setPath("/meta").build();
然後就是Client端的第5步了將請求對象傳入, 獲取響應信息
//5. 將請求對象傳入, 獲取響應信息
CustomProtos.GetMetaInfoResponseProto metaData = proxy.getMetaInfo(null, obj);
Client端的最後一步輸出響應信息.
//6. 輸出數據
System.out.println(metaData.getInfo());
------------------華麗的分割線-------------------------------------------------------------------
咦,到這裏有點懵, 請求server端的代碼呢??? 請求怎麼發出去的??? 怎麼拿到響應信息的呢????
嗯嗯,是動態代理. ProtobufRpcEngine 的getProxy 方法 .
return new ProtocolProxy<T>(protocol, (T) Proxy.newProxyInstance(
protocol.getClassLoader(), new Class[]{protocol}, invoker), false);
主要是這個
(T) Proxy.newProxyInstance(
protocol.getClassLoader(), new Class[]{protocol}, invoker)
public static Object newProxyInstance(ClassLoader loader,
Class<?>[] interfaces,
InvocationHandler h) {........................
}
這個方法的作用就是創建一個代理類對象，
它接收三個參數，我們來看下幾個參數的含義：
loader ： 一個classloader對象，定義了由哪個classloader對象對生成的代理類進行加載
interfaces： 一個interface對象數組，表示我們將要給我們的代理對象提供一組什麼樣的接口，如果我們提供了這樣一個接口對象數組，那麼也就是聲明瞭代理類實現了這些接口，代理類就可以調用接口中聲明的所有方法。
h： 一個InvocationHandler對象，表示的是當動態代理對象調用方法的時候會關聯到哪一個InvocationHandler對象上，並最終由其調用。
我們直接看 ProtobufRpcEngine#Invoker中的 invoke方法
/**
*
* ProtobufRpcEngine.Invoker.invoker() 方法主要做了三件事情:
* 1.構造請求頭域，
* 使用protobuf將請求頭序列化，這個請求頭域 記錄了當前RPC調用是什麼接口的什麼方法上的調用;
* 2.通過RPC.Client類發送請求頭以 及序列化好的請求參數。
* 請求參數是在ClientNamenodeProtocolPB調用時就已經序列化好的，
* 調用Client.call()方法時，
* 需要將請求頭以及請求參數使用一個RpcRequestWrapper對象封裝;
* 3.獲取響應信息，序列化響應信息並返回。
*
*
* This is the client side invoker of RPC method. It only throws
* ServiceException, since the invocation proxy expects only
* ServiceException to be thrown by the method in case protobuf service.
*
* ServiceException has the following causes:
* <ol>
* <li>Exceptions encountered on the client side in this method are
* set as cause in ServiceException as is.</li>
* <li>Exceptions from the server are wrapped in RemoteException and are
* set as cause in ServiceException</li>
* </ol>
*
* Note that the client calling protobuf RPC methods, must handle
* ServiceException by getting the cause from the ServiceException. If the
* cause is RemoteException, then unwrap it to get the exception thrown by
* the server.
*/
@Override
public Message invoke(Object proxy, final Method method, Object[] args)
throws ServiceException {
long startTime = 0;
if (LOG.isDebugEnabled()) {
startTime = Time.now();
}
// pb接口的參數只有兩個，即RpcController + Message
if (args.length != 2) { // RpcController + Message
throw new ServiceException(
"Too many or few parameters for request. Method: ["
+ method.getName() + "]" + ", Expected: 2, Actual: "
+ args.length);
}
if (args[1] == null) {
throw new ServiceException("null param while calling Method: ["
+ method.getName() + "]");
}
// if Tracing is on then start a new span for this rpc.
// guard it in the if statement to make sure there isn't
// any extra string manipulation.
// todo 這個是啥
Tracer tracer = Tracer.curThreadTracer();
TraceScope traceScope = null;
if (tracer != null) {
traceScope = tracer.newScope(RpcClientUtil.methodToTraceString(method));
}
//構造請求頭域，標明在什麼接口上調用什麼方法
RequestHeaderProto rpcRequestHeader = constructRpcRequestHeader(method);
if (LOG.isTraceEnabled()) {
LOG.trace(Thread.currentThread().getId() + ": Call -> " +
remoteId + ": " + method.getName() +
" {" + TextFormat.shortDebugString((Message) args[1]) + "}");
}
//獲取請求調用的參數，例如RenameRequestProto
final Message theRequest = (Message) args[1];
final RpcWritable.Buffer val;
try {
//調用RPC.Client發送請求
val = (RpcWritable.Buffer) client.call(RPC.RpcKind.RPC_PROTOCOL_BUFFER,
new RpcProtobufRequest(rpcRequestHeader, theRequest), remoteId,
fallbackToSimpleAuth, alignmentContext);
} catch (Throwable e) {
if (LOG.isTraceEnabled()) {
LOG.trace(Thread.currentThread().getId() + ": Exception <- " +
remoteId + ": " + method.getName() +
" {" + e + "}");
}
if (traceScope != null) {
traceScope.addTimelineAnnotation("Call got exception: " +
e.toString());
}
throw new ServiceException(e);
} finally {
if (traceScope != null) traceScope.close();
}
if (LOG.isDebugEnabled()) {
long callTime = Time.now() - startTime;
LOG.debug("Call: " + method.getName() + " took " + callTime + "ms");
}
if (Client.isAsynchronousMode()) {
final AsyncGet<RpcWritable.Buffer, IOException> arr
= Client.getAsyncRpcResponse();
final AsyncGet<Message, Exception> asyncGet
= new AsyncGet<Message, Exception>() {
@Override
public Message get(long timeout, TimeUnit unit) throws Exception {
return getReturnMessage(method, arr.get(timeout, unit));
}
@Override
public boolean isDone() {
return arr.isDone();
}
};
ASYNC_RETURN_MESSAGE.set(asyncGet);
return null;
} else {
return getReturnMessage(method, val);
}
}
艾瑪,這個有點長啊.
挑幾點重要的說.
- 構造請求頭域，標明在什麼接口上調用什麼方法
- 獲取請求調用的參數
- 調用RPC.Client發送請求
- 獲取響應信息
下面分別來說:
1.構造請求頭域，標明在什麼接口上調用什麼方法
//構造請求頭域，標明在什麼接口上調用什麼方法
RequestHeaderProto rpcRequestHeader = constructRpcRequestHeader(method);
這裏很簡單,就是根據協議定義,將 協議名稱, 調用的方法名稱, 版本號, 三個值傳入,構建一個消息頭.
private RequestHeaderProto constructRpcRequestHeader(Method method) {
RequestHeaderProto.Builder builder = RequestHeaderProto
.newBuilder();
builder.setMethodName(method.getName());
builder.setDeclaringClassProtocolName(protocolName);
builder.setClientProtocolVersion(clientProtocolVersion);
return builder.build();
}
2.獲取請求調用的參數
獲取請求調用的參數，這個是才client端代碼就創建好的,通過參數傳進來的.
// 獲取請求調用的參數，這個是才client端代碼就創建好的,通過參數傳進來的.
// 例如 GetMetaInfoRequestProto
final Message theRequest = (Message) args[1];
3.調用RPC.Client發送請求
這個是核心,畢竟是發送請求,我們來重點關注一下.
//調用RPC.Client發送請求
val = (RpcWritable.Buffer) client.call(RPC.RpcKind.RPC_PROTOCOL_BUFFER,
new RpcProtobufRequest(rpcRequestHeader, theRequest), remoteId,
fallbackToSimpleAuth, alignmentContext);
Client這個我就不說了,ProtobufRpcEngine#Invoker構造方法裏面創建的.
/**
* This constructor takes a connectionId, instead of creating a new one.
*/
private Invoker(Class<?> protocol, Client.ConnectionId connId,
Configuration conf, SocketFactory factory) {
// 設置ConnectionId , ConnectionId裏面保存着Client連接Server的信息
this.remoteId = connId;
// 獲取/創建 客戶端
this.client = CLIENTS.getClient(conf, factory, RpcWritable.Buffer.class);
// 獲取協議的名稱
this.protocolName = RPC.getProtocolName(protocol);
// 獲取協議的版本 version
this.clientProtocolVersion = RPC
.getProtocolVersion(protocol);
}
我們直接看Client#call 方法.
方法有點長. 摘取主要的說.
3.1.創建 Call 對象
3.2.獲取&建立連接
3.3.發送RPC請求
3.4.獲取響應
/**
*
* Make a call, passing <code>rpcRequest</code>, to the IPC server defined by
* <code>remoteId</code>, returning the rpc response.
*
* @param rpcKind
* @param rpcRequest - contains serialized method and method parameters
* @param remoteId - the target rpc server
* @param serviceClass - service class for RPC
* @param fallbackToSimpleAuth - set to true or false during this method to
* indicate if a secure client falls back to simple auth
* @param alignmentContext - state alignment context
* @return the rpc response
* Throws exceptions if there are network problems or if the remote code
* threw an exception.
*/
Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
ConnectionId remoteId, int serviceClass,
AtomicBoolean fallbackToSimpleAuth, AlignmentContext alignmentContext)
throws IOException {
/**
* 創建 Call 對象
* Client.call()方法將RPC請求封裝成一個Call對象，
* Call對象中保存了RPC調用的完 成標誌、返回值信息以及異常信息;
* 隨後，Client.cal()方法會創建一個 Connection對象，
* Connection對象用於管理Client與Server的Socket連接。
*/
final Call call = createCall(rpcKind, rpcRequest);
call.setAlignmentContext(alignmentContext);
//獲取&建立連接
//用ConnectionId作爲key，
// 將新建的Connection對象放入Client.connections字段中保 存
// (
// 對於Connection對象，
// 由於涉及了與Server建立Socket連接，會比較耗費資 源，
// 所以Client類使用一個HashTable對象connections保存那些沒有過期的 Connection，
// 如果可以複用，則複用這些Connection對象
// );
// 以callId作爲key，將 構造的Call對象放入Connection.calls字段中保存。
final Connection connection = getConnection(remoteId, call, serviceClass,
fallbackToSimpleAuth);
try {
//檢測是否是異步請求.
checkAsyncCall();
try {
//發送RPC請求
connection.sendRpcRequest(call); // send the rpc request
} catch (RejectedExecutionException e) {
throw new IOException("connection has been closed", e);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
IOException ioe = new InterruptedIOException(
"Interrupted waiting to send RPC request to server");
ioe.initCause(ie);
throw ioe;
}
} catch(Exception e) {
if (isAsynchronousMode()) {
releaseAsyncCall();
}
throw e;
}
if (isAsynchronousMode()) {
final AsyncGet<Writable, IOException> asyncGet
= new AsyncGet<Writable, IOException>() {
@Override
public Writable get(long timeout, TimeUnit unit)
throws IOException, TimeoutException{
boolean done = true;
try {
final Writable w = getRpcResponse(call, connection, timeout, unit);
if (w == null) {
done = false;
throw new TimeoutException(call + " timed out "
+ timeout + " " + unit);
}
return w;
} finally {
if (done) {
releaseAsyncCall();
}
}
}
@Override
public boolean isDone() {
synchronized (call) {
return call.done;
}
}
};
ASYNC_RPC_RESPONSE.set(asyncGet);
return null;
} else {
//服務器成功發回響應信息，返回RPC響應
return getRpcResponse(call, connection, -1, null);
}
}
3.1.創建 Call 對象
/**
* 創建 Call 對象
* Client.call()方法將RPC請求封裝成一個Call對象，
* Call對象中保存了RPC調用的完 成標誌、返回值信息以及異常信息;
* 隨後，Client.cal()方法會創建一個 Connection對象，
* Connection對象用於管理Client與Server的Socket連接。
*/
final Call call = createCall(rpcKind, rpcRequest);
/**
*
* @param rpcKind rpcKind參數用於描述RPC請求的序列化工具類型
* @param rpcRequest rpcRequest參數則用於記錄序列化後的RPC請求
* @return
*/
Call createCall(RPC.RpcKind rpcKind, Writable rpcRequest) {
return new Call(rpcKind, rpcRequest);
}
這裏就是構造方法, 生成一個唯一的callId ,進行初始化 .
private Call(RPC.RpcKind rpcKind, Writable param) {
this.rpcKind = rpcKind;
this.rpcRequest = param;
//獲取 callId
final Integer id = callId.get();
if (id == null) {
// AtomicInteger callIdCounter 自增
this.id = nextCallId();
} else {
callId.set(null);
this.id = id;
}
final Integer rc = retryCount.get();
if (rc == null) {
this.retry = 0;
} else {
this.retry = rc;
}
// 設置異常處理類
this.externalHandler = EXTERNAL_CALL_HANDLER.get();
}
3.2.獲取&建立連接
用ConnectionId作爲key，
將新建的Connection對象放入Client.connections字段中保 存
(
對於Connection對象，
由於涉及了與Server建立Socket連接，會比較耗費資 源，
所以Client類使用一個ConcurrentMap對象connections保存那些沒有過期的 Connection，
如果可以複用，則複用這些Connection對象
);
以callId作爲key，將 構造的Call對象放入Connection.calls字段中保存。
final Connection connection = getConnection(remoteId, call, serviceClass,
fallbackToSimpleAuth);
/** Get a connection from the pool, or create a new one and add it to the
* pool. Connections to a given ConnectionId are reused.
* 獲取連接
*
* */
private Connection getConnection(ConnectionId remoteId,
Call call, int serviceClass, AtomicBoolean fallbackToSimpleAuth)
throws IOException {
if (!running.get()) {
// the client is stopped
throw new IOException("The client is stopped");
}
Connection connection;
/* we could avoid this allocation for each RPC by having a
* connectionsId object and with set() method. We need to manage the
* refs for keys in HashMap properly. For now its ok.
*/
while (true) {
// These lines below can be shorten with computeIfAbsent in Java8
//首先嚐試從Client.connections隊列中獲取Connection對象
connection = connections.get(remoteId);
if (connection == null) {
//如果connections隊列中沒有保存，則構造新的對象
connection = new Connection(remoteId, serviceClass);
// putIfAbsent 如果對應的 key 已經有值了,
// 則忽略本次操作,直接返回舊值
Connection existing = connections.putIfAbsent(remoteId, connection);
if (existing != null) {
connection = existing;
}
}
//將待發送請求對應的Call對象放入Connection.calls隊列
if (connection.addCall(call)) {
break;
} else {
// This connection is closed, should be removed. But other thread could
// have already known this closedConnection, and replace it with a new
// connection. So we should call conditional remove to make sure we only
// remove this closedConnection.
connections.remove(remoteId, connection);
}
}
// If the server happens to be slow, the method below will take longer to
// establish a connection.
//調用setupIOstreams()方法，初始化Connection對象並獲取IO流
connection.setupIOstreams(fallbackToSimpleAuth);
return connection;
}
裏面創建connection就不細說了, 其實就是創建connection對象,然後做一下請求地址和請求參數的設置. 並沒有和server端進行請求.
//如果connections隊列中沒有保存，則構造新的對象
connection = new Connection(remoteId, serviceClass);
connection.setupIOstreams 這個纔是建立連接的關鍵方法.
//調用setupIOstreams()方法，初始化Connection對象並獲取IO流
connection.setupIOstreams(fallbackToSimpleAuth);
/**
*
*
* Connect to the server and set up the I/O streams. It then sends
* a header to the server and starts
* the connection thread that waits for responses.
*
* Client.call()方法調用 Connection.setupIOstreams() 方法建立與Server的Socket連接。
* setupIOstreams() 方法還會啓動Connection線程，
* Connection線程會監聽Socket並讀取Server發回的響應信息。
*
*/
private synchronized void setupIOstreams( AtomicBoolean fallbackToSimpleAuth) {
if (socket != null || shouldCloseConnection.get()) {
return;
}
UserGroupInformation ticket = remoteId.getTicket();
if (ticket != null) {
final UserGroupInformation realUser = ticket.getRealUser();
if (realUser != null) {
ticket = realUser;
}
}
try {
connectingThread.set(Thread.currentThread());
if (LOG.isDebugEnabled()) {
LOG.debug("Connecting to "+server);
}
Span span = Tracer.getCurrentSpan();
if (span != null) {
span.addTimelineAnnotation("IPC client connecting to " + server);
}
short numRetries = 0;
Random rand = null;
while (true) {
//建立到Server的Socket連接，
// 並且在這個Socket連接上 獲得InputStream和OutputStream對象。
setupConnection(ticket);
ipcStreams = new IpcStreams(socket, maxResponseLength);
// 調用writeConnectionHeader()方法在連接建立時發送連接頭域。
writeConnectionHeader(ipcStreams);
if (authProtocol == AuthProtocol.SASL) {
try {
authMethod = ticket
.doAs(new PrivilegedExceptionAction<AuthMethod>() {
@Override
public AuthMethod run()
throws IOException, InterruptedException {
return setupSaslConnection(ipcStreams);
}
});
} catch (IOException ex) {
if (saslRpcClient == null) {
// whatever happened -it can't be handled, so rethrow
throw ex;
}
// otherwise, assume a connection problem
authMethod = saslRpcClient.getAuthMethod();
if (rand == null) {
rand = new Random();
}
handleSaslConnectionFailure(numRetries++, maxRetriesOnSasl, ex,
rand, ticket);
continue;
}
if (authMethod != AuthMethod.SIMPLE) {
// Sasl connect is successful. Let's set up Sasl i/o streams.
ipcStreams.setSaslClient(saslRpcClient);
// for testing
remoteId.saslQop =
(String)saslRpcClient.getNegotiatedProperty(Sasl.QOP);
LOG.debug("Negotiated QOP is :" + remoteId.saslQop);
if (fallbackToSimpleAuth != null) {
fallbackToSimpleAuth.set(false);
}
} else if (UserGroupInformation.isSecurityEnabled()) {
if (!fallbackAllowed) {
throw new IOException("Server asks us to fall back to SIMPLE " +
"auth, but this client is configured to only allow secure " +
"connections.");
}
if (fallbackToSimpleAuth != null) {
fallbackToSimpleAuth.set(true);
}
}
}
if (doPing) {
ipcStreams.setInputStream(new PingInputStream(ipcStreams.in));
}
//調用writeConnectionContext()方法寫入連接上下文。
writeConnectionContext(remoteId, authMethod);
// update last activity time
//調用touch()方法更新上次活躍時間。
touch();
span = Tracer.getCurrentSpan();
if (span != null) {
span.addTimelineAnnotation("IPC client connected to " + server);
}
// 開啓接收線程
// 調用start()方法啓動Connection線程監聽並接收Server發回的響應信息。
// start the receiver thread after the socket connection has been set up
start();
return;
}
} catch (Throwable t) {
if (t instanceof IOException) {
markClosed((IOException)t);
} else {
markClosed(new IOException("Couldn't set up IO streams: " + t, t));
}
close();
} finally {
connectingThread.set(null);
}
}
在這個方法中通過setupConnection(ticket); 方法與server端建立連接.
//建立到Server的Socket連接，
// 並且在這個Socket連接上 獲得InputStream和OutputStream對象。
setupConnection(ticket);
調用writeConnectionHeader()方法在連接建立時發送連接頭信息。
writeConnectionHeader(ipcStreams);
/* Write the connection context header for each connection
* Out is not synchronized because only the first thread does this.
*/
private void writeConnectionContext(ConnectionId remoteId,
AuthMethod authMethod)
throws IOException {
// Write out the ConnectionHeader
IpcConnectionContextProto message = ProtoUtil.makeIpcConnectionContext(
RPC.getProtocolName(remoteId.getProtocol()),
remoteId.getTicket(),
authMethod);
RpcRequestHeaderProto connectionContextHeader = ProtoUtil
.makeRpcRequestHeader(RpcKind.RPC_PROTOCOL_BUFFER,
OperationProto.RPC_FINAL_PACKET, CONNECTION_CONTEXT_CALL_ID,
RpcConstants.INVALID_RETRY_COUNT, clientId);
// do not flush. the context and first ipc call request must be sent
// together to avoid possibility of broken pipes upon authz failure.
// see writeConnectionHeader
final ResponseBuffer buf = new ResponseBuffer();
connectionContextHeader.writeDelimitedTo(buf);
message.writeDelimitedTo(buf);
synchronized (ipcStreams.out) {
ipcStreams.sendRequest(buf.toByteArray());
}
}
開啓接收線程 調用start()方法啓動Connection線程監聽並接收Server發回的響應信息。
// 開啓接收線程
// 調用start()方法啓動Connection線程監聽並接收Server發回的響應信息。
// start the receiver thread after the socket connection has been set up
start();
在這裏會調用Connection裏面的run()方法. 接收並處理返回消息.
@Override
public void run() {
if (LOG.isDebugEnabled())
LOG.debug(getName() + ": starting, having connections "
+ connections.size());
try {
while (waitForWork()) {//wait here for work - read or close connection
//接收到返回信息
receiveRpcResponse();
}
} catch (Throwable t) {
// This truly is unexpected, since we catch IOException in receiveResponse
// -- this is only to be really sure that we don't leave a client hanging
// forever.
LOG.warn("Unexpected error reading responses on connection " + this, t);
markClosed(new IOException("Error reading responses", t));
}
close();
if (LOG.isDebugEnabled())
LOG.debug(getName() + ": stopped, remaining connections "
+ connections.size());
}
/**
接收到返回信息
Receive a response.
* Because only one receiver, so no synchronization on in.
* receiveRpcResponse()方法接收RPC響應。
* receiveRpcResponse()方法 會從輸入流中讀取序列化對象RpcResponseHeaderProto，
* 然後根據RpcResponseHeaderProto 中記錄的callid字段獲取對應的Call的對象。
*
* 接下來receiveRpcResponse()方法會從輸入流中 讀取響應消息，
* 然後調用Call.setRpcResponse()將響應消息保存在Call對象中。
* 如果服務器 在處理RPC請求時拋出異常，
* 則receiveRpcResponse()會從輸入流中讀取異常信息，並構造 異常對象，
* 然後調用Call.setException()將異常保存在Call對象中。
*
*/
private void receiveRpcResponse() {
if (shouldCloseConnection.get()) {
return;
}
//更新請求時間的
touch();
try {
//通過ipcStreams 獲取響應對象
ByteBuffer bb = ipcStreams.readResponse();
//將響應對象,採用RpcWritable 進行包裝.
RpcWritable.Buffer packet = RpcWritable.Buffer.wrap(bb);
// 獲取響應的響應頭.
RpcResponseHeaderProto header = packet.getValue(RpcResponseHeaderProto.getDefaultInstance());
// 檢測頭信息.
checkResponse(header);
// 獲取call唯一標識callId
int callId = header.getCallId();
if (LOG.isDebugEnabled())
LOG.debug(getName() + " got value #" + callId);
// 獲取頭信息裏面的響應狀態.
RpcStatusProto status = header.getStatus();
//如果調用成功，則讀取響應消息，在call實例中設置
if (status == RpcStatusProto.SUCCESS) {
//構建實例
Writable value = packet.newInstance(valueClass, conf);
//將Call 從 正在執行的calls緩存隊列中移除.
final Call call = calls.remove(callId);
//將Call 設置請求信息
call.setRpcResponse(value);
if (call.alignmentContext != null) {
call.alignmentContext.receiveResponseState(header);
}
}
// verify that packet length was correct
if (packet.remaining() > 0) {
throw new RpcClientException("RPC response length mismatch");
}
//RPC調用失敗
if (status != RpcStatusProto.SUCCESS) { // Rpc Request failed
//取出響應中的異常消息
final String exceptionClassName = header.hasExceptionClassName() ?
header.getExceptionClassName() :
"ServerDidNotSetExceptionClassName";
final String errorMsg = header.hasErrorMsg() ?
header.getErrorMsg() : "ServerDidNotSetErrorMsg" ;
final RpcErrorCodeProto erCode =
(header.hasErrorDetail() ? header.getErrorDetail() : null);
if (erCode == null) {
LOG.warn("Detailed error code not set by server on rpc error");
}
RemoteException re = new RemoteException(exceptionClassName, errorMsg, erCode);
//在Call對象中設置異常
if (status == RpcStatusProto.ERROR) {
final Call call = calls.remove(callId);
call.setException(re);
} else if (status == RpcStatusProto.FATAL) {
// Close the connection
markClosed(re);
}
}
} catch (IOException e) {
markClosed(e);
}
}
3.3.發送RPC請求
先構造RPC請求頭. 將請求頭和RPC請求 ,在 另外一個線程池 sendParamsExecutor的run方法中使用ipcStreams.sendRequest(buf.toByteArray()); 發送出去. 裏面寫的太麻煩了, 自己看吧.
//發送RPC請求
connection.sendRpcRequest(call); // send the rpc request
/**
*
* 發送RPC請求到Server。
*
* Initiates a rpc call by sending the rpc request to the remote server.
* Note: this is not called from the Connection thread, but by other
* threads.
* @param call - the rpc request
*
* RPC發送請求線程會調用Connection.sendRpcRequest()方法發送RPC請求到Server，
* 這 裏要特別注意，這個方法不是由Connection線程調用的，
* 而是由發起RPC請求的線程調用 的。
*
*/
public void sendRpcRequest(final Call call)
throws InterruptedException, IOException {
if (shouldCloseConnection.get()) {
return;
}
// Serialize the call to be sent. This is done from the actual
// caller thread, rather than the sendParamsExecutor thread,
// so that if the serialization throws an error, it is reported
// properly. This also parallelizes the serialization.
//
// Format of a call on the wire:
// 0) Length of rest below (1 + 2)
// 1) RpcRequestHeader - is serialized Delimited hence contains length
// 2) RpcRequest
//
// Items '1' and '2' are prepared here.
//先構造RPC請求頭
RpcRequestHeaderProto header = ProtoUtil.makeRpcRequestHeader(
call.rpcKind, OperationProto.RPC_FINAL_PACKET, call.id, call.retry,
clientId, call.alignmentContext);
final ResponseBuffer buf = new ResponseBuffer();
//將RPC請求頭寫入 輸出流 ResponseBuffer
header.writeDelimitedTo(buf);
//將RPC請求(包括請求元數據和請求參數)封裝成ProtobufWrapper ==> 寫入輸出流
RpcWritable.wrap(call.rpcRequest).writeTo(buf);
//這裏使用 線程池 將請求發送出去，
// 請求包括三個部分:
// 1 長度;
// 2 RPC請求頭;
// 3 RPC請求(包括 請求元數據以及請求參數)
synchronized (sendRpcRequestLock) {
Future<?> senderFuture = sendParamsExecutor.submit(new Runnable() {
@Override
public void run() {
try {
synchronized (ipcStreams.out) {
if (shouldCloseConnection.get()) {
return;
}
if (LOG.isDebugEnabled()) {
LOG.debug(getName() + " sending #" + call.id
+ " " + call.rpcRequest);
}
// RpcRequestHeader + RpcRequest
ipcStreams.sendRequest(buf.toByteArray());
ipcStreams.flush();
}
} catch (IOException e) {
// exception at this point would leave the connection in an
// unrecoverable state (eg half a call left on the wire).
// So, close the connection, killing any outstanding calls
//如果發生發送異常，則直接關閉連接
markClosed(e);
} finally {
//the buffer is just an in-memory buffer, but it is still polite to
// close early
//之前申請的buffer給關閉了，比較優雅
IOUtils.closeStream(buf);
}
}
});
try {
senderFuture.get();
} catch (ExecutionException e) {
Throwable cause = e.getCause();
// cause should only be a RuntimeException as the Runnable above
// catches IOException
if (cause instanceof RuntimeException) {
throw (RuntimeException) cause;
} else {
throw new RuntimeException("unexpected checked exception", cause);
}
}
}
}
3.4.獲取響應
//服務器成功發回響應信息，返回RPC響應
return getRpcResponse(call, connection, -1, null);
在這裏,會不斷輪訓call的狀態, 如果狀態爲done則代表數據已經處理完,並且已經獲取到響應信息.
響應信息由connection中的run方法進行處理.
/** @return the rpc response or, in case of timeout, null. */
private Writable getRpcResponse(final Call call, final Connection connection,
final long timeout, final TimeUnit unit) throws IOException {
synchronized (call) {
while (!call.done) {
try {
//等待RPC響應
AsyncGet.Util.wait(call, timeout, unit);
if (timeout >= 0 && !call.done) {
return null;
}
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
throw new InterruptedIOException("Call interrupted");
}
}
if (call.error != null) {
if (call.error instanceof RemoteException) {
call.error.fillInStackTrace();
throw call.error;
} else { // local exception
InetSocketAddress address = connection.getRemoteAddress();
throw NetUtils.wrapException(address.getHostName(),
address.getPort(),
NetUtils.getHostname(),
0,
call.error);
}
} else {
return call.getRpcResponse();
}
}
4. 處理Client 端的響應信息
Client 獲取響應信息, 不過是同步還是異步獲取響應信息,都會調用這個方法: getReturnMessage(method, val);
getReturnMessage(method, val);
private Message getReturnMessage(final Method method,
final RpcWritable.Buffer buf) throws ServiceException {
Message prototype = null;
try {
//獲取返回參數
prototype = getReturnProtoType(method);
} catch (Exception e) {
throw new ServiceException(e);
}
Message returnMessage;
try {
// 序列化響應信息
returnMessage = buf.getValue(prototype.getDefaultInstanceForType());
if (LOG.isTraceEnabled()) {
LOG.trace(Thread.currentThread().getId() + ": Response <- " +
remoteId + ": " + method.getName() +
" {" + TextFormat.shortDebugString(returnMessage) + "}");
}
} catch (Throwable e) {
throw new ServiceException(e);
}
//返回結果
return returnMessage;
}
最後我們看client端最後的步驟. 輸出返回的信息.
