11148893 : 4月16號09:37:49.679[finagle/netty3-2][ERROR]: $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$[****]$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ 服務提供者,提供的服務名:[trade-*****-serv] 被調用失敗 !!!! $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$[LOOM]$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
java.util.concurrent.CancellationException
at com.twitter.util.ExecutorServiceFuturePool$$anonfun$apply$1.applyOrElse(FuturePool.scala:139)
at com.twitter.util.ExecutorServiceFuturePool$$anonfun$apply$1.applyOrElse(FuturePool.scala:136)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at com.twitter.util.Promise.raise(Promise.scala:441)
at com.twitter.util.Promise.raise(Promise.scala:446)
at com.twitter.util.Promise.raise(Promise.scala:446)
at com.twitter.util.Promise.raise(Promise.scala:446)
at com.twitter.util.Promise.raise(Promise.scala:446)
at com.twitter.util.Promise.raise(Promise.scala:446)
at com.twitter.util.Promise$Chained.raise(Promise.scala:194)
at com.twitter.util.Promise.raise(Promise.scala:446)
at com.twitter.util.Promise.raise(Promise.scala:446)
at com.twitter.util.Promise$Chained.raise(Promise.scala:194)
at com.twitter.util.Promise.raise(Promise.scala:446)
at com.twitter.util.Promise.raise(Promise.scala:446)
at com.twitter.util.Promise.raise(Promise.scala:446)
at com.twitter.util.Promise$Chained.raise(Promise.scala:194)
at com.twitter.finagle.dispatch.GenSerialServerDispatcher$$anonfun$1.apply$mcV$sp(ServerDispatcher.scala:68)
at com.twitter.util.Future$$anonfun$ensure$1.apply(Future.scala:625)
at com.twitter.util.Future$$anonfun$ensure$1.apply(Future.scala:625)
at com.twitter.util.Promise$Monitored.apply(Promise.scala:68)
at com.twitter.util.Promise$Monitored.apply(Promise.scala:59)
at com.twitter.util.Promise$$anon$2.run(Promise.scala:326)
at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:186)
at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:157)
at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:212)
at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:86)
at com.twitter.util.Promise.runq(Promise.scala:312)
at com.twitter.util.Promise.updateIfEmpty(Promise.scala:616)
at com.twitter.finagle.transport.ChannelTransport.com$twitter$finagle$transport$ChannelTransport$$fail(ChannelTransport.scala:34)
at com.twitter.finagle.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:50)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.SimpleChannelHandler.channelClosed(SimpleChannelHandler.java:216)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:106)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:493)
at org.jboss.netty.handler.codec.frame.FrameDecoder.channelClosed(FrameDecoder.java:371)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)
at com.twitter.finagle.thrift.ThriftFrameCodec.handleUpstream(ThriftFrameCodec.scala:17)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.bootstrap.NettyDispatcher.handleUpstream(SourceFile:35)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:375)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: com.twitter.finagle.CancelledRequestException: request cancelled
at com.twitter.finagle.NoStacktrace(Unknown Source)線上10.開頭的雲主機上的服務偶爾會報出這個錯誤，查詢官方FAQ，相關解釋如下：
What are CancelledRequestException and CancelledConnectionException?¶
When a client connected to a Finagle server disconnects, the server raises a cancellation interrupt on the pending Future. This is done to conserve resources and avoid unnecessary work: the upstream client may have timed the request out, for example. Interrupts on futures propagate, and so if that server is in turn waiting for a response from a downstream server it will cancel this pending request, and so on.
The topology, visually:
Upstream ---> (Finagle Server -> Finagle Client) ---> Downstream
Interrupts propagate between the Finagle Server and Client only if the Future returned from the Server is chained [1] to the Client.
A simplified code snippet that exemplifies the intra-process structure:
import com.twitter.finagle.Mysql
import com.twitter.finagle.mysql
import com.twitter.finagle.Service
import com.twitter.finagle.Http
import com.twitter.finagle.http
import com.twitter.util.Future
val client: Service[mysql.Request, mysql.Result] =
Mysql.client.newService(...)
val service: Service[http.Request, http.Response] =
new Service[http.Request, http.Response] {
def apply(req: http.Request): Future[http.Response] = {
client(...).map(req: mysql.Result => http.Response())
}
}
val server = Http.server.serve(..., service)
[1] “Chained” in this context means that calling Future#raise will reach the interrupt handler on the Future that represents the RPC call to the client. This is clearly the case in the above example where the call to the client is indeed the returned Future. However, this will still hold if the client call was in the context of a Future combinator (ex. Future#select, Future#join, etc.)
This is the source of the CancelledRequestException – when a Finagle client receives the cancellation interrupt while a request is pending, it fails that request with this exception. A special case of this is when a request is in the process of establishing a session and is instead interrupted with a CancelledConnectionException
You can disable this behavior by using the MaskCancelFilter:
import com.twitter.finagle.filter.MaskCancelFilter
import com.twitter.finagle.Http
import com.twitter.finagle.http.
val service: Service[http.Request, http.Response] =
Http.client.newService("http://twitter.com")
val masked = new MaskCancelFilter[http.Request, http.Response]
val maskedService = masked.andThen(service)
Note
Most protocols do not natively support request cancellations (though modern RPC protocols like Mux do). In practice, this means that for these protocols, we need to disconnect the client to signal cancellation, which in turn can cause undue connection churn.
什麼是請求取消異常和連接取消異常？
當一個連接到Finagle server的client斷開連接，server會提出對一個等待狀態的Future計算的中斷，這樣做是爲了節約資源和避免不必要的工作：例如上游的client可能已經請求超時了。Future計算的中斷在傳播，因此如果服務器在等待下游服務器的響應，它將取消這個掛起的請求，等等。
形象化的拓撲圖如下：
上游--》Finagle 服務器 --》Finagle 客戶端 --》下游
只有當Future結果被客戶端“鏈接”的服務器返回了，中斷纔會在在服務器和客戶端傳播。
A simplified code snippet that exemplifies the intra-process structure:
.......
.......
.......
CancelledRequestException的根源是當一個Finagle客戶端收到了取消的中斷結果，而與此同時客戶端的請求還處在等待狀態，它會用該異常失敗此次請求。這一個特殊的情況是當一個請求在建立會話的過程，會用一個cancelledconnectionexception中斷 。
您可以通過使用MaskCancelFilter禁用此行爲：
......
......
注：
大多數協議本身不支持請求的取消（雖然現代的RPC協議像Mux那樣）。實際上，這意味着對這些協議而言，我們需要切斷客戶端與取消中斷信號的聯繫，因爲中斷取消信號會導致不適當的連接衝突。