問題信息
問題描述
服務調用方同學反饋，XX應用調用我們對外提供的庫存查詢服務，找不到服務
···
錯誤信息
調用方輸出的錯誤日誌信息
[2019-01-24 16:46:43,985] INFO -:-:-:- [main] com.wwwarehouse.commons.dubbo.DubboConsumerAuthFilter.invoke(DubboConsumerAuthFilter.java:51) - dubboRpcContext:>>>
com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method compositeQuery in the service com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService. Tried 3 times of the providers [192.168.72.158:20880] (1/1) from the registry 192.168.6.21:2181 on the consumer 192.168.72.158 using the dubbo version 2.8.4. Last error is: Failed to invoke remote method: compositeQuery, provider: dubbo://192.168.72.158:20880/com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService?anyhost=true&application=commonindustry-biz-test&batchChangeInventory.timeout=60000&batchChangeInventoryQty.timeout=60000&changeInventory.timeout=600000&check=false&default.check=false&default.connections=10&default.group=mock&dubbo=2.8.4&eliminatePlanAdjustmentDatas.timeout=60000&generic=false&group=impl2&interface=com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService&lockPlanAdjustmentDatas.timeout=5000&methods=selectSubItemsByGroupUkidAndItemUkid,queryConsumeRealityQTY,queryDeliveryFromStorageQTY,queryMaxQtyGroupForItem,unLockPlanAdjustmentDatas,selectAvailableListByGroupRelationUkid,inventoryQuery,selectItemsByGroupUkidAndIdentifyCode,compositeQuery,lockPlanAdjustmentDatas,getInventoryQueryFilterInfo,batchChangeInventoryQty,listByUkids,updateInventoryStatistics,sumInventoryQty,changeInventoryQty,changeInventory,deliveryFromStorage,selectQualitiesByGroupUkidAndItemUkid,batchChangeInventory,getGoodsStockInfo,warehouseEntry,selectConsumeListByIdentifyCode,addInventory,selectConsumeListByGroupUkid,selectConsumeListByGroupUkids,getInventoryQueryList,getStorageGoodsStock,selectConsumeListByGroupCode,eliminatePlanAdjustmentDatas&organization=wwwarehouse&owner=xdw&pid=17332&protocol=dubbo&revision=1.0-20190124.081517-966&side=consumer×tamp=1548319602539&updateInventoryStatistics.timeout=5000, cause: com.alibaba.dubbo.remoting.RemotingException: Not found exported service: com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880 in [impl2/com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880], may be version or group mismatch , channel: consumer: /192.168.72.158:60792 --> provider: /192.168.72.158:20880, message:RpcInvocation [methodName=compositeQuery, parameterTypes=[class com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery], arguments=[null], attachments={path=com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService, input=829, name=OWNER_UKID, dubbo=2.8.4, version=0.0.0}]
com.alibaba.dubbo.remoting.RemotingException: Not found exported service: com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880 in [impl2/com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880], may be version or group mismatch , channel: consumer: /192.168.72.158:60792 --> provider: /192.168.72.158:20880, message:RpcInvocation [methodName=compositeQuery, parameterTypes=[class com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery], arguments=[null], attachments={path=com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService, input=829, name=OWNER_UKID, dubbo=2.8.4, version=0.0.0}]
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol.getInvoker(DubboProtocol.java:222)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:91)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:108)
at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:227)
at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72)
at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
at com.alibaba.dubbo.common.bytecode.proxy0.compositeQuery(proxy0.java)
at com.wwwarehouse.xdw.commonindustry.ServiceTest.compositeQuery(ServiceTest.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
服務提供方的錯誤日誌信息：
[2019-01-24 16:46:43,985] INFO -:-:-:- [main] com.wwwarehouse.commons.dubbo.DubboConsumerAuthFilter.invoke(DubboConsumerAuthFilter.java:51) - dubboRpcContext:>>>
com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method compositeQuery in the service com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService. Tried 3 times of the providers [192.168.72.158:20880] (1/1) from the registry 192.168.6.21:2181 on the consumer 192.168.72.158 using the dubbo version 2.8.4. Last error is: Failed to invoke remote method: compositeQuery, provider: dubbo://192.168.72.158:20880/com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService?anyhost=true&application=commonindustry-biz-test&batchChangeInventory.timeout=60000&batchChangeInventoryQty.timeout=60000&changeInventory.timeout=600000&check=false&default.check=false&default.connections=10&default.group=mock&dubbo=2.8.4&eliminatePlanAdjustmentDatas.timeout=60000&generic=false&group=impl2&interface=com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService&lockPlanAdjustmentDatas.timeout=5000&methods=selectSubItemsByGroupUkidAndItemUkid,queryConsumeRealityQTY,queryDeliveryFromStorageQTY,queryMaxQtyGroupForItem,unLockPlanAdjustmentDatas,selectAvailableListByGroupRelationUkid,inventoryQuery,selectItemsByGroupUkidAndIdentifyCode,compositeQuery,lockPlanAdjustmentDatas,getInventoryQueryFilterInfo,batchChangeInventoryQty,listByUkids,updateInventoryStatistics,sumInventoryQty,changeInventoryQty,changeInventory,deliveryFromStorage,selectQualitiesByGroupUkidAndItemUkid,batchChangeInventory,getGoodsStockInfo,warehouseEntry,selectConsumeListByIdentifyCode,addInventory,selectConsumeListByGroupUkid,selectConsumeListByGroupUkids,getInventoryQueryList,getStorageGoodsStock,selectConsumeListByGroupCode,eliminatePlanAdjustmentDatas&organization=wwwarehouse&owner=xdw&pid=17332&protocol=dubbo&revision=1.0-20190124.081517-966&side=consumer×tamp=1548319602539&updateInventoryStatistics.timeout=5000, cause: com.alibaba.dubbo.remoting.RemotingException: Not found exported service: com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880 in [impl2/com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880], may be version or group mismatch , channel: consumer: /192.168.72.158:60792 --> provider: /192.168.72.158:20880, message:RpcInvocation [methodName=compositeQuery, parameterTypes=[class com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery], arguments=[null], attachments={path=com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService, input=829, name=OWNER_UKID, dubbo=2.8.4, version=0.0.0}]
com.alibaba.dubbo.remoting.RemotingException: Not found exported service: com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880 in [impl2/com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880], may be version or group mismatch , channel: consumer: /192.168.72.158:60792 --> provider: /192.168.72.158:20880, message:RpcInvocation [methodName=compositeQuery, parameterTypes=[class com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery], arguments=[null], attachments={path=com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService, input=829, name=OWNER_UKID, dubbo=2.8.4, version=0.0.0}]
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol.getInvoker(DubboProtocol.java:222)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:91)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:108)
at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:227)
at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72)
at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
at com.alibaba.dubbo.common.bytecode.proxy0.compositeQuery(proxy0.java)
at com.wwwarehouse.xdw.commonindustry.ServiceTest.compositeQuery(ServiceTest.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
排查過程
dubbo admin上看了下，服務暴露是正常的。目標方法是新提供的，暫無業務調用。看了其他的方法，有正常消費。當時認爲服務是ok的，讓調用方自身排查下。
調用方排查環境配置ok，未找到原因。
自行測試了下，發現一個詭異的現象。查詢對象不設置條件時，服務可以調用。如果設置，提示找不到服務。
測試代碼如下
CompositeQuery compositeQuery = CompositeQuery.newQuery();
// 下面這行是設置條件，如果不設置，服務可以調用。如果設置，提示找不到服務
// compositeQuery.addExpression(new CompositeQuery.Expression(DBField.OWNER_UKID, Operator.EQ, "-100"));
try {
System.out.println("results:" + imConsumeRealityService.compositeQuery(compositeQuery));
} catch (Exception e) {
e.printStackTrace();
}
結合錯誤提示，看了下源碼。暴露的服務有組名，請求調用的目標服務沒有組名 (impl2是dubbo service group name)，調用方配置檢查ok，有配置組名。如果配置有問題，應該都有問題。
重新查看服務方的日誌，前後關聯日誌都看了下，dubbo輸出了一個error級別的警告
[DUBBO] Decode argument failed: com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery.expressions:
'com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery$Expression' could not be instantiated,
dubbo version: biz, current host: 172.18.253.160 com.alibaba.com.caucho.hessian.io.HessianFieldException:
com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery.expressions:
'com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery$Expression' could not be instantiated at
com.alibaba.com.caucho.hessian.io.JavaDeserializer.logDeserializeError(JavaDeserializer.java:671) ~[resourcecenter-biz.jar:?] at
com.alibaba.com.caucho.hessian.io.JavaDeserializer$ObjectFieldDeserializer.deserialize(JavaDeserializer.java:400) ~[resourcecenter-biz.jar:?] at
com.alibaba.com.caucho.hessian.io.JavaDeserializer.readObject(JavaDeserializer.java:233) ~[resourcecenter-biz.jar:?] at
com.alibaba.com.caucho.hessian.io.JavaDeserializer.readObject(JavaDeserializer.java:157) ~[resourcecenter-biz.jar:?] at
com.alibaba.com.caucho.hessian.io.Hessian2Input.readObjectInstance(Hessian2Input.java:2067) ~[resourcecenter-biz.jar:?] at
com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1592) ~[resourcecenter-biz.jar:?] at
com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1576) ~[resourcecenter-biz.jar:?] at
com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:94) ~[resourcecenter-biz.jar:?] at
com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation.decode(DecodeableRpcInvocation.java:135) [resourcecenter-biz.jar:?] at
com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation.decode(DecodeableRpcInvocation.java:74) [resourcecenter-biz.jar:?] at
com.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec.decodeBody(DubboCodec.java:138) [resourcecenter-biz.jar:?] at
com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:134) [resourcecenter-biz.jar:?] at
com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:95) [resourcecenter-biz.jar:?] at
com.alibaba.dubbo.rpc.protocol.dubbo.DubboCountCodec.decode(DubboCountCodec.java:46) [resourcecenter-biz.jar:?] at
...
CompositeQuery$Expression, 這個對象，沒有定義無參的構造器。新增公有無參構造器後問題解決。
爲什麼參數解析失敗，會導致dubbo service拿不到group信息。
重新調試了下，服務方接收到請求後，對輸入的字節流進行解析。先做的參數讀取，然後讀取附屬信息(如group、version等)。參數解析時候出現了異常，導致獲取到的附屬信息不對，沒有拿到group信息。
public class DecodeableRpcInvocation extends RpcInvocation implements Codec, Decodeable {
public Object decode(Channel channel, InputStream input) throws IOException {
// ...
Object[] args = new Object[pts.length];
for (int i = 0; i < args.length; i++) {
try {
// 參數解析，從輸入流中讀取參數對象
args[i] = in.readObject(pts[i]);
} catch (Exception e) {
if (log.isWarnEnabled()) {
log.warn("Decode argument failed: " + e.getMessage(), e);
}
}
}
// ...
// 讀取附屬信息
Map<String, String> map = (Map<String, String>) in.readObject(Map.class);
if (map != null && map.size() > 0) {
Map<String, String> attachment = getAttachments();
if (attachment == null) {
attachment = new HashMap<String, String>();
}
attachment.putAll(map);
// 設置附屬參數信息
setAttachments(attachment);
}
// ...
}
}
讀取到了正確的附屬信息
讀取到錯誤的附屬信息
問題原因
參數對象裏面有個內部類，沒有定義公有無參構造器。序列化時候失敗了
剛開始方向找偏了，這段時間dubbo服務暴露經常有問題，服務沒有暴露出去。剛開始看到以爲是服務暴露問題。確認dubbo admin服務ok，以爲調用方出了問題，導致排查用了不少時間。
細心！！！ 善用經驗！！！