問題描述
異常日誌
2019-10-22 12:23:43.824 [pool-8-thread-115] INFO c.s.firstShoot.vehicle.VehicleFirstShootPerDay - 開始創建roaringbitmap:20191010000000->20191009235959
2019-10-22 12:23:43.824 [pool-8-thread-117] INFO com.suncreate.utils.VehicleBitmapWRUtils - 執行查詢的sql：×××××××××××××××
2019-10-22 12:23:43.824 [pool-8-thread-116] INFO com.suncreate.utils.VehicleBitmapWRUtils - 執行查詢的sql：×××××××××××××××
2019-10-22 12:23:43.839 [hconnection-0x29f53b77-shared--pool1-t1481] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:23:53.940 [AsyncResolver-bootstrap-executor-0] INFO c.n.d.shared.resolver.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
2019-10-22 12:24:07.894 [hconnection-0x29f53b77-shared--pool1-t1483] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:07.895 [hconnection-0x29f53b77-shared--pool1-t1482] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:07.903 [hconnection-0x29f53b77-shared--pool1-t1484] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:07.908 [hconnection-0x29f53b77-shared--pool1-t1486] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:07.909 [hconnection-0x29f53b77-shared--pool1-t1485] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:07.909 [hconnection-0x29f53b77-shared--pool1-t1487] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:07.910 [hconnection-0x29f53b77-shared--pool1-t1489] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:07.922 [hconnection-0x29f53b77-shared--pool1-t1495] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:07.922 [hconnection-0x29f53b77-shared--pool1-t1488] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:07.922 [hconnection-0x29f53b77-shared--pool1-t1494] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:07.922 [hconnection-0x29f53b77-shared--pool1-t1497] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:07.934 [hconnection-0x29f53b77-shared--pool1-t1496] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:07.936 [hconnection-0x29f53b77-shared--pool1-t1492] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:07.949 [hconnection-0x29f53b77-shared--pool1-t1493] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:07.950 [hconnection-0x29f53b77-shared--pool1-t1491] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:07.953 [hconnection-0x29f53b77-shared--pool1-t1490] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:24:08.151 [pool-8-thread-117] INFO com.suncreate.utils.VehicleBitmapWRUtils - 回溯的bitmap：579288
2019-10-22 12:24:08.338 [pool-8-thread-116] INFO com.suncreate.utils.VehicleBitmapWRUtils - 查詢時間範圍的bitmap：844086
2019-10-22 12:24:08.352 [http-nio-45593-exec-3] INFO com.suncreate.controller.FirstShootController - 查詢結果總計：430286
2019-10-22 12:24:09.203 [http-nio-45593-exec-3] INFO com.suncreate.controller.FirstShootController - 查詢結果轉換後總計：428340
上述日誌 兩條認證日誌之間間隔了十幾秒？？？？？？？？？？？？？？？？？
排查java的堆棧
"hconnection-0x29f53b77-shared--pool1-t1003" #3842 daemon prio=5 os_prio=0 tid=0x00007f99c0001800 nid=0x2b463 runnable [0x00007f97b06c3000]
java.lang.Thread.State: RUNNABLE
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at java.net.InetAddress.getByName(InetAddress.java:1076)
at sun.security.krb5.PrincipalName.<init>(PrincipalName.java:414)
at sun.security.krb5.PrincipalName.<init>(PrincipalName.java:471)
at sun.security.jgss.krb5.Krb5NameElement.getInstance(Krb5NameElement.java:123)
at sun.security.jgss.krb5.Krb5MechFactory.getNameElement(Krb5MechFactory.java:95)
at sun.security.jgss.GSSManagerImpl.getNameElement(GSSManagerImpl.java:203)
at sun.security.jgss.GSSNameImpl.getElement(GSSNameImpl.java:477)
- locked <0x000000066dcb3c00> (a sun.security.jgss.GSSNameImpl)
at sun.security.jgss.GSSNameImpl.init(GSSNameImpl.java:201)
at sun.security.jgss.GSSNameImpl.<init>(GSSNameImpl.java:170)
at sun.security.jgss.GSSManagerImpl.createName(GSSManagerImpl.java:138)
at com.sun.security.sasl.gsskerb.GssKrb5Client.<init>(GssKrb5Client.java:107)
at com.sun.security.sasl.gsskerb.FactoryImpl.createSaslClient(FactoryImpl.java:63)
at javax.security.sasl.Sasl.createSaslClient(Sasl.java:384)
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.createKerberosSaslClient(HBaseSaslRpcClient.java:138)
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.<init>(HBaseSaslRpcClient.java:118)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
- locked <0x000000066dcb3d50> (a org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:165)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:750)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:747)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1781)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:747)
- locked <0x000000066dcb3d50> (a org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:950)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:914)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1288)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:35518)
at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:404)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:211)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:65)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:218)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:398)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:372)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:139)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:79)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
從堆棧中可以看出是由於hconnection在進行地址解析，耗費了這麼長時間？？？？？
解決方案
第一種方法：/etc/hosts加1.1.1.1 hadoop.hadoop.com
第二種方法：dns加超時時間，/ect/resolv.conf 加options attempts:1，options timeout:1
2019-10-22 12:26:50.168 [pool-8-thread-130] INFO c.s.firstShoot.vehicle.VehicleFirstShootPerDay - 開始創建roaringbitmap:20191022000000->20191021235959
2019-10-22 12:26:50.168 [pool-8-thread-130] INFO c.s.firstShoot.vehicle.VehicleFirstShootPerDay - bitmapAll的大小:8
2019-10-22 12:26:50.168 [pool-8-thread-131] INFO com.suncreate.utils.VehicleBitmapWRUtils - 執行查詢的sql：select * from VEHICLE_FIRST_SHOOT_BITMAP_PER_DAY where PASS_DD_TIME >= CAST(TO_DATE('2019-10-12 00:00:00.000') AS DATE) and PASS_DD_TIME < CAST(TO_DATE('2019-10-22 00:00:00.000') AS DATE) and TOLLGATE_ID in ('34010502001199000003','34010503001199000005','34010503001199000012','34010503001199000026','34010503001199000040','34010503001199000054','34010503001199000068','34010503001199000096','34010503001199000103','34010503001199000110','34010503001199000117','34010516001199003001','34010516001199003015','34010517001199003003','34010517001199003017','34010517001199003031','34010517001199003038','34010517001199003045','34010517001199003052','34010517001199003066','34010517001199003073','34010517001199003080')
2019-10-22 12:26:50.169 [pool-8-thread-132] INFO com.suncreate.utils.VehicleBitmapWRUtils - 執行查詢的sql：select * from VEHICLE_FIRST_SHOOT_BITMAP_TEMP where PASS_DD_TIME = CAST(TO_DATE('2019-10-12 00:00:00.000') AS DATE) and days = 90 and tollgate_id in ('34010502001199000003','34010503001199000005','34010503001199000012','34010503001199000026','34010503001199000040','34010503001199000054','34010503001199000068','34010503001199000096','34010503001199000103','34010503001199000110','34010503001199000117','34010516001199003001','34010516001199003015','34010517001199003003','34010517001199003017','34010517001199003031','34010517001199003038','34010517001199003045','34010517001199003052','34010517001199003066','34010517001199003073','34010517001199003080')
2019-10-22 12:26:50.179 [hconnection-0x29f53b77-shared--pool1-t1572] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:26:50.180 [hconnection-0x29f53b77-shared--pool1-t1575] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:26:50.188 [hconnection-0x29f53b77-shared--pool1-t1578] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:26:50.188 [hconnection-0x29f53b77-shared--pool1-t1580] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:26:50.195 [hconnection-0x29f53b77-shared--pool1-t1579] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:26:50.196 [hconnection-0x29f53b77-shared--pool1-t1577] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:26:50.197 [hconnection-0x29f53b77-shared--pool1-t1576] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:26:50.198 [hconnection-0x29f53b77-shared--pool1-t1573] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:26:50.198 [hconnection-0x29f53b77-shared--pool1-t1574] INFO org.apache.hadoop.hbase.ipc.RpcClientImpl - RPC Server Kerberos principal name for service=ClientService is hbase/[email protected]
2019-10-22 12:26:50.427 [pool-8-thread-131] INFO com.suncreate.utils.VehicleBitmapWRUtils - 查詢時間範圍的bitmap：518682
2019-10-22 12:26:50.466 [pool-8-thread-132] INFO com.suncreate.utils.VehicleBitmapWRUtils - 回溯的bitmap：2006142
2019-10-22 12:26:50.474 [http-nio-45593-exec-8] INFO com.suncreate.controller.FirstShootController - 查詢結果總計：101578
2019-10-22 12:26:50.720 [http-nio-45593-exec-8] INFO com.suncreate.controller.FirstShootController - 查詢結果轉換後總計：101436