異常原因:
在SparkStreaming中使用Redis進行高併發寫,發現有時候會出現以下異常:
redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Broken pipe (Write failed)
at redis.clients.jedis.Connection.flush(Connection.java:303)
at redis.clients.jedis.Connection.getMany(Connection.java:317)
at redis.clients.jedis.Pipeline.sync(Pipeline.java:98)
at redis.clients.jedis.Pipeline.clear(Pipeline.java:84)
at redis.clients.jedis.Pipeline.close(Pipeline.java:157)
at com.amt.spark.streaming.UserBehaviorStreaming$1$12$1.call(UserBehaviorStreaming.java:353)
at com.amt.spark.streaming.UserBehaviorStreaming$1$12$1.call(UserBehaviorStreaming.java:331)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:225)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:225)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at redis.clients.jedis.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52)
at redis.clients.jedis.util.RedisOutputStream.flush(RedisOutputStream.java:133)
at redis.clients.jedis.Connection.flush(Connection.java:300)
解決過程:
百度谷歌嘗試了各種方法,發現一點用都沒有,異常原因給我引導到錯誤的排查方式去了。
解決方法: