Spark報錯——AnnotatedConnectException拒絕連接

1、問題描述

在運行spark任務時候報錯如下：

17/11/03 10:27:54 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) from 192.168.1.16:37205
java.io.IOException: Failed to connect to /192.168.1.16:37205
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)
    at org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:97)
    at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
    at org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43)
    at org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
    at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: 拒絕連接: /192.168.1.16:37205
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:640)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
    ... 2 more

2、分析&解決

1、spark安裝時候沒有預留內存

出現上述問題可能是一個原因可能是內存問題。在安裝spark的時候沒有對其使用內存進行限制，那麼spark UI上面看到的內存就是機器的所有內存，即機器所有的內存都是可以用於跑spark任務，所有在提交spark任務的時候應該要給機器預留一下內存空間，我當時總共480G內存spark任務使用了450G，通過減少executor內存和數量最後問題解決了.

2、代碼執行了shuffle操作，並且memory_only模式

持久化到內存時候發生溢出，那麼有些數據就會被刪除，在下一步執行時候就找不到了，所有就報錯。解決方法是修改持久化級別，修改爲memory and disk

3、其他原因

本人水平有限，只知道這兩種，其他的也沒有碰到過，暫時不清楚，若解決不了你的問題望見諒哈