【Spring Boot集成DOCKER+FDFS上傳圖片ERROR】無法獲取服務端連接資源：can't create connection to/10.111.114.6:23000] with root cause.
前言
同事想要一套fdfs環境，但是公司要求相關服務只能裝在docker容器中，且對容器的創建樣式有一定要求，於是我毅然決然地開始了踩坑之路，特此記錄。
容器創建
下載鏡像：
docker image pull delron/fastdfs
使用docker-compose.yml，其中docker compose明細如下：
version: '2.2'
services:
fastdfs_server2:
image: docker.io/delron/fastdfs
container_name: fastdfs_server2
hostname: fastdfs-server2
volumes:
- ../fastdfs:/opt/fastdfs
cpus: 1
mem_limit: 2G
privileged: true
# command:
#- bash
#- -c
#- 'tail -f /dev/null'
environment:
TRACKER_SERVER: "10.111.114.6:22122"
ports:
- 5005:8888
- 5006:22122
- 23000:23000
networks:
20190116_aidata_network:
ipv4_address: 10.111.114.6
networks:
20190116_aidata_network:
external: true
啓動容器後，本地上傳文件是成功的，相關命令如下：
docker exec -it fastdfs_server2 /bin/bash;
# 上傳命令
/usr/bin/fdfs_upload_file /etc/fdfs/client.conf /opt/fastdfs/播求.jpg;
根據訪問返回上傳地址拼接宿主機IP訪問相應的url：
OK，很完美是不是？
接下來用sprintboot集成一下，測試一下增刪改查（坑來了）。
- 這裏說明一下，因爲有定製需求，我的docker_compose.yml創建出來後會有command內的命令失效（不執行）現象，問題點我也定位到了，是因爲有個啓動腳本最後一句是"tail -f “$FASTDFS_LOG_FILE” "，導致進程卡住無法往後執行命令，所以我對容器本身做了一些定製化修改，主要的修改內容是/usr/bin/start1.sh腳本，目的是在進程卡死之前執行一些我想執行的操作，然後把這個版本保存爲鏡像（方便以後使用），我貼一下修改後的start1.sh代碼：
#!/bin/bash
#set -e
if [ "$1" = "monitor" ] ; then
if [ -n "$TRACKER_SERVER" ] ; then
sed -i "s|tracker_server=.*$|tracker_server=${TRACKER_SERVER}|g" /etc/fdfs/client.conf
fi
fdfs_monitor /etc/fdfs/client.conf
exit 0
elif [ "$1" = "storage" ] ; then
FASTDFS_MODE="storage"
sed -i "s|store_path0.*$|store_path0=/var/fdfs|g" /etc/fdfs/mod_fastdfs.conf
sed -i "s|url_have_group_name =.*$|url_have_group_name = true|g" /etc/fdfs/mod_fastdfs.conf
/usr/local/nginx/sbin/nginx
else
FASTDFS_MODE="tracker"
fi
if [ -n "$PORT" ] ; then
sed -i "s|^port=.*$|port=${PORT}|g" /etc/fdfs/"$FASTDFS_MODE".conf
fi
if [ -n "$TRACKER_SERVER" ] ; then
sed -i "s|tracker_server=.*$|tracker_server=${TRACKER_SERVER}|g" /etc/fdfs/storage.conf
sed -i "s|tracker_server=.*$|tracker_server=${TRACKER_SERVER}|g" /etc/fdfs/client.conf
sed -i "s|tracker_server=.*$|tracker_server=${TRACKER_SERVER}:22122|g" /etc/fdfs/mod_fastdfs.conf
fi
if [ -n "$GROUP_NAME" ] ; then
sed -i "s|group_name=.*$|group_name=${GROUP_NAME}|g" /etc/fdfs/storage.conf
fi
FASTDFS_LOG_FILE="${FASTDFS_BASE_PATH}/logs/${FASTDFS_MODE}d.log"
PID_NUMBER="${FASTDFS_BASE_PATH}/data/fdfs_${FASTDFS_MODE}d.pid"
echo "try to start the $FASTDFS_MODE node..."
if [ -f "$FASTDFS_LOG_FILE" ]; then
rm "$FASTDFS_LOG_FILE"
fi
# start the fastdfs node.
fdfs_${FASTDFS_MODE}d /etc/fdfs/${FASTDFS_MODE}.conf start
# wait for pid file(important!),the max start time is 5 seconds,if the pid number does not appear in 5 seconds,start failed.
TIMES=5
while [ ! -f "$PID_NUMBER" -a $TIMES -gt 0 ]
do
sleep 1s
TIMES=`expr $TIMES - 1`
done
/usr/bin/fdfs_storaged /etc/fdfs/storage.conf stop
/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf stop
#sh /opt/fastdfs/iptables.sh
rm -rf /etc/fdfs/tracker.conf /etc/fdfs/client.conf /etc/fdfs/storage.conf /usr/local/nginx/conf/nginx.conf /etc/fdfs/mod_fastdfs.conf
cp /opt/fastdfs/tracker.conf /opt/fastdfs/client.conf /opt/fastdfs/storage.conf /opt/fastdfs/mod_fastdfs.conf /etc/fdfs
cp /opt/fastdfs/nginx.conf /usr/local/nginx/conf
/etc/init.d/fdfs_trackerd start
/etc/init.d/fdfs_storaged start
/usr/local/nginx/sbin/nginx
# if the storage node start successfully, print the started time.
# if [ $TIMES -gt 0 ]; then
# echo "the ${FASTDFS_MODE} node started successfully at $(date +%Y-%m-%d_%H:%M)"
# # give the detail log address
# echo "please have a look at the log detail at $FASTDFS_LOG_FILE"
# # leave balnk lines to differ from next log.
# echo
# echo
# # make the container have foreground process(primary commond!)
# tail -F --pid=`cat $PID_NUMBER` /dev/null
# # else print the error.
# else
# echo "the ${FASTDFS_MODE} node started failed at $(date +%Y-%m-%d_%H:%M)"
# echo "please have a look at the log detail at $FASTDFS_LOG_FILE"
# echo
# echo
# fi
tail -f "$FASTDFS_LOG_FILE"
固化容器爲鏡像的命令我順便也貼一下：
docker commit -a 'jack-roy' -m 'add iptables' fastdfs_server1 docker.io/delron/fastdfs:v1
Spring Boot集成
pom文件：
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.jackroy.www</groupId>
<artifactId>fastdfs_test</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>fastdfs_test</name>
<description>fastdfs_test</description>
<packaging>jar</packaging>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.5.RELEASE</version>
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
<fastjson.version>1.2.49</fastjson.version>
<springboot.version>2.0.5.RELEASE</springboot.version>
<skipTests>true</skipTests>
</properties>
<dependencies>
<dependency>
<groupId>com.github.tobato</groupId>
<artifactId>fastdfs-client</artifactId>
<version>1.26.6</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<version>${springboot.version}</version>
<exclusions>
<exclusion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<version>${springboot.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>jstl</artifactId>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>${fastjson.version}</version>
</dependency>
</dependencies>
<build>
<finalName>jack-roy</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.5</version>
</plugin>
</plugins>
</build>
</project>
上傳API：
package com.jackroy.www.controller;
import com.github.tobato.fastdfs.domain.fdfs.StorePath;
import com.github.tobato.fastdfs.domain.proto.storage.DownloadByteArray;
import com.github.tobato.fastdfs.service.FastFileStorageClient;
import com.sun.deploy.net.URLEncoder;
import org.apache.commons.io.FilenameUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;
import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
/**
* @Copyright: www.jackroy.com
* @Version: V1.0
* @Author Jack-Roy
* 2019/12/13 17:43
*/
@RestController
public class Test {
@Autowired
private FastFileStorageClient fastFileStorageClient;
@PostMapping("/uppload")
public StorePath test(@RequestParam MultipartFile file) throws IOException {
StorePath storePath = fastFileStorageClient.uploadFile(file.getInputStream(), file.getSize(), FilenameUtils.getExtension(file.getOriginalFilename()), null);
return storePath;
}
}
application.yml相關配置：
server:
port: 8080
suffix: .jsp
logging:
file: logs/fdfs.log
level:
com.jackroy.www: DEBUG
learning: trace
fdfs:
so-timeout: 1501
network_timeout: 2000
connect-timeout: 2000
thumb-image: #縮略圖生成參數
width: 150
height: 150
tracker-list:
- 宿主機IP:5006
啓動項目:
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.0.5.RELEASE)
2019-12-16 16:38:35.351 INFO 15064 --- [ main] com.jackroy.www.Application : Starting Application on it31826 with PID 15064 (D:\project\fastdfs_test\target\classes started by liujiayi in D:\project\fastdfs_test)
2019-12-16 16:38:35.365 DEBUG 15064 --- [ main] com.jackroy.www.Application : Running with Spring Boot v2.0.5.RELEASE, Spring v5.0.9.RELEASE
2019-12-16 16:38:35.365 INFO 15064 --- [ main] com.jackroy.www.Application : No active profile set, falling back to default profiles: default
2019-12-16 16:38:35.469 INFO 15064 --- [ main] ConfigServletWebServerApplicationContext : Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@6c80d78a: startup date [Mon Dec 16 16:38:35 CST 2019]; root of context hierarchy
2019-12-16 16:38:37.142 INFO 15064 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2019-12-16 16:38:37.183 INFO 15064 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2019-12-16 16:38:37.183 INFO 15064 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet Engine: Apache Tomcat/8.5.34
2019-12-16 16:38:37.195 INFO 15064 --- [ost-startStop-1] o.a.catalina.core.AprLifecycleListener : The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [D:\JAVA\JDK\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;D:\Xshell\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\JAVA\JDK\bin;D:\JAVA\JDK\jre\bin;D:\scala-2.11.8\bin;D:\apache-maven-3.5.4\bin;D:\Git\cmd;D:\GIT\usr\bin;D:\node-v12.13.1-win-x64;D:\python3.5.4\Scripts\;D:\python3.5.4\;.]
2019-12-16 16:38:37.326 INFO 15064 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2019-12-16 16:38:37.326 INFO 15064 --- [ost-startStop-1] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 1857 ms
2019-12-16 16:38:37.390 INFO 15064 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean : Servlet dispatcherServlet mapped to [/]
2019-12-16 16:38:37.395 INFO 15064 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'characterEncodingFilter' to: [/*]
2019-12-16 16:38:37.395 INFO 15064 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2019-12-16 16:38:37.395 INFO 15064 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'httpPutFormContentFilter' to: [/*]
2019-12-16 16:38:37.396 INFO 15064 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'requestContextFilter' to: [/*]
2019-12-16 16:38:37.580 INFO 15064 --- [ main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2019-12-16 16:38:37.752 INFO 15064 --- [ main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@6c80d78a: startup date [Mon Dec 16 16:38:35 CST 2019]; root of context hierarchy
2019-12-16 16:38:37.830 INFO 15064 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/delete],methods=[DELETE]}" onto public java.lang.String com.jackroy.www.controller.Test.delete(java.lang.String)
2019-12-16 16:38:37.831 INFO 15064 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/uppload],methods=[POST]}" onto public com.github.tobato.fastdfs.domain.fdfs.StorePath com.jackroy.www.controller.Test.test(org.springframework.web.multipart.MultipartFile) throws java.io.IOException
2019-12-16 16:38:37.831 INFO 15064 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/download],methods=[GET]}" onto public void com.jackroy.www.controller.Test.downLoad(java.lang.String,java.lang.String,java.lang.String,javax.servlet.http.HttpServletResponse) throws java.io.IOException
2019-12-16 16:38:37.832 INFO 15064 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/test],methods=[GET]}" onto public java.lang.String com.jackroy.www.controller.TestApi.test()
2019-12-16 16:38:37.834 INFO 15064 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2019-12-16 16:38:37.835 INFO 15064 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2019-12-16 16:38:37.857 INFO 15064 --- [ main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2019-12-16 16:38:37.858 INFO 15064 --- [ main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2019-12-16 16:38:38.102 INFO 15064 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
2019-12-16 16:38:38.103 INFO 15064 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Bean with name 'fdfsConnectionPool' has been autodetected for JMX exposure
2019-12-16 16:38:38.107 INFO 15064 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Located MBean 'fdfsConnectionPool': registering with JMX server as MBean [com.github.tobato.fastdfs.domain.conn:name=fdfsConnectionPool,type=FdfsConnectionPool]
2019-12-16 16:38:38.172 INFO 15064 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2019-12-16 16:38:38.185 INFO 15064 --- [ main] com.jackroy.www.Application : Started Application in 3.407 seconds (JVM running for 3.877)
毫無報錯（這說明連接tracker的22122端口是正常的）。
請求報錯
用postman請求一下接口，發現報錯，其中postman返回的結果是：
{
"timestamp": "2019-12-16T08:11:40.836+0000",
"status": 500,
"error": "Internal Server Error",
"message": "無法獲取服務端連接資源：can't create connection to/10.111.114.6:23000",
"path": "/uppload"
}
IDEA控制檯報無法獲取服務端連接資源：
2019-12-16 16:42:24.546 ERROR 15064 --- [nio-8080-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.github.tobato.fastdfs.exception.FdfsConnectException: 無法獲取服務端連接資源：can't create connection to/10.111.114.6:23000] with root cause
前往docker容器裏查詢日誌發現沒有相關記錄，稍微冷靜了一下，想到問題的關鍵了：
10.111.114.6作爲storage的綁定ip被註冊到跟蹤器tracker上（23000作爲storage的通訊地址在storage.conf內指定），本地環境中通過連接宿主機的5006端口能夠訪問docker內部的22122端口，並拿到可用來存儲的storage地址，這個地址就是10.111.114.6，問題是該地址對於docker和宿主機是可見的，但是對我本來來說卻是不可見的，因此當接口得到地址“10.111.114.6:23000”後想要上傳圖片，卻無法進行連接。
解決辦法
收集了很多資料後，我認爲修改iptables的NAT表規則能夠有效的解決這類問題，我們將請求tracker:2212的數據源地址修改爲宿主機地址，這樣一來，我們storage的註冊請求發送到tracker上時，註冊ip就變成宿主機ip，以此保證服務在本地的可見性。
# 進入docker container
docker exec -it fastdfs_server2 /bin/bash;
# 安裝iptables
yum install iptables -y;
# 添加iptables NAT表規則
iptables -t nat -A POSTROUTING -p tcp -m tcp --dport 22122 -d 10.111.114.6 -j SNAT --to 宿主機IP;
# 查看NAT表規則
iptables -L -t nat;
後續操作
完成iptables相關操作以後，牢記：
先殺掉storage進程，再啓動storage使其重新註冊；
然後重啓本地項目，使用postman的post請求上傳圖片：
{
"group": "group1",
"path": "M00/00/00/Cm9yBV33SOqAfOHNAAIRFIQciyw918.jpg",
"fullPath": "group1/M00/00/00/Cm9yBV33SOqAfOHNAAIRFIQciyw918.jpg"
}
訪問一波：
成了~
後記
可以預見的是，即使我這般寫了，但還是會有很多坑，碰到類似問題的朋友可在下方留言交流。
