OSS實現多文件多線程的斷點上傳（java）

本文地址：http://blog.csdn.net/zrk1000/article/details/46288927

前面寫了關於OSS實現多文件多線程的斷點下載的文章，今天寫下使用OSS-SDK實現多文件多線程的斷點上傳，實現方式和斷點下載使用的mainPool和pool嵌套線程池的方式相同，父線程池管理文件個數，子線程池pool每個文件的上傳線程數。

uploadMainPool 代碼片段：

    public static ExecutorService uploadMainPool = null;
    static{
        uploadMainPool = Executors.newFixedThreadPool(Constant.CONCURRENT_FILE_NUMBER,new ThreadFactory() {
            public Thread newThread(Runnable r) {
                Thread s = Executors.defaultThreadFactory().newThread(r);
                s.setDaemon(true);
                return s;
            }
        });
    }

在static中實例化爲固定大小的線程池，由於默認的ThreadFactory創建的線程爲非守護狀態，爲了避免java程序不能退出的問題，保證在文件上傳完成後當前java程序結束在jvm中的運行，需要重寫ThreadFactory，使其創建的線程爲守護線程。

Constant類中自定義了程序中需要使用到的變量，可在類中直接定義或讀取配置文件，Constant.CONCURRENT_FILE_NUMBER定義了併發文件數。

uploadMainPool 中的每個線程負責一個文件，每個線程上傳文件時創建子線程池，由子線程池分塊上傳文件；
使用OSS-SDK分塊上傳就必須使用到其提供的MultipartUpload上傳方式,具體參見oss的sdk，由於oss支持在雲端記錄上傳事件上傳的碎片（就是一個上傳操作中的塊），我們可以使用這一點完成斷點上傳，當然這裏的斷點並不能做到記錄到每個上傳塊上傳停止的位置，是取決於OSS-SDK的限制，但是上傳中能記錄到已經上傳的塊就已經足夠了
這裏仍然使用定時序列化子線程池線程對象的方式，定時將包含了上傳塊信息的線程序列化到文件,在再次上傳同一文件時反序列化，直接丟到子線程池上傳即可。下面是uploadMainPool 中的線程OSSUploadFile代碼清單

OSSUploadFile代碼：

package cloudStorage.oss;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.TimeUnit;
import org.apache.log4j.Logger;
import cloudStorage.basis.Constant;
import cloudStorage.basis.Global;
import cloudStorage.basis.OSSClientFactory;
import cloudStorage.oss.upload.UploadPartObj;
import cloudStorage.oss.upload.UploadPartThread;
import cloudStorage.util.MD5Util;
import cloudStorage.util.ObjectSerializableUtil;
import com.aliyun.oss.ClientException;
import com.aliyun.oss.OSSClient;
import com.aliyun.oss.OSSErrorCode;
import com.aliyun.oss.OSSException;
import com.aliyun.oss.model.CannedAccessControlList;
import com.aliyun.oss.model.CompleteMultipartUploadRequest;
import com.aliyun.oss.model.InitiateMultipartUploadRequest;
import com.aliyun.oss.model.InitiateMultipartUploadResult;
import com.aliyun.oss.model.ObjectMetadata;
import com.aliyun.oss.model.PartETag;
import com.aliyun.oss.model.PutObjectResult;


/**
 * @Description: 使用普通方式上傳小文件，使用Multipart上傳方式進行多線程分段上傳較大文件
 * @author: zrk
 * @time: 2015年3月30日 上午10:45:12
 */
public class OSSUploadFile implements Callable<Integer>{

    public static final Logger LOGGER = Logger.getLogger(OSSUploadFile.class);

    //外層線程池
    public static ExecutorService uploadMainPool = null;
    static{
        uploadMainPool = Executors.newFixedThreadPool(Constant.CONCURRENT_FILE_NUMBER,new ThreadFactory() {
            public Thread newThread(Runnable r) {
                Thread s = Executors.defaultThreadFactory().newThread(r);
                s.setDaemon(true);
                return s;
            }
        });
    }
    //內層線程池
    private ExecutorService pool ;
    private String sourcePath;//上次路徑
    private String bucketName;//bucketName
    private String key;//雲端存儲路徑
    /**
     * oss上傳 支持斷點續傳
     * @param sourcePath    源文件路徑
     * @param bucketName    bucketName
     * @param key           存儲key  -在oss的存儲路徑
     */
    public OSSUploadFile(String sourcePath,String bucketName,String key) {
        //實例化單文件上次線程池
        pool = Executors.newFixedThreadPool(Constant.SINGLE_FILE_CONCURRENT_THREADS);
        this.sourcePath = sourcePath;
        this.bucketName = bucketName;
        this.key = key;
    }

    /**
     * 執行當前線程
     * @return
     */
    @SuppressWarnings("finally")
    public Integer uploadFile() {
        Integer r = Global.ERROR;
         //向uploadMainPool中submit當前線程
        Future<Integer> result = uploadMainPool.submit(this);
        try {
            r=result.get();
        } catch (InterruptedException | ExecutionException e) {
            e.printStackTrace();
        } finally{
            return r;
        }

    }


    /**
     * oss上傳
     * @param sourcePath    源文件路徑
     * @param bucketName    bucketName
     * @param key           存儲key  存儲路徑
     * @return Integer
     */
    @Override
    public Integer call(){

        OSSClient client  = OSSClientFactory.getInstance();    

        File uploadFile = new File(sourcePath);
        if (!uploadFile.exists()){ 
            LOGGER.info("==無法找到文件：" + sourcePath);
            return Global.FILE_NOT_FOUND_ERROR;
        }
        int result = Global.ERROR;

        key = key.contains("\\\\")?key.replaceAll("\\\\", "/"):key.contains("\\")?key.replaceAll("\\", "/"):key; 
        // 準備Bucket
        result = ensureBucket(client,bucketName);
        if(result == Global.ERROR )return result;
        // 使用multipart的方式上傳文件
        result = uploadBigFile(client, bucketName, key, uploadFile);
        pool = null;
        return result;
    }


    // 通過Multipart的方式上傳一個大文件
    private int uploadBigFile(OSSClient client, String bucketName, String key,
            File uploadFile)  {

        //自定義的每個上傳分塊大小
        Integer partSize = Constant.UPLOAD_PART_SIZE;

        //需要上傳的文件分塊數
        int partCount = calPartCount(uploadFile,partSize);

        //文件的MD5值
        String fileDM5Str = "";
        String uploadId = "";

        //序列化的文件路徑（與上傳文件同路徑使用.up.temp後綴）
        String serializationFilePath = sourcePath+".up.temp";

        boolean isSerializationFile = false;

        //子線程池的線程對象封裝類（用於序列化的）
        UploadPartObj uploadPartObj = null;
        //獲取文件MD5值
        fileDM5Str = MD5Util.md5Hex(uploadFile);

        //若存在上傳失敗留下的序列化文件則反序列化對象
        if(new File(serializationFilePath).exists()){
            uploadPartObj = (UploadPartObj)ObjectSerializableUtil.load(serializationFilePath);
            isSerializationFile = true;
        }       

        //序列化文件不存在，分配分塊給子線程池線程對象
        if(uploadPartObj==null||!isSerializationFile){
            uploadPartObj = new UploadPartObj();
            try {
                //初始化MultipartUpload 返回uploadId  
                uploadId = initMultipartUpload(client, bucketName, key,fileDM5Str);
            } catch (OSSException|ClientException e) {
                e.printStackTrace();
                LOGGER.error("=="+e.getMessage());
                return Global.OSS_SUBMIT_ERROR;
            } 
            for (int i = 0; i < partCount ; i++) {
                long start = partSize * i;
                long curPartSize = partSize < uploadFile.length() - start ?
                        partSize : uploadFile.length() - start;
                //構造上傳線程，UploadPartThread是執行每個分塊上傳任務的線程
                uploadPartObj.getUploadPartThreads().add(new UploadPartThread(client, bucketName, key,uploadFile, uploadId, i + 1,partSize * i, curPartSize));
            }
        }

        try {
            int i = 0;
            //upload方法提交分塊上傳線程至子線程池上傳，while循環用於上傳失敗重複上傳，Constant.RETRY定義重複次數
            while (upload(uploadPartObj,serializationFilePath).isResult()==false) {
                if(++i == Constant.RETRY)break;
            }
        } catch (Exception e) {
            e.printStackTrace();
            LOGGER.info("==" + e.getMessage());
            return Global.THREAD_ERROR;
        } 

        if(!uploadPartObj.isResult()){
            return Global.NETWORK_ERROR;
        }
        try {
            //完成一個multi-part請求。
            completeMultipartUpload(client, bucketName, key, uploadPartObj);
        } catch (Exception e) {
            e.printStackTrace();
            LOGGER.info("==" + e.getMessage());
            ObjectSerializableUtil.save(uploadPartObj,serializationFilePath);
            return Global.OSS_SUBMIT_ERROR;
        }
        return Global.SUCCESS;
    }

    /**
     * 多線程上傳單個文件
     * @param uploadPartObj
     * @param serializationFilePath
     * @return
     */
    private UploadPartObj upload(UploadPartObj uploadPartObj,String serializationFilePath){

        try {

            uploadPartObj.setResult(true);

            //向子線程池中submit單個文件所有分塊上傳線程
            for (int i=0;i<uploadPartObj.getUploadPartThreads().size();i++) {
                if(uploadPartObj.getUploadPartThreads().get(i).getMyPartETag()==null)
                    pool.submit(uploadPartObj.getUploadPartThreads().get(i));
            }

            //shutdown子線程池，池內所上傳任務執行結束後停止當前線程池
            pool.shutdown();
            while (!pool.isTerminated()) {
                //循環檢查線程池，同時在此序列化uploadPartObj
                ObjectSerializableUtil.save(uploadPartObj,serializationFilePath);
                pool.awaitTermination(Constant.SERIALIZATION_TIME, TimeUnit.SECONDS);
            }

            //判斷上傳結果
            for (UploadPartThread uploadPartThread: uploadPartObj.getUploadPartThreads()) {
                if(uploadPartThread.getMyPartETag()==null)
                    uploadPartObj.setResult(false);
            }
            //上傳成功 刪除序列化文件
            if (uploadPartObj.isResult()==true) 
                ObjectSerializableUtil.delSerlzFile(serializationFilePath);
        } catch (Exception e) {
            LOGGER.info("=="+e.getMessage());

        }
        return uploadPartObj;


    }

    // 根據文件的大小和每個Part的大小計算需要劃分的Part個數。
    private static int calPartCount(File f,Integer partSize) {
        int partCount = (int) (f.length() / partSize);
        if (f.length() % partSize != 0){
            partCount++;
        }
        return partCount;
    }

    // 創建Bucket
    private int ensureBucket(OSSClient client, String bucketName){
            try {
                // 創建bucket
                client.createBucket(bucketName);
                //設置bucket的訪問權限，public-read-write權限
                client.setBucketAcl(bucketName, CannedAccessControlList.PublicRead);
            } catch (OSSException e) {
                e.printStackTrace();
                LOGGER.info("==" + e.getMessage());
                return Global.ERROR;
            } catch (ClientException e) {
                if (!OSSErrorCode.BUCKET_ALREADY_EXISTS.equals(e.getErrorCode())) {
                    // 如果Bucket已經存在，則忽略
                LOGGER.info("==bucketName已經存在");
                }else{
                    e.printStackTrace();
                    LOGGER.info("=="+e.getMessage());
                    return Global.ERROR;
                }
            }
            return Global.SUCCESS;
    }

    // 初始化一個Multi-part upload請求。
    private static String initMultipartUpload(OSSClient client,String bucketName, String key,String fileDM5Str) throws OSSException,ClientException{
        ObjectMetadata objectMetadata =new ObjectMetadata();
        objectMetadata.getUserMetadata().put(Global.X_OSS_META_MY_MD5,fileDM5Str);
        InitiateMultipartUploadRequest initUploadRequest = new InitiateMultipartUploadRequest(bucketName, key, objectMetadata);
        InitiateMultipartUploadResult initResult = client.initiateMultipartUpload(initUploadRequest);
        String uploadId = initResult.getUploadId();
        return uploadId;
    }

    // 完成一個multi-part請求。
    private static void completeMultipartUpload(OSSClient client, String bucketName, String key,UploadPartObj uploadPartObj){
        List<PartETag> eTags = new ArrayList<PartETag>(); 
        for (UploadPartThread uploadPartThread : uploadPartObj.getUploadPartThreads()) {
            eTags.add(new PartETag(uploadPartThread.getMyPartETag().getPartNumber(),uploadPartThread.getMyPartETag().geteTag()));
        }
        //爲part按partnumber排序
        Collections.sort(eTags, new Comparator<PartETag>(){
            public int compare(PartETag arg0, PartETag arg1) {
                PartETag part1= arg0;
                PartETag part2= arg1;
                return part1.getPartNumber() - part2.getPartNumber();
            }  
        });
        CompleteMultipartUploadRequest completeMultipartUploadRequest =
                new CompleteMultipartUploadRequest(bucketName, key, uploadPartObj.getUploadPartThreads().get(0).getUploadId(), eTags);
        client.completeMultipartUpload(completeMultipartUploadRequest);
    }
}

此處引用了多個類
cloudStorage.basis.Constant;//定義程序中使用的變量
cloudStorage.basis.Global;//定義了全局的靜態值，錯誤狀態值
cloudStorage.basis.OSSClientFactory;//OSSClient工廠
cloudStorage.oss.upload.UploadPartObj;//分塊上傳線程類封裝
cloudStorage.oss.upload.UploadPartThread;//分塊上傳線程
cloudStorage.util.ObjectSerializableUtil;//序列工具類
cloudStorage.util.MD5Util;//MD5工具類

調上傳方法也是阻塞式，需要給調用者返回結果，所以使用Callable和Future返回int型狀態值，下面是子線程池pool中的分塊上傳線程UploadPartThread的代碼清單

UploadPartThread 代碼：

package cloudStorage.oss.upload;

import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import java.io.Serializable;
import java.util.Date;
import java.util.concurrent.Callable;

import org.apache.log4j.Logger;

import cloudStorage.basis.OSSClientFactory;
import cloudStorage.service.InDateService;

import com.aliyun.oss.OSSClient;
import com.aliyun.oss.model.UploadPartRequest;
import com.aliyun.oss.model.UploadPartResult;

/**
 * @Description: 上傳每個part的線程類  可序列化 用於上傳的斷點續傳
 * @author: zrk
 * @time: 2015年4月1日 上午10:35:34
 */
public class UploadPartThread implements Callable<UploadPartThread> ,Serializable {

    private static final long serialVersionUID = 1L;
    public static final Logger LOGGER = Logger.getLogger(UploadPartThread.class);

    private File uploadFile;
    private String bucket;
    private String object;
    private long start;
    private long size;
    private int partId;
    private String uploadId;

    private MyPartETag myPartETag; 

    public UploadPartThread(OSSClient client,String bucket, String object,
            File uploadFile,String uploadId, int partId,
            long start, long partSize) {
        this.uploadFile = uploadFile;
        this.bucket = bucket;
        this.object = object;
        this.start = start;
        this.size = partSize; 
        this.partId = partId;
        this.uploadId = uploadId;
    }

    @Override
    public UploadPartThread call() {

        InputStream in = null;
        try {
            in = new FileInputStream(uploadFile);
            in.skip(start);

            UploadPartRequest uploadPartRequest = new UploadPartRequest();
            uploadPartRequest.setBucketName(bucket);
            uploadPartRequest.setKey(object);
            uploadPartRequest.setUploadId(uploadId);
            uploadPartRequest.setInputStream(in);
            uploadPartRequest.setPartSize(size);
            uploadPartRequest.setPartNumber(partId);
            //MyPartETag是對uploadPartResult.getPartETag()的返回值PartETag的封裝，主要是爲了能序列化PartETag，MyPartETag僅比PartETag多實現了Serializable接口
            myPartETag = new MyPartETag(uploadPartResult.getPartETag());
        } catch (Exception e) {
            e.printStackTrace();
            LOGGER.error("=="+e.getMessage());
        } finally {
            if (in != null){
                try {
                    in.close();
                } catch (Exception e) {
                    LOGGER.error("==關閉讀入流失敗："+e.getMessage());
                }
            }
            return this;
        }
    }

    public String getUploadId() {
        return uploadId;
    }

    public void setUploadId(String uploadId) {
        this.uploadId = uploadId;
    }

    public MyPartETag getMyPartETag() {
        return myPartETag;
    }

    public void setMyPartETag(MyPartETag myPartETag) {
        this.myPartETag = myPartETag;
    }
}

每個UploadPartThread 上傳線程執行結束都會將自己作爲返回值返回，當前文件是否上傳完整有每個線程的返回值決定。MyPartETag是對uploadPartResult.getPartETag()的返回值PartETag的封裝，主要是爲了能序列化PartETag中的值，MyPartETag僅比PartETag多實現了Serializable接口

ObjectSerializableUtil 代碼點擊查看

UploadPartObj 代碼：

package cloudStorage.oss.upload;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
/**
 * 單個文件的上傳線程集合
 * @Description: TODO
 * @author: zrk
 * @time: 2015年5月6日 下午1:56:54
 */
public class UploadPartObj implements Serializable{

    private static final long serialVersionUID = 1L;

    List<UploadPartThread> uploadPartThreads = Collections.synchronizedList(new ArrayList<UploadPartThread>());

    boolean result = true;

    public List<UploadPartThread> getUploadPartThreads() {
        return uploadPartThreads;
    }
    public void setUploadPartThreads(List<UploadPartThread> uploadPartThreads) {
        this.uploadPartThreads = uploadPartThreads;
    }
    public boolean isResult() {
        return result;
    }
    public void setResult(boolean result) {
        this.result = result;
    }
}

所有上傳當前文件的線程都會操作uploadPartThreads ，所以uploadPartThreads 使用集合Collections.synchronizedList將其轉換爲一個線程安全的類。UploadPartObj 封裝了UploadPartThreads和一個用於標識上傳成功與失敗的boolean值，定時保存序列化文件就直接序列化UploadPartObj ，UploadPartObj 的線程安全是至關重要的。

MyPartETag 代碼：

package cloudStorage.oss.upload;
import java.io.Serializable;
import com.aliyun.oss.model.PartETag;
/**
 * @Description: 封裝PartETag 用於序列化
 * @author: zrk
 * @time: 2015年4月1日 上午10:52:50
 */
public class MyPartETag implements Serializable {

    private static final long serialVersionUID = 1L;

    private int partNumber;

    private String eTag;


    public MyPartETag(PartETag partETag ) {
        super();
        this.partNumber = partETag.getPartNumber();
        this.eTag = partETag.getETag();
    }

    public int getPartNumber() {
        return partNumber;
    }

    public void setPartNumber(int partNumber) {
        this.partNumber = partNumber;
    }

    public String geteTag() {
        return eTag;
    }

    public void seteTag(String eTag) {
        this.eTag = eTag;
    }
}

OSSClientFactory 代碼：

package cloudStorage.basis;
import com.aliyun.oss.OSSClient;
public class OSSClientFactory {
    private static OSSClient ossClient = null;
    private OSSClientFactory() {
    }
    public static OSSClient getInstance() {
        if (ossClient == null) {
            // 可以使用ClientConfiguration對象設置代理服務器、最大重試次數等參數。
            // ClientConfiguration config = new ClientConfiguration();
            ossClient = new OSSClient(Constant.OSS_ENDPOINT,Constant.ACCESS_ID, Constant.ACCESS_KEY);
        }
        return ossClient;
    }
}

Constant 代碼：

package cloudStorage.basis;

import cloudStorage.service.OSSConfigService;

/**
 * @Description: 
 * @author: zrk
 * @time: 2015年4月1日 下午5:22:28
 */
public class Constant {
    public static String OSS_ENDPOINT = "http://oss.aliyuncs.com/";
    public static String ACCESS_ID;
    public static String ACCESS_KEY;
    public static Integer DOWNLOAD_PART_SIZE ; // 每個上傳Part的大小
    public static Integer UPLOAD_PART_SIZE ; // 每個上傳Part的大小
    public static int CONCURRENT_FILE_NUMBER ; // 併發文件數。
    public static int SINGLE_FILE_CONCURRENT_THREADS ; // 單文件併發線程數。
    public static int RETRY ;//失敗重試次數
    public static int SERIALIZATION_TIME;//斷點保存時間間隔(秒)
    //。。。
}

Constant中數值是加載外部配置文件，也可在這兒直接配置，個別參數值斷點下載時使用，只看上傳的話請忽略。

Global代碼：

package cloudStorage.basis;
/**
 * @Description: TODO
 * @author: zrk
 * @time: 2015年4月1日 下午5:22:46
 */
public class Global {
    public static final int SUCCESS = 1;
    public static final int ERROR = 10;
    public static final int FILE_NOT_FOUND_ERROR = 11;
    public static final int THREAD_ERROR = 12;
    public static final int NETWORK_ERROR = 13;
    public static final int OSS_SUBMIT_ERROR = 14;
//  META
    public static final String X_OSS_META_MY_MD5 = "x-oss-meta-my-md5";
}

調用上傳方式：

實例化OSSUploadFile後調用uploadFile方法：
return new OSSUploadFile(sourcePath,bucketName, key).uploadFile();

多線程上傳供大家參考，後面自己在使用過程中繼續優化。基於OSS-SDK的分塊斷點下載請參考OSS實現多文件多線程的斷點下載（java）。

本文地址：http://blog.csdn.net/zrk1000/article/details/46288927

zrk1000

發佈了27 篇原創文章 · 獲贊 49 · 訪問量 20萬+

私信關注

OSS實現多文件多線程的斷點上傳（java）

OSS實現多文件多線程的斷點上傳（java）

CORS error 但是 status code 是200 OK

壓縮上傳的GPU數據的方案

使用skopeo同步鏡像

dwz 動態構造樹

時間段判斷工具類InDateUtil

OSS實現多文件多線程的斷點上傳（java）

java 對象序列化工具類

OSS實現多文件多線程的斷點下載（java）

Mac下配置sublime實現LaTeX

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結