自定義線程池來實現文檔轉碼

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我司在很久之前，一位很久之前的同事寫過一個文檔轉圖片的服務，具體業務如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"用戶在客戶端上傳文檔，可以是ppt，word，pdf 等格式，用戶上傳完成可以在客戶端預覽上傳的文檔，預覽的時候採用的是圖片形式（不要和我說用別的方式預覽，現在已經來不及了）"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"當用戶把文檔上傳到雲端之後（阿里雲），把文檔相關的信息記錄在數據庫，然後等待轉碼完成"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"服務器有一個轉碼服務（其實就是一個windows service）不停的在輪訓待轉碼的數據，如果有待轉碼的數據，則從數據庫取出來，然後根據文檔的網絡地址下載到本地進行轉碼（轉成多張圖片）"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"當文檔轉碼完畢，把轉碼出來的圖片上傳到雲端，並把雲端圖片的信息記錄到數據庫"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"客戶端有預覽需求的時候，根據數據庫來判斷有沒有轉碼成功，如果成功，則獲取數據來顯示。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"文檔預覽的整體過程如以上所說，老的轉碼服務現在什麼問題呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"由於一個文檔同時只能被一個線程進行轉碼操作，所以老的服務採用了把待轉碼數據劃分管道的思想，一共有六個管道，映射到數據庫大體就是 Id=》管道ID 這個樣子。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"一個控制檯程序，根據配置文件信息，讀取某一個管道待轉碼的文檔，然後單線程進行轉碼操作"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"一共有六個管道，所以服務器上起了六個cmd的黑窗口......"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"有的時候個別文檔由於格式問題或者其他問題 轉碼過程中會卡住，具體的表現爲：停止了轉碼操作。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"如果程序卡住了，需要運維人員重新啓動轉碼cmd窗口（這種維護比較蛋疼）"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"後來機緣巧合，這個程序的維護落到的菜菜頭上，維護了一週左右，大約重啓了10多次，終於忍受不了了，重新搞一個吧。仔細分析過後，刨除實際文檔轉碼的核心操作之外，整個轉碼流程其實還有很多注意點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"需要保證轉碼服務不被卡住，如果和以前一樣就沒有必要重新設計了"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"儘量避免開多個進程的方式，其實在這個業務場景下，多個進程和多個線程作用是一致的。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"每個文檔只能被轉碼一次，如果一個文檔被轉碼多次，不僅浪費了服務器資源，而且還有可能會有數據不一致的情況發生"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"轉碼失敗的文檔需要有一定次數的重試，因爲一次失敗不代表第二次失敗，所以一定要給失敗的文檔再次被操作的機會"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"因爲程序不停的把文檔轉碼成本地圖片，所以需要保證這些文件在轉碼完成在服務器上刪除，不然的話，時間長了會生成很多無用的文件"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"說了這麼多，其實需要注意的點還是很多的。以整個的轉碼流程來說，本質上是一個任務池的生產和消費問題，任務池中的任務就是待轉碼的文檔，生產者不停的把待轉碼文檔丟進任務池，消費者不停的把任務池中文檔轉碼完成。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"link","attrs":{"href":"#線程池","title":null}},{"type":"text","text":"線程池"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這很顯然和線程池很類似，菜菜之前就寫過一個線程池的文章，有興趣的同學可以去翻翻歷史。今天我們就以這個線程池來解決這個轉碼問題。線程池的本質是初始化一定數目的線程，不停的執行任務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":" //線程池定義 \n public class LXThreadPool:IDisposable\n {\n bool PoolEnable = true; //線程池是否可用 \n List ThreadContainer = null; //線程的容器\n ConcurrentQueue JobContainer = null; //任務的容器\n int _maxJobNumber; //線程池最大job容量\n\n ConcurrentDictionary JobIdList = new ConcurrentDictionary(); //job的副本，用於排除某個job 是否在運行中\n\n\n public LXThreadPool(int threadNumber,int maxJobNumber=1000)\n {\n if(threadNumber<=0 || maxJobNumber <= 0)\n {\n throw new Exception(\"線程池初始化失敗\");\n }\n _maxJobNumber = maxJobNumber;\n ThreadContainer = new List(threadNumber);\n JobContainer = new ConcurrentQueue();\n for (int i = 0; i < threadNumber; i++)\n {\n var t = new Thread(RunJob);\n t.Name = $\"轉碼線程{i}\";\n ThreadContainer.Add(t);\n t.Start();\n }\n //清除超時任務的線程\n var tTimeOutJob = new Thread(CheckTimeOutJob);\n tTimeOutJob.Name = $\"清理超時任務線程\";\n tTimeOutJob.Start();\n }\n\n //往線程池添加一個線程,返回線程池的新線程數\n public int AddThread(int number=1)\n {\n if(!PoolEnable || ThreadContainer==null || !ThreadContainer.Any() || JobContainer==null|| !JobContainer.Any())\n {\n return 0;\n }\n while (number <= 0)\n {\n var t = new Thread(RunJob);\n ThreadContainer.Add(t);\n t.Start();\n number -= number;\n }\n return ThreadContainer?.Count ?? 0;\n }\n\n //向線程池添加一個任務,返回0：添加任務失敗 1：成功\n public int AddTask(Action