乾貨 | 攜程 Web CI/CD 實踐

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在攜程的日常Web開發生命週期中，本地代碼開發階段可通過NFES框架（攜程內部一個支持SSR框架，其中還包含許多公共基礎業務模塊及UI組件）來快速完成項目需求。但開發完代碼之後常常會遇到如下幾點問題："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"環境問題：Web\/Node資源本地構建\/測試環境和生產環境差異化大，導致有些問題無法及時發現 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"QA流程：Web\/Node端提交代碼流程沒有規範的QA環節，代碼質量不可控"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"構建流程：資源本地構建與鏡像構建是獨立的，版本易混淆 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"代碼開發完後的各個步驟比較分散，難集中管理"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、目標"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過引入CI\/CD解決方案，建立完善的準備環境\/測試\/資源構建\/鏡像構建一整個流程的鏈路，使它可幫助項目以更快的速度和更高的質量來交付。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、實現與實踐"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NFES 的 Web CI\/CD 的實現，簡單來說就是通過管道化 (GitDev Pipeline) 的執行過程來完成持續集成和持續交付，這篇文章先不涉及持續部署。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其管道 (Pipeline) 中集成QA，資源構建，生成鏡像等多個Stage，而每個Stage中都包含詳細的Step來完成其功能。接下來我們來詳細從管道 (Pipeline) 中的Stage\/Step的角度來介紹下NFES的Web CI\/CD。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"管道在這裏可以理解爲實現目標的頂層組件，整個NFES Web CI\/CD就是這樣的組件組合而成。目前Web\/Node相關的管道分爲三個Stage："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/cf\/cf80d27fd9aec43a85a1402e30196443.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）Install Stage"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"a. Install Step，安裝用戶項目下的依賴模塊"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）Verify Stage"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏集成了三個Step："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"a. Lint Step，靜態代碼檢測"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"b. Test Step，單元測試\/UI測試"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"c. Build Step，資源構建"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3）SonarAndImage Stage"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"a. Sonar Step，Sonar代碼檢測並上傳，此步驟依賴於Verify Stage中的Lint\/Test Step"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"b. Image Step，構建Docker鏡像，此步驟依賴於Verify Stage中的Build Step"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面三個Stage是依次順序執行，而在同個Stage中的多個Step則是併發執行的。這些執行順序的控制可通過編寫.gitlab-ci.yml文件來完成。這裏先簡單介紹下.gitlab-ci.yml CI\/CD配置的編寫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":".gitlab-ci.yml是放在倉庫根目錄中的文件，默認倉庫會去這個文件中讀取CI\/CD的相關配置。在此文件配置中你可以定義如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"定義環境變量 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"需要順序或者並行運行的腳本命令 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前後Step依賴關係 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此Step所需使用緩存和設置緩存 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"觸發的條件分支"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具體常用配置代碼如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\n#配置所需的基礎鏡像地址\nimage: xxxxxxxxx\n\n#配置相關變量\nvariables:\n PROXY: http:\/\/proxy\n HTTP_PROXY: $PROXY\n\n#配置Stages的名稱及順序\nstages:\n - Install\n - Verify\n\n......\n# Install Stage的詳細配置\nTest: #Step的名稱\n stage: Verify #屬於哪個Stage\n artifacts: #配置產物存檔文件,可在Pipeline運行界面取到配置的文件,但此存檔只能保存默認一週\n paths:\n - reports\/\n exclude: #忽略某些文件不作爲產物存檔文件\n - .git\n - .git\/**\n when: always \n cache: #配置緩存\n key: keyName\n paths:\n - node_modules #所需緩存的文件\/文件夾\n policy: pull #如需獲取緩存的文件,這裏定製policy屬性爲pull\n allow_failure: true #此步驟是否允許失敗,如果允許,即使步驟執行失敗,仍舊可執行下個Stage\n dependencies: #配置此Step依賴哪個Step\n - Install\n script: #配置所需執行的命令\n - cd \/testFolder \n - node index\n only:\n - master #配置分支 只有配置的分支纔會執行相關的Pipeline\n\n......"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在日常開發使用中，攜程的GitDev CI\/CD則提供公用的配置模版，如用戶沒有特殊Step的需求，可通過選擇Step模版或者選擇應用類型模版來自動生成上面的配置文件，無需關注yml的詳細配置。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來我們詳細看下NFES Web CI\/CD的Install，Verify和SonarAndImage三個Stage做了哪些事情？"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.1 Install Stage"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Install Stage中只包含一個Step，即執行安裝用戶項目下的模塊依賴。此階段安裝結束後的nodemodules則會作爲緩存給之後的Step使用，可節省很多不必要的重複安裝模塊的時間。當然如果在同一個commitID的情況下，多次執行這個Install Stage，則後面幾次安裝的nodemodules其實就是取第一次安裝的緩存。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.2 Verify Stage"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Verify Stage默認會包含三個步驟Lint，Test，Build。這個Stage其實是一個規範的QA環節，而Build的Step爲什麼要放在此處，就是想構建與測試併發執行，從而縮短整個Pipeline的運行時間。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"詳細的各個Step的實現如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）Lint Step集成了eslint靜態代碼檢測功能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"靜態代碼檢測功能通過封裝的全局模塊來完成代碼檢測，其默認使用eslint:recommended推薦規則。如用戶需要自定義eslint規則可以直接把規則寫在當前項目的eslintrc.json文件中，模塊會自動整合其默認規則。如想要忽略檢查某些文件，則把規則寫在.eslintignore文件中。執行完成後會生成eslint-report.json，此文件會作爲artifacts可在pipeline的step任務頁面中直接下載查看，也會通過後面的Sonar Step上傳到Sonar。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）Test Step集成了單元測試以及UI測試 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"集成的單測框架又可分爲mocha和jest，Web端統一使用jest，NFES測試鏡像中默認已有jest相關模塊，如無特殊需求則不需要在用戶項目的依賴中安裝測試相關依賴的模塊。如需自定義jest相關配置可寫在用戶項目下的jest.config.js中。單元測試的運行命令統一爲：npm run test，其執行結果會以html\/json\/clover\/lcov輸出，輸出結果中lcov和clover.xml文件與GitDev做集成，使其結果與代碼的commitID進行綁定，這樣每次代碼提交就可在界面上直接查看本次提交代碼的具體單測運行結果。這裏也可設置對每次代碼提交的單元測試覆蓋率的要求，如其覆蓋率不低於60%，否則不能進行下一步驟。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每次代碼提交的CommitID的單元測試結果展示如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/5d\/5d481a76e82816ab88373f6d69d38fbc.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"集成的UI測試是作爲一個可選Step，我們提供了集成puppeteer\/cucumber的鏡像，用戶如有UI測試的需求可自行在Test Stage中添加該UI Test Step。在UI測試中增加了視頻錄製的功能，每個Case對應一個視頻，等用戶的UI Cases執行完成後，則會自動生成報表併發布到資源站點上，方便用戶查看及排查問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"UI測試報表結果中錄製視頻（部分截圖）展示如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b7\/b748012af0244d590d3a62f4c5c53908.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3）Build Step集成頁面的資源構建 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏的構建其實就是把在線構建搬到了Pipeline的Build Step中。首先是構建環境的搭建，分爲兩塊：框架所需的依賴模塊環境和用戶項目依賴的模塊環境。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於NFES框架的依賴模塊環境，Build Step使用的構建鏡像中已經集成了NFES項目所需的開發態模塊（我們對開發態模塊加載做了些優化，把如Babel插件，webpack，loader等通用的模塊全都集成到cli的全局模塊中，然後預裝到構建鏡像)。執行構建時，更改構建時項目所需開發態模塊路徑指向預裝路徑，這樣就可以不需要安裝框架依賴模塊。而對於用戶項目依賴的模塊環境則可以重用Install Stage中的node_modules中的緩存，這兩點使用戶項目安裝模塊的時間大幅度減少。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"搭建完構建環境後，執行相關在線構建命令開始構建，構建的過程及日誌都可通過Pipeline界面得到。構建完成後接下來是構建產物的處理。這裏的NFES項目構建產物可分爲Web端資源\/node服務端資源。Web端的資源可以直接發佈並獲得相應的資源地址，此Web資源地址也會及時更新到node服務端資源中的資源路徑。最後通過配置中artifacts屬性來確定哪些node端的資源文件需要上傳給下一步Image Stage來構建發佈鏡像。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.3 SonarAndImage Stage"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SonarAndImage包含了Sonar和Image兩個Step， 這個Stage是目前管道中最後一個專門收集與處理前面依賴Step產物的Stage。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）Sonar Step "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此步驟是依賴於Test和Lint這兩個Step， 用來收集依賴的這兩個Step執行的結果並上傳至Sonar中。用戶可以在sonarqube的網站查看歷史的代碼質量報告。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）Image Step "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此步驟是依賴於Build Step，它是獲取Build的構建產物與基礎鏡像一起構建出發佈鏡像並推送到Hub中，爲接下來的應用發佈做準備。到此步驟整個NFES Web CI\/CD的流程就結束了。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、小結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上就是整個NFES Web CI\/CD的實現與實踐。目前幾乎所有的NFES項目都已經切到CI\/CD的流程上，它帶來了集中式流程化管理，一站式對用戶透明的資源構建與鏡像構建更簡單快捷，開發效率得到了很大的提高。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者簡介"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"西傑，攜程軟件技術專家，關注前端技術及其生態，致力於提升前端開發效能及質量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：攜程技術（ID：ctriptech）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/sIENp-pPjBPwWTJKZcy-vg","title":"xxx","type":null},"content":[{"type":"text","text":"乾貨 | 攜程 Web CI\/CD 實踐"}]}]}]}