騰訊遊戲運維技術發展：走向 AIops ，運維與業務運營加速融合

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"伴隨着移動互聯網浪潮的發展崛起，國產手遊巧妙的避開了長期被日、韓、歐美等國家佔領的 PC遊戲市場，快速打造了基於移動智能設備的全新手遊體驗，在全球市場上取得突破性進展，並且形成了自己的影響力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然而在這一過程中，隨着越來越多的國內遊戲產品開始走向全球，參與引領行業發展，其對於後臺技術的發展也提出了越來越高的要求。"},{"type":"text","marks":[{"type":"strong"}],"text":"其中，運維技術作爲任意款遊戲產品運行發展的強有力支撐單元，正面臨着全新的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲國內遊戲行業市場佔有率最高的企業，騰訊遊戲業務長期佔據國內市場總份額50%以上，穩居行業第一，"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#191919","name":"user"}}],"text":"騰訊遊戲運維業務的發展更是可以進一步看做是整個行業發展的縮影。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#191919","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在由 InfoQ 主辦的 "},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2020\/shanghai?utm_source=infoq&utm_medium=conference","title":"xxx","type":null},"content":[{"type":"text","text":"2020 QCon （上海）全球軟件開發大會"}]},{"type":"text","text":"期間，InfoQ 記者特別採訪了騰訊遊戲運維總監塗彥先生，就騰訊遊戲運維技術發展演進過程是怎樣的？新時期的AIops運維技術實踐具有哪些經驗等方面話題？進行了交流。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"以下爲Q\/A內容整理（有刪減），供行業人士參考。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"從手工操作到智能運維"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ："},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"從事遊戲運維近15年，過程中哪些運維技術、理念的出現讓你印象深刻？騰訊遊戲在運維方面是如何參與這些變化的？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"單純從國內遊戲運維領域技術發展演進的角度來看，在經歷了早期的純手工化運維，再到後來的基於 Command 窗口命令行、Linux 腳本批量化管理等發展階段後，如今遊戲行業運維技術的發展已經從腳本過渡到一個工具化、產品化的階段，並正呈現出向智能化過度的趨勢。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以騰訊爲例，從2012年開始，騰訊遊戲內部便把運維腳本語言統一升級爲 Python，實現了整個遊戲的日常運維工作全面標準化與自動化升級。現在我們只需要使用藍鯨這一個平臺工具，就可以實現騰訊遊戲旗下幾百個業務的自動化運維管理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與此同時，伴隨着運維工作的自動化持續建設，騰訊遊戲內部的服務器數量多達數十萬臺，遊戲業務對於數據服務的使用也越來越多。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在服務器數量達到了一定規模的量級之後，每一臺服務器各自不同的生命週期、硬件故障、以及服務器內部結構等數據信息都存在差異，如何對這些數據進行科學合理的管理以及資源分配成爲了新的挑戰。與此同時，不同遊戲玩家在體驗方面對於高效實時性能等方面的要求，同樣對騰訊遊戲的運維工作提出了新的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"智能化運維（AIops）工作的開展，成爲了騰訊遊戲運維真實業務開展過程中不得不去思考的一個問題。與之相對應的，我們在人員甄選方面，也出現了更多運維開發、數據開發這樣的新能力，強調新進入的人員在懂運維技術的同時，還要懂開發技術或智能化技術。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：你是怎麼理解AIOps的？能否通過一個例子形象直觀的展開介紹？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以目前最火爆的生存類遊戲爲例，當大量的玩家進入遊戲場景的時候，其實對於遊戲策劃人員來說，他們需要去了解這張地圖設計的是否合理？是否存在值得改進的地方，於是就需要通過很多的數據去分析。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在傳統工作模式中，對於這些數據分析，是離線以及單維度的，策劃團隊很難立體去看玩家在遊戲中的實際表現。而如果可以立體，實時，多維去看這些，將給開發團隊帶來更好的感知，從而更加優化好玩家的遊戲體驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前我們通過 AIops的方式，就已經能夠實現通過熱力地圖，便真實的記錄下單位時間內地圖上的擊中分佈，比率的情況，第一時間知道哪一個區域的玩家失敗率最高？並通過分析得出改進建議。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"del"}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們就真實的幫助遊戲策劃人員發現，在一次版本更新之後，某一大型遊戲裏面一處地圖的角落裏，玩家從箱子上跳上來之後被擊中的次數最多，後來發現是這個地方的其實是需要做出調整的。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"運維與業務運營、策劃正走向融合 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ："},{"type":"text","text":"在你們看來，遊戲運維是跟運營、策劃等合在一起的？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"對的，在我們看來，傳統的AIops主要是用於解決軟硬件相關、發現故障等問題的，但是對於我們而言，我們會更傾向於怎麼樣去幫業務解決實際問題，它其實還是有一個很大的提升的。通過上面的例子可以看到，其實我們的運維業務已經跟業務相關的遊戲設計、策劃等關聯起來了，這已經區別於傳統的運維模式了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在以往的運維過程中，我們在幫助業務去解決問題的時候發現，包括一些斷線重連，遊戲卡頓等問題，其實它已經不再是簡單的通過一維或二維數據就可以快速得到答案的，現在遊戲產品的後臺運維其實需要收集更加多維、海量的數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這樣的情況下，智能運維要做的事情其實相當於是我們把運營規劃的事，從需求到最終的落地，並且把運維、開發以及運營這些事情都和在一起來做了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ："},{"type":"text","text":"運營與智能運維的結合，這個概念只有騰訊一家在提嗎？還是其他的同行也認同這個趨勢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這幾年我們在行業裏的交流其實也有不少，整體上還是覺得行業裏做得更多的是偏向於硬件和網絡相關的。比如說做硬件故障維護、磁盤替換、網絡告警。但是也有越來越多的企業開始做業務預警相關的工作，相當於說這裏面有一部分智能運維的工作是去做預測的，這一塊其實目前像百度、阿里等運維團隊也都有在做。 "}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"運維部門不是利潤中心, 而是成本中心"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：作爲這一次 QCon 大會《彈性工程與運維》專題的出品人，你參與組織這一專題的目的是什麼？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前看來，整個雲計算的發展，對運維行業也帶來了很大的轉型挑戰。"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#3E3A39","name":"user"}}],"text":"在高併發的場景下，如何做好彈性調度以提升業務的資源利用率，這是我們專題重點關注的方向，希望能給大家解決複雜的業務問題提供有效地幫助和啓發。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"遊戲業務對於後端服務器的要求非常高，特別是網絡遊戲，有大量的計算都是通過後端完成的，所以，在每一款遊戲的背後，都有大量的服務器資源基於GPU、CPU去進行運算。伴隨着雲計算背景下雲遊戲等概念的出現，用戶對於雲端遊戲的呼聲越來越大，這無形之間對新的遊戲運維提出了很高的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於雲時代產生的雲遊戲而言，由於需要達到的效果是將大量的內容存儲、圖像渲染等放置到雲端，終端只起到一個交互的作用，這導致的結果是，雲遊戲在用戶的終端設備性能很低的情況下也能產生一個很好的效果，但是對於後端的服務器的要求卻變得更高了。這相當於是將終端設備性能提升的問題轉移到了後端服務器裏面，於是與之相對應的是，後端的開發、運維人員將要面臨的問題也就更多，成本壓力也越來越大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"運維部門作爲企業的成本部門，在保障業務運行的情況下，更多的還是需要去考慮如何更好的與業務結合，產生更多的增值服務。以及思考如何通過基於雲的彈性伸縮方式實現運維成本的優化？這是我們這次專題裏重點討論的一個話題。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"推薦QCon全球軟件開發大會"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2021年4月22-24日"},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/beijing?utm_source=infoq&utm_medium=conference","title":"xxx","type":null},"content":[{"type":"text","text":"QCon全球軟件開發大會（北京站）"}]},{"type":"text","text":"再次恢復爲3天時間，會議規模在2000人左右，無論是嘉賓垂直還是話題關注度都會更強，更高。 12個技術專題，73位演講嘉賓，大會內容持續上新中，認知的夥伴可以保持關注。"},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/beijing?utm_source=infoq&utm_medium=conference","title":"xxx","type":null},"content":[{"type":"text","text":"直達官網"}]},{"type":"text","text":"查看大會詳情。"}]}]}