{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天分享的主題是微博數據庫資源調度平臺的架構實踐，是我們內部啓動的⼀個項⽬。主要實現的功能有兩個：⼀個是"},{"type":"text","marks":[{"type":"strong"}],"text":"資源的智能調度"},{"type":"text","text":"，⼀個是"},{"type":"text","marks":[{"type":"strong"}],"text":"成本優化"},{"type":"text","text":"。都是⾮常實⽤的場景。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"⼀會⼉我會從4個⽅⾯來做下具體介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"項⽬啓動的背景。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"包括微博當前數據庫資源的規模現狀和運維⼈員配⽐，我們對資源治理的期望和⽬標，以及我們對資源調度的理解。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"然後介紹⼀下這個調度平臺的宗旨和使命。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"就是我們希望通過這個平臺達到什麼⽬的，希望它解決我們哪些痛點。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"再重點介紹⼀下平臺的架構選型、實現⽅式、邏輯設計和核⼼功能。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"最後分享兩個案例。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"來說明我們是如何通過這個調度平臺實現應對熱點事件和成本優化的。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、背景"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、微博資源規模現狀"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先來看下微博資源規模的現狀："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3d\/3d2d3cc889895994f36c46231dde3b34.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從兩個維度來看：⼀個是整體資源體量，⼀個是資源使⽤的種類。整體資源體量上，服務器規模還是⽐較⼤的，全⽹服務器達到了萬臺⽔平，總實例數近10萬個，總訪問量在萬億級別，資源使⽤的種類也⽐較豐富多樣，關係型數據庫有MySQL、PostgreSQL，緩存類有Redis、Memcached，消息隊列有Kafka、MCQ、Qservice等等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"運維⼈員配⽐⾮常低，或者說⼈均運維實例數⾮常多，差不多每個⼈要管理1500個實例，這個數量已經⾮常⼤了，如果沒有藉助⾃動化、標準化的運維⼿段，是很難管理好這麼多實例的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、如何做好資源治理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如此規模下如何做好資源治理是我們每天都在考慮的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/9d\/9d61a1267e67559abcc895e5dbf16ac0.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源治理是個⽐較抽象、概括的說法，我們需要拿很多東⻄界定它才能讓它具像化。⼤家可以按照這個思路來捋下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1）資源具體指什麼（定義）"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務器，MySQL、Redis等這樣的服務，服務發現、域名，Agent代理"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源治理的⽬標是什麼（⽬標）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"穩定性保障"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"快速暴露問題，快速解決問題"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"挖掘潛在隱患，制定應對預案"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可控性保障"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源SLA可控"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源調度可控"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源容量⽔位可控"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2）怎麼治理（⽅法論）"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過資源調度實現"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3）怎麼衡量資源治理的好壞（SLA）"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"滿⾜業務的SLA"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源治理⾥⾯也包含很多細分領域和模塊，⽐如數據採集、指標決策等等。這些問題都找到答案後，我們對什麼是資源治理就有個⼤概的輪廓了。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3、什麼是資源調度"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上⾯提到 資源調度是資源治理的具體實現，我們要想做好資源治理，就要做好資源調度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/9c\/9c9a29f6d108fc29566bd4c5b54ffc78.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們⽇常對資源的很多操作，都可以抽象成對資源的調度。⽐如：切主、升級、遷移、擴容、縮容等等。資源調度的核⼼是服務⾃愈。只有服務⾃愈覆蓋範圍⾜夠⼴，資源運維的規模纔可能⾜夠⼤，服務訪問質量纔可控，資源治理纔可以真正落地。我們⼈均運維的1500個實例，就像運維15個實例⼀樣，因爲99%的情況都被服務⾃愈覆蓋了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天要介紹的這個資源調度平臺叫Shanks，它把我們⽇常運維中遇到的資源和操作都抽象成標準化的類，然後和周邊⽣態⼀起保障資源治理有序進⾏。周邊⽣態有很多，⽐如恢復中⼼、資源雲、指標決策系統等等。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Shanks資源調度平臺"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、Shanks整體架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在介紹⼀下Shanks的整體架構，可以分成四個部分："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/0e\/0e3d413c5c10b5530a476ec44f8cc821.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中Shanks Server和Shanks Agent的代碼是放在⼀個⼯程下的，上線的時候，會同時⽣成Shanks Server和Shanks Agent兩個⼆進制包。然後Shanks Server會部署在指定的服務器上，Shanks Agent會部署在所有服務器上，所有服務器上的Agent都是⼀樣的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Shanks Agent負責metric指標採集、上報給Shankks Server和指標決策系統（採集包括服務器本身的指標，還有上⾯服務的指標）。指標決策提供監控看板和基於採集指標的分析數據。同時，Shanks Server下發的命令，可以通過Agent直接執⾏。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Shanks Server是主要的資源調度者，調度的接⼝通過Http API暴露出來，Agent收到Server的指令後，實施具體的調度內容。⽐如遷移某個實例，Server會發出遷移指令，然後Server先執⾏找機器邏輯，找到⽬標服務器。讓⽬標服務器上的Agent執⾏新實例部署，搭建主從關係，上線。然後讓源服務器上的Agent下線⽼實例，遷移結束。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、五個主要功能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下⾯要介紹的是Shanks的五個主要功能："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/09\/092389e6ff30603fc193eb228efe6010.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7c\/7c87cd31509fb2b3fb4b81b19abe4fb7.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/54\/54e26d894e9a53a3502676ef8407e75c.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/6f\/6fe3f9989c3920d7da9532d8cbb31f0b.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/da\/da62fc811b5c40330daa3940e86eed30.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中彈性調度可以從兩個維度來理解：⼀個實例維度，⼀個配置維度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實例維度是指當某個具體實例冗餘度不⾜的時候，彈性調度會通過擴容實例個數來增加冗餘度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置維度是指，當某個具體實例已⽤內存快達到配置的最⼤內存的時候，彈性調度會通過修改配置增加最⼤內存。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"案例分享"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當某個資源實例觸發了報警事件，這個實例就會被打標，我們通過標籤就可以檢索出哪些實例觸發了報警閾值，哪些指標有問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/37\/3791646d65dbe453f08c18400fc06ec4.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"成本優化還是從兩個⽅⾯來把控：⼀個是提⾼資源利⽤率，降低冗餘度；⼀個是提⾼資源智能調度的能⼒或者彈性擴縮容的能⼒。這兩點做好，就能把成本優化到很理想的狀態。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d4\/d4c769ddb68b77a1afdcb1fc0f211887.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Q&A"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q1：若干個計算任務如何分配給若干個服務器，使得所用的服務器數量儘可 能少?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A1 ："},{"type":"text","text":"對於計算任務的調度和我們今天分享的 對於資源的調用有點類似，都 是通過調度邏輯合理利用已有的物理資源(CPU、內存、存儲等等)。我們採 用的策略是預分配策略，根據agent採集上報的指標信息，我們很容易知道當前 物理資源的佔用情況，我們可以根據這個情況，來對需要調度的資源進行調 度，但是調度採用預先分配的策略，比如我計劃把資源a(這裏的資源可以理解 成具體的服務，比如mysql、redis，或者具體的計算任務)部署到物理空間B 上，這個資源a需要多少物理資源我是預先在物理空間B上標記好，下次再進行 調度的時候我看到物理空間B的資源(CPU、內存、存儲)剩餘就是已經減去 資源a的了。對於計算任務也是類似的場景。這個策略的遷移是需要做好資源隔 離。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q2：聽完分享，感覺shanks很像 service mesh，方便說下選擇自研的考慮嗎?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A2 : "},{"type":"text","text":"service mesh是這幾年比較新的東⻄，service mesh裏面涉及到的服 務發現的場景確實和今天談到的服務發現很像。提供這種功能的工具確實有很 多，比如consul、etcd，我們選擇自研的原因很簡單，一是本身實現起來不 難，二是我們運維的資源類型很多，有些定製的需求，自研可能更容易實現。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q3：服務效率的評價標準應該怎麼定呢?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A3 : "},{"type":"text","text":"服務效率可能從投入產出比來說會好些，比如我投入10個人，開發了 3個月，做出了這套調度系統，我們利用這個調度系統，每年可以節省非常可觀 的開銷，這中投入產出比或者服務效率是容易讓人接受的。如果我投入10個開 發了2年才做出了這套系統，我們利用這個系統，每年節省的開銷很有限，那我 們就認爲服務效率不夠好。我們在項目啓動的時候確實衡量過這個成效，原則 就是儘可能快的做出能覆蓋80、90%的場景。然後快速迭代。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q4：請問下老師有哪些常用的資源調度策略?優缺點可以簡單說一下嗎?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A4 : "},{"type":"text","text":"模式大體可以分爲兩種，一種是隔離類的資源調度，比如資源綁定， 資源容器化;一種的非隔離類的調度，比如我把很多資源部署在一臺物理服務 器上。調度策略上，我們大體有幾個原則:一是平衡策略，一是近地緣策略。平衡策略就是，把資源使用率高上面的服務調度到資源使用率相對低的地方。近地緣策略是指，優先選擇同機房、同可用區的服務器作爲目標服務器。我們 在選擇目標服務器的時候有一套打分機制就是基於上面的策略來做的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q5：跨雲怎麼進行統一資源調度?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A5 : "},{"type":"text","text":"跨雲的場景現在已經很常⻅了，我們也同時用了多個廠商的雲資源。每個廠商都會提供通用的API來實現相應的功能，所以從調度上看沒什麼區 別，唯一的區別是跨雲的網絡延時會有點⻓，控制好這個就ok了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q6：你們團隊的組織架構能否介紹下哈?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A6 ："},{"type":"text","text":"我所在的部⻔是基礎平臺，裏面可以分爲應用運維、資源運維、大數 據運維、基礎組件這幾個大的方向。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q7：請問老師，如何劃分資源粒度呢?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A7 : "},{"type":"text","text":"一個資源實例或者一個進程是我們調度的最小資源粒度，比如一個 mysql實例。比資源實例大一個級別的是一組實例，比如一個mysql集羣，裏面 可能有幾個實例，比一組實例大一個級別的是一類資源，比如全網所有的 mysql。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"講師介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"張磊，"},{"type":"text","text":"新浪微博 基礎平臺資深DBA "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前必要商城、每⽇優鮮、達令家⾼級DBA，負責數據庫資源的⽇常維護和治理，完成從0到1數據庫⾃動化運維平臺的建設，推進數據庫資源架構標準化和分佈式使⽤姿勢落地；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"⽬前擔任新浪微博資深數據庫⼯程師，負責微博數據庫資源的運維保障和⾃動化運維平臺的重構、優化和完善。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：dbaplus社羣（ID：dbaplus）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/l3sYfQYwry1TogBCBA6N0w","title":"xxx","type":null},"content":[{"type":"text","text":"新浪微博數據庫資源調度設計：萬級實例精細化管理"}]}]}]}
新浪微博數據庫資源調度設計：萬級實例精細化管理
{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天分享的主題是微博數據庫資源調度平臺的架構實踐，是我們內部啓動的⼀個項⽬。主要實現的功能有兩個：⼀個是"},{"type":"text","marks":[{"type":"strong"}],"text":"資源的智能調度"},{"type":"text","text":"，⼀個是"},{"type":"text","marks":[{"type":"strong"}],"text":"成本優化"},{"type":"text","text":"。都是⾮常實⽤的場景。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"⼀會⼉我會從4個⽅⾯來做下具體介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"項⽬啓動的背景。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"包括微博當前數據庫資源的規模現狀和運維⼈員配⽐，我們對資源治理的期望和⽬標，以及我們對資源調度的理解。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"然後介紹⼀下這個調度平臺的宗旨和使命。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"就是我們希望通過這個平臺達到什麼⽬的，希望它解決我們哪些痛點。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"再重點介紹⼀下平臺的架構選型、實現⽅式、邏輯設計和核⼼功能。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"最後分享兩個案例。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"來說明我們是如何通過這個調度平臺實現應對熱點事件和成本優化的。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、背景"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、微博資源規模現狀"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先來看下微博資源規模的現狀："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3d\/3d2d3cc889895994f36c46231dde3b34.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從兩個維度來看：⼀個是整體資源體量，⼀個是資源使⽤的種類。整體資源體量上，服務器規模還是⽐較⼤的，全⽹服務器達到了萬臺⽔平，總實例數近10萬個，總訪問量在萬億級別，資源使⽤的種類也⽐較豐富多樣，關係型數據庫有MySQL、PostgreSQL，緩存類有Redis、Memcached，消息隊列有Kafka、MCQ、Qservice等等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"運維⼈員配⽐⾮常低，或者說⼈均運維實例數⾮常多，差不多每個⼈要管理1500個實例，這個數量已經⾮常⼤了，如果沒有藉助⾃動化、標準化的運維⼿段，是很難管理好這麼多實例的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、如何做好資源治理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如此規模下如何做好資源治理是我們每天都在考慮的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/9d\/9d61a1267e67559abcc895e5dbf16ac0.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源治理是個⽐較抽象、概括的說法，我們需要拿很多東⻄界定它才能讓它具像化。⼤家可以按照這個思路來捋下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1）資源具體指什麼（定義）"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務器，MySQL、Redis等這樣的服務，服務發現、域名，Agent代理"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源治理的⽬標是什麼（⽬標）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"穩定性保障"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"快速暴露問題，快速解決問題"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"挖掘潛在隱患，制定應對預案"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可控性保障"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源SLA可控"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源調度可控"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源容量⽔位可控"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2）怎麼治理（⽅法論）"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過資源調度實現"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3）怎麼衡量資源治理的好壞（SLA）"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"滿⾜業務的SLA"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源治理⾥⾯也包含很多細分領域和模塊，⽐如數據採集、指標決策等等。這些問題都找到答案後，我們對什麼是資源治理就有個⼤概的輪廓了。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3、什麼是資源調度"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上⾯提到 資源調度是資源治理的具體實現，我們要想做好資源治理，就要做好資源調度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/9c\/9c9a29f6d108fc29566bd4c5b54ffc78.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們⽇常對資源的很多操作，都可以抽象成對資源的調度。⽐如：切主、升級、遷移、擴容、縮容等等。資源調度的核⼼是服務⾃愈。只有服務⾃愈覆蓋範圍⾜夠⼴，資源運維的規模纔可能⾜夠⼤，服務訪問質量纔可控，資源治理纔可以真正落地。我們⼈均運維的1500個實例，就像運維15個實例⼀樣，因爲99%的情況都被服務⾃愈覆蓋了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天要介紹的這個資源調度平臺叫Shanks，它把我們⽇常運維中遇到的資源和操作都抽象成標準化的類，然後和周邊⽣態⼀起保障資源治理有序進⾏。周邊⽣態有很多，⽐如恢復中⼼、資源雲、指標決策系統等等。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Shanks資源調度平臺"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、Shanks整體架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在介紹⼀下Shanks的整體架構，可以分成四個部分："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/0e\/0e3d413c5c10b5530a476ec44f8cc821.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中Shanks Server和Shanks Agent的代碼是放在⼀個⼯程下的，上線的時候，會同時⽣成Shanks Server和Shanks Agent兩個⼆進制包。然後Shanks Server會部署在指定的服務器上，Shanks Agent會部署在所有服務器上，所有服務器上的Agent都是⼀樣的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Shanks Agent負責metric指標採集、上報給Shankks Server和指標決策系統（採集包括服務器本身的指標，還有上⾯服務的指標）。指標決策提供監控看板和基於採集指標的分析數據。同時，Shanks Server下發的命令，可以通過Agent直接執⾏。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Shanks Server是主要的資源調度者，調度的接⼝通過Http API暴露出來，Agent收到Server的指令後，實施具體的調度內容。⽐如遷移某個實例，Server會發出遷移指令，然後Server先執⾏找機器邏輯，找到⽬標服務器。讓⽬標服務器上的Agent執⾏新實例部署，搭建主從關係，上線。然後讓源服務器上的Agent下線⽼實例，遷移結束。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、五個主要功能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下⾯要介紹的是Shanks的五個主要功能："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/09\/092389e6ff30603fc193eb228efe6010.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7c\/7c87cd31509fb2b3fb4b81b19abe4fb7.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/54\/54e26d894e9a53a3502676ef8407e75c.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/6f\/6fe3f9989c3920d7da9532d8cbb31f0b.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/da\/da62fc811b5c40330daa3940e86eed30.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中彈性調度可以從兩個維度來理解：⼀個實例維度，⼀個配置維度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實例維度是指當某個具體實例冗餘度不⾜的時候，彈性調度會通過擴容實例個數來增加冗餘度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置維度是指，當某個具體實例已⽤內存快達到配置的最⼤內存的時候，彈性調度會通過修改配置增加最⼤內存。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"案例分享"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當某個資源實例觸發了報警事件，這個實例就會被打標，我們通過標籤就可以檢索出哪些實例觸發了報警閾值，哪些指標有問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/37\/3791646d65dbe453f08c18400fc06ec4.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"成本優化還是從兩個⽅⾯來把控：⼀個是提⾼資源利⽤率，降低冗餘度；⼀個是提⾼資源智能調度的能⼒或者彈性擴縮容的能⼒。這兩點做好，就能把成本優化到很理想的狀態。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d4\/d4c769ddb68b77a1afdcb1fc0f211887.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Q&A"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q1：若干個計算任務如何分配給若干個服務器，使得所用的服務器數量儘可 能少?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A1 ："},{"type":"text","text":"對於計算任務的調度和我們今天分享的 對於資源的調用有點類似，都 是通過調度邏輯合理利用已有的物理資源(CPU、內存、存儲等等)。我們採 用的策略是預分配策略，根據agent採集上報的指標信息，我們很容易知道當前 物理資源的佔用情況，我們可以根據這個情況，來對需要調度的資源進行調 度，但是調度採用預先分配的策略，比如我計劃把資源a(這裏的資源可以理解 成具體的服務，比如mysql、redis，或者具體的計算任務)部署到物理空間B 上，這個資源a需要多少物理資源我是預先在物理空間B上標記好，下次再進行 調度的時候我看到物理空間B的資源(CPU、內存、存儲)剩餘就是已經減去 資源a的了。對於計算任務也是類似的場景。這個策略的遷移是需要做好資源隔 離。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q2：聽完分享，感覺shanks很像 service mesh，方便說下選擇自研的考慮嗎?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A2 : "},{"type":"text","text":"service mesh是這幾年比較新的東⻄，service mesh裏面涉及到的服 務發現的場景確實和今天談到的服務發現很像。提供這種功能的工具確實有很 多，比如consul、etcd，我們選擇自研的原因很簡單，一是本身實現起來不 難，二是我們運維的資源類型很多，有些定製的需求，自研可能更容易實現。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q3：服務效率的評價標準應該怎麼定呢?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A3 : "},{"type":"text","text":"服務效率可能從投入產出比來說會好些，比如我投入10個人，開發了 3個月，做出了這套調度系統，我們利用這個調度系統，每年可以節省非常可觀 的開銷，這中投入產出比或者服務效率是容易讓人接受的。如果我投入10個開 發了2年才做出了這套系統，我們利用這個系統，每年節省的開銷很有限，那我 們就認爲服務效率不夠好。我們在項目啓動的時候確實衡量過這個成效，原則 就是儘可能快的做出能覆蓋80、90%的場景。然後快速迭代。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q4：請問下老師有哪些常用的資源調度策略?優缺點可以簡單說一下嗎?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A4 : "},{"type":"text","text":"模式大體可以分爲兩種，一種是隔離類的資源調度，比如資源綁定， 資源容器化;一種的非隔離類的調度，比如我把很多資源部署在一臺物理服務 器上。調度策略上，我們大體有幾個原則:一是平衡策略，一是近地緣策略。平衡策略就是，把資源使用率高上面的服務調度到資源使用率相對低的地方。近地緣策略是指，優先選擇同機房、同可用區的服務器作爲目標服務器。我們 在選擇目標服務器的時候有一套打分機制就是基於上面的策略來做的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q5：跨雲怎麼進行統一資源調度?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A5 : "},{"type":"text","text":"跨雲的場景現在已經很常⻅了，我們也同時用了多個廠商的雲資源。每個廠商都會提供通用的API來實現相應的功能，所以從調度上看沒什麼區 別，唯一的區別是跨雲的網絡延時會有點⻓，控制好這個就ok了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q6：你們團隊的組織架構能否介紹下哈?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A6 ："},{"type":"text","text":"我所在的部⻔是基礎平臺，裏面可以分爲應用運維、資源運維、大數 據運維、基礎組件這幾個大的方向。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q7：請問老師，如何劃分資源粒度呢?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"A7 : "},{"type":"text","text":"一個資源實例或者一個進程是我們調度的最小資源粒度，比如一個 mysql實例。比資源實例大一個級別的是一組實例，比如一個mysql集羣，裏面 可能有幾個實例，比一組實例大一個級別的是一類資源，比如全網所有的 mysql。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"講師介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"張磊，"},{"type":"text","text":"新浪微博 基礎平臺資深DBA "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前必要商城、每⽇優鮮、達令家⾼級DBA，負責數據庫資源的⽇常維護和治理，完成從0到1數據庫⾃動化運維平臺的建設，推進數據庫資源架構標準化和分佈式使⽤姿勢落地；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"⽬前擔任新浪微博資深數據庫⼯程師，負責微博數據庫資源的運維保障和⾃動化運維平臺的重構、優化和完善。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：dbaplus社羣（ID：dbaplus）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/l3sYfQYwry1TogBCBA6N0w","title":"xxx","type":null},"content":[{"type":"text","text":"新浪微博數據庫資源調度設計：萬級實例精細化管理"}]}]}]}
發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章