保障視頻連線畫質清晰且流暢，騰訊會議有這些優化實踐

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、屏幕分享場景編碼技術優化實踐"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"在視頻會議中，視頻的應用場景分兩部分：屏幕分享和攝像頭視頻。"},{"type":"text","text":"屏幕內容是由電子設備生成的圖像，攝像頭內容是由攝像頭採集的視頻，兩種視頻內容的特徵差異巨大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/e8\/1e\/e840cc100dc903e78ce044618897391e.jpg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"傳統的視頻編碼採用的是預測加變換的混合編碼結構，這種方式適合攝像頭採集的視頻。但是對於屏幕內容，它的編碼效率不太好。爲了提升壓縮效率， HEVC (High Efficiency Video Coding)針對屏幕內容專門推出了HEVC-SCC編碼標準，可以大大提升屏幕內容的編碼效率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"HEVC-SCC在HEVC原有標準的基礎上，增加了幀內塊拷貝、調色板模式、自適應色彩轉換、自適應運動矢量精度幾個編碼工具集。這幾個工具集中，"},{"type":"text","marks":[{"type":"strong"}],"text":"幀內塊拷貝（IBC）和調色板模式（Palette Mode）對視頻壓縮效率的提升最爲明顯。"},{"type":"text","text":"IBC採用當前幀已重建塊作爲預測塊，可以認爲是基於當前編碼圖像內的運動補償；Palette Mode枚舉顏色值生成顏色表，然後爲每個樣本傳遞一個索引以指示它屬於顏色表中的哪種顏色，它特別適合顏色數比較少的編碼塊。"},{"type":"text","marks":[{"type":"strong"}],"text":"根據官方統計，對於屏幕內容，在HEVC的基礎上 增加IBC和Palette Mode兩個工具集，可以帶來50%以上的壓縮效率的提升。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1. TSE兼顧壓縮效率與編碼複雜度"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"加入SCC（屏幕內容編碼技術， Screen Content Coding）雖然可以提升壓縮效率，但是同時也帶來了編碼複雜度的急劇增加。對於騰訊會議這種實時通信場景，編碼複雜度過高會影響實時性，無法得到應用。因此，我們在實現了IBC和Palette編碼後，也做了大量性能優化，推出了屏幕內容編碼器TSE (Tencent Screen Encoder)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"我們在算法實現、模式選擇、彙編優化等方面對TSE編碼器做了全方位優化："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於IBC編碼模式，採用了基於hash表搜索的運動估計代替傳統的基於塊匹配的運動估計。對於8x8塊，我們將其分成4個4x4個小塊，計算它的hash值，然後將這個塊的座標存入該hash值對應的鏈表。另外，對於hash查找的順序也做了優化，位置較近的點先計算，大大提升hash搜索速度。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於Palette編碼，最複雜的模塊就是顏色表的生成和查找顏色表，我們在YUV數據生成顏色表數據以及根據YUV數據快速查找到顏色表中最匹配的顏色數據方面做了大量優化，提升調色板編碼速度。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了IBC和Palette本身的算法優化，在模式選擇、快速算法方面也做了很多優化，比如通過幀內幀間預測結果提前退出Palette模式判決等。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外也加入了SIMD優化，進一步提升速度。 "},{"type":"text","text":"  "},{"type":"text","text":" "}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/d2\/38\/d297a55b50d2236a613343c7c3900238.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過優化之後，TSE編碼器相比開源的X265編碼器在各方面都具有明顯優勢。首先來看壓縮效率，對於攝像頭採集的數據，TSE的壓縮效率有20%左右的提升。"},{"type":"text","marks":[{"type":"strong"}],"text":"對於屏幕內容，"},{"type":"text","marks":[{"type":"strong"}],"text":"TSE的提升更加明顯，相比X265(ultrafast)的壓縮效率可以提升70%以上"},{"type":"text","text":"，對它的medium模式也可以提升50%以上。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在編碼速度方面，TSE也相比X265有明顯優勢。"},{"type":"text","marks":[{"type":"strong"}],"text":"對於屏幕內容序列，"},{"type":"text","marks":[{"type":"strong"}],"text":"TSE的編碼速度不到X265的一半"},{"type":"text","text":"，對於攝像頭序列，也是比它要快10%以上。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/d8\/af\/d843587b2978f9221701eede85feccaf.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了客觀質量數據的提升，TSE在圖像主觀質量方面的提升也是比較明顯的。左邊上面是用X265編碼的，下面是用TSE編碼的。可以看到採用X265編碼，文字周邊噪點非常多，整體質量較差。用TSE編碼之後，文字周邊非常清晰乾淨，圖像質量明顯優於x265。除了文字，TSE在紋理方面保真度方面也有優勢，右邊這兩個圖中，採用TSE編碼後，很好的保留了原圖的紋理輪廓，線條會更加細膩清晰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/5b\/fc\/5ba8ff9ebca854015cf6e6c4c7e0bffc.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2. YUV444編碼解決顏色失真、文字模糊問題 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於屏幕分享場景，除了採用SCC相關編碼技術，我們也推出了YUV444編碼技術。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們一般的視頻編碼都是採用YUV420格式，這種編碼模式會對UV色度分量進行下采樣，而下采樣會導致顏色出現失真。比如下面這張圖，上面是發送端，這張圖是彩色的，但是經過屏幕分享到接收端後，它變成了一張灰度圖。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/yy\/32\/yy300c680072cfc18fbdfe1f38c2a332.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了解決色彩失真的問題，我們在TSE裏面採用了YUV444編碼格式，來解決因爲色度降採樣導致的顏色失真。而且，根據相關資料，在YUV444編碼模式下，SCC更能發揮壓縮效率方面的優勢。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏有幾個對比圖，可以看到YUV444編碼的圖像，文字會更加清晰，顏色也會更加接近原圖。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/ac\/07\/ac4d69b8306a45a93b71c07f84138807.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、攝像頭場景編碼技術優化實踐"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1. SVC編碼"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面我們介紹了屏幕分享場景下的優化，接下來我們來介紹在攝像頭場景下的編碼優化。我們都知道視頻編碼裏有三種幀類型，I幀、P幀、B幀。在實時通訊場景下，一般是IPPP的編碼結構，逐幀參考，也就是參考前一幀，一般不會使用B幀。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"對於視頻會議場景，多人會議中各個用戶的網絡情況可能各不相同，有的人網絡比較好，有的人網絡條件差一些。如何適配不同用戶的網絡狀況，在各種網絡狀態下都能獲得一個比較好的視頻體驗呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"舉個例子，視頻會議有三個接收端，下行帶寬分別爲1M、2M和4M，那麼發送端應該編碼多大的碼率呢？如果只使用1M的碼率發送，可以滿足接收端1的要求，但是網絡比較好的接收端3本來有4M的下行帶寬，也只能收到1M的碼率，他的視頻體驗就會被影響。那麼如果發送端直接下發4M的話，接收端1可能會卡死，而它卡死之後會申請I幀，發送端發I幀，也會影響其他用戶的體驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/61\/5b\/6114c3607a296516e50a542804e4065b.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"爲了解決多人視頻會議中的這個問題，優化體驗，騰訊會議採用了"},{"type":"text","marks":[{"type":"strong"}],"text":"SVC靈活分層編碼技術。"},{"type":"text","text":"SVC採用分層參考機制，以上圖爲例，會把所有的幀分成三層，D0層只能參考D0層的幀，D1層可以參考D0層或者D1層的幀，D2層可以參考所有層的幀。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣做的好處是，如果用戶網絡比較差的話，只需要發D0層的幀過去，這樣即使只收到0、4、8號幀，也可以正常的解碼。同時對於那些網絡比較好的用戶，可以把所有的幀都發過去。SVC靈活分層編碼技術既保證了網絡好的用戶的視頻高清流暢體驗，又能保證網絡帶寬比較差的用戶的基本的視頻體驗，而有效的解決了多終端網絡適配的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏有一個採用SVC編碼和普通編碼的對比效果。左邊這個採用普通的IPPP參考模式，它基本上是卡死的，用戶體驗非常差。右邊這個採用SVC編碼，可以看到雖然相對全幀率它的幀率有些損失，但是能夠保證基本的視頻通訊體驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/02\/98\/0249bc4cb7f27d7704bd849d41137098.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2. ROI編碼 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ROI (Region of Interest) 編碼很多人比較瞭解，是基於感興趣區域的編碼。在攝像頭場景下，我們主要關注的一般還是在人臉區域，對背景區域的關注低。騰訊會議加入了人臉檢測算法和基於ROI的編碼算法，實時檢測出人臉區域，然後將它傳入編碼器，編碼器進行幀內的碼率重分配，對人臉區域增加碼率，提升編碼效果，對非ROI區域降低碼率，保證總的碼率不超出。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"視頻會議對實時性能要求很高，所以對算法的複雜度也比較敏感，所以我們研發了一套高效的檢測算法，可以把一幀1080P的圖像檢測耗時控制在0.5毫秒以內。基於ROI區域優化碼率控制算法，可以在低帶寬下提升主觀質量，在高帶寬下可以保證主觀質量基本不變的情況下，降低20%到30%左右的碼率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/9a\/b3\/9a3d3bac386fd3cc739d5bd45c61ebb3.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、靈活多變的編解碼策略"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面我們介紹了一些騰訊會議在視頻編解碼方面的優化技術。最後我想說的是，視頻編解碼的優化，不單單是編解碼內核的優化，還有與之配套的一系列編解碼策略的優化，編解碼模塊與採集，渲染，網絡等模塊是深度耦合的，只有系統級的優化，才能獲得最佳的視頻效果。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在視頻會議場景下，除了軟件編解碼，還有硬件編解碼。兩種編解碼各自的優劣非常明顯，軟件編解碼壓縮效率高，硬件編解碼功耗低，省電。在不同的場景、不同的平臺上，需要選用不同的軟硬件方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，編碼分辨率，幀率的選擇也是影響視頻效果的重要方面。比如到底是選擇高幀率低分辨率還是高分辨率低幀率？在固定碼率下，如何根據場景運動劇烈程度來動態調整幀率、分辨率，比如PPT場景，屏幕靜止的時候，我們可以降幀率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對不同的機器性能配置，在編碼的時候也需要選用不同的preset檔位。對於性能好的機器就會以複雜度優先考慮，以提升編碼質量；對於性能比較差的機器，就會以速度優先來犧牲部分的編碼質量，保證基本的視頻體驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SVC編碼的策略調整也是非常重要的。SVC雖然多人視頻場景下，網絡適應性好，但是SVC編碼相對普通參考幀模式，壓縮效率是有損失的。所以什麼時候採用SVC，什麼時候採用普通參考幀模式也是很有講究的。我們需要根據用戶網絡下行帶寬動態的開關SVC。而如果用SVC的話，也要根據用戶的網絡，動態調整SVC下發層數。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule"},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"頭圖"},{"type":"text","text":"：Unsplash"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者"},{"type":"text","text":"：王詩濤 - 騰訊多媒體實驗室視頻技術專家    "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文"},{"type":"text","text":"："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/VlwVuYCSQD5K1JJbKq32oA","title":"","type":null},"content":[{"type":"text","text":"騰訊技術開放日 | 保障視頻連線畫質清晰且流暢，騰訊會議有這些優化實踐"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"來源"},{"type":"text","text":"：騰訊多媒體實驗室 - 微信公衆號 [ID：TencentAVLab]"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"轉載"},{"type":"text","text":"：著作權歸作者所有。商業轉載請聯繫作者獲得授權，非商業轉載請註明出處。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}