AI 在視頻領域運用—彈幕穿人

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如今,B 站已經成爲了國內最大的視頻彈幕網站,其他視頻平臺、漫畫、閱 讀等內容平臺也都增加了彈幕功能。彈幕已經成爲一種重要的內容互動的手段, 因此研發一套接入靈活、玩法豐富的彈幕組件就顯得非常重要 。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"全文3979字,預計閱讀時間9分鐘。","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"引言:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如今B 站已經成爲了國內最大的視頻彈幕網站,其他視頻平臺、漫畫、閱讀等內容平臺也都增加了彈幕功能。彈幕已經成爲一種重要的內容互動的手段。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"廠內在短視頻,長視頻,直播等媒體類產品線較多,面臨着同樣的彈幕問題,諸如彈幕量大影響視頻體驗,彈幕互動(點贊),角色彈幕,彩色彈幕等諸多業務需求。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前市面上開源的彈幕實現主要是基於DanmakuFlame, 該庫代碼已經停止維護,並且支持的功能比較單一,接入維護困難,擴展困難。基於以上原因因此研發一套接入靈活、玩法豐富的彈幕組件就顯得非常重要。BDDMBarrage即是基於滿足以上需求開發的彈幕sdk,該sdk支持自定義彈幕樣式,支持彈幕數據源注入,彈幕穿人等功能。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該文會主旨講述彈幕穿人的解決方案,爲方便理解,我們會從宏觀的彈幕架構出發,深入到人像分離技術,算法,服務端部署,彈幕遮罩管理,遮罩緩存以及開發中遇到的棘手問題包括解決該問題的技術策略,還有端側的性能優化的諸多方面展開討論,最後會分享一些對未來技術暢享和規劃。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"先體驗一下最終的實現效果圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f8/f845533f9302025db2b0b5330ffa7be5.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、彈幕架構圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e7/e715c83704b3666327153f8dc7a9cde3.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"彈幕數據管理模塊:","attrs":{}},{"type":"text","text":"其主要功能是爲彈幕渲染模塊提供數據。首先,會在彈幕緩存模塊內查找是否有該時間點的彈幕數據,如果有數據,則直接提供給彈幕渲染模塊;如果沒有數據,則觸發彈幕數據的網絡請求,獲得彈幕數據後再提供給彈幕渲染模塊。因爲彈幕對時效性要求比較高,所以該模塊設計了一個預取策略,這樣保證彈幕渲染模塊在獲取數據的時候,儘可能地命中緩存模塊,減少由於網絡請求而產生的時間延遲。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"彈幕渲染模塊:","attrs":{}},{"type":"text","text":"其中彈幕時間引擎可以方便地統一調控彈幕整體速度,比如倍速、慢速、暫停等;彈幕調度模塊提供一套嚴密的軌道算法:可以根據彈幕自身的長度設定合理的速度,並且能夠保證同一軌道的任何兩條彈幕不會碰撞。針對彈幕的樣式以及彈幕自身交互,提供一套自定義方案。接入方可以按照自己的APP形態去任意設計彈幕的UI,也能夠發掘出新的玩法從而提升整個APP的互動氛圍,比如番樂APP設計了依據劇情的角色彈幕,運營彈幕,也可以增加VIP彈幕等。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"彈幕穿人模塊:","attrs":{}},{"type":"text","text":"該模塊是利用AI的圖像分割技術產生一系列的蒙版文件,然後根據視頻播放器對視頻的裁剪方式,對蒙版進行處理生成合適的遮罩,最後按照視頻的時間軸把相應的遮罩渲染到彈幕View上。具體方案會在後面介紹。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、彈幕穿人","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在幾個產品線接入彈幕組件以後,發現大家都不約而同把彈幕軌道數設置爲3,具體原因是擔心彈幕過多而影響用戶的視頻消費體驗。這樣雖然保證了視頻觀看的體驗,但是大大弱化了彈幕營造的氛圍,爲此我們彈幕組件的rd們開始暗戳戳地準備一個黑科技——彈幕穿人技術,希望以此能解決問題。下面一段視頻給大家看下我們當前的進展。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"彈幕穿人架構圖:","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/65/65b42eac8737c2e99b6bd172fdfd73ac.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"整個流程自下而上,分成算法側、服務端、客戶端三層:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先,算法側按每秒 32 幀的頻率進行視頻抽幀,對每一幀進行人臉識別,配合人臉跟蹤和平滑處理,生成每一幀的人臉元數據;其次,服務端將多個幀的人臉元數據進行相似度濾重,然後根據每3分鐘一個元數據包。在客戶端sdk側會根據播放進度預拉取服務側的對應時間段的壓縮包,播放到相應幀將彈幕試圖與人臉元數據做一個混合渲染。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"下面着重介紹下每個模塊或子模塊完成的任務:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"1、算法側","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1)視頻抽幀模塊:將視頻流按每秒 32 幀(可配置)的頻率抽幀。抽幀頻率越高,遮罩越平滑,遮罩顯示畫質會更細膩,但後面人臉識別算法耗時也隨之增加,手機的性能損耗也會隨之增大,如內存耗電等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2)模型訓練模塊:提供多張多角度劇中出現的人物圖像,給模型訓練模塊來訓練,生成對應人臉庫,再配合已訓練完成的明星庫,這兩個庫可以大大提高人臉檢測的準確度;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3)人臉檢測:識別每一幀圖像中的人臉,並給出人臉輪廓數據;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4)人臉相似度:爲減少網絡數據傳輸壓力,會對相似度大於95%的兩幀,丟棄一幀,或者數幀。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"2、服務端","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1)視頻抽幀元數據管理:管理算法側提供的幀數據,以視頻維度,視頻內時間段維度將大量的視頻元數據進行分包,建立映射索引,提供到SDK的可以是某個時間段內視頻的元數據組","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2)合併:算法側吐出的都是每一幀的元數據,但客戶端關心的是一張人臉的變化過程,服務端會把元數據合併,去重組成人臉組數據;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3)彈幕服務:提供基礎的彈幕數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"3、客戶端sdk","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"1)渲染模塊:渲染模塊 有兩套方案:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"▌其一是直接通過Canvas的混合模式繪製setXfermode模式繪製。該模式會對canvas上的兩個圖層進行選擇性疊加,這樣在頭像部分的圖層上,我們選擇只繪製遮罩層,而不繪製彈幕層即可實現遮罩效果;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"▌其二使用OpenGL,根據傳入的遮罩圖,在Fragment Shader 處,輸出對應的繪製顏色即可。最初使用的方案是OpenGL的繪製,通過源碼閱讀發現兩種實現方案在底層實現上是一致的,Canvas也是Surface提供出來的可繪製api,選擇第一種既可以,方便簡潔;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"2)人臉數據緩存:","attrs":{}},{"type":"text","text":"緩存整個視頻的索引表,根據索引表定位到具體的遮罩包, 根據當前的播放進度在遮罩包內便宜取處對應的遮罩;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"3)彈幕基礎控制API,以及配置API。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、服務部署","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"1: 環境:","attrs":{}},{"type":"text","text":"環境依賴:FFmpeg、Python2.7、OpenCV、numpy","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"人臉檢測服務2qps","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"人像分割服務10qps","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"2:  離線數據存儲結構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"離線處理過程中的文件存放目錄及文件後綴:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目錄名:{vid}_{media_id}根據視頻vid及media_id生成對應文件夾,包含如下子文件:","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"frame(抽幀文件 .jpg)","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"humanseg(人像分割處理後的base64圖片信息 .json)","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"contour_png(圖像處理過程中生成的輪廓圖 .png)","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"contour_svg(轉存爲svg格式的圖片 .svg)","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"zip(最終打包文件 .zip)","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":6,"align":null,"origin":null},"content":[{"type":"text","text":"mapping(索引文件 .json)","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":7,"align":null,"origin":null},"content":[{"type":"text","text":"log(腳本日誌)","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"3: 抽幀腳本:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"抽幀使用的百度內部人像腳本:","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/01/01ee127192f6532601956a075ae58f9b.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、SDK內置人臉模型碰到的問題","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"廠內也嘗試了使用端內置人臉模型的方案。碰到如下問題:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、視頻的播放每16s一幀,會產生大量的幀數據,模型識別速度在性能上碰到了瓶頸,會存在丟幀的情況,導致遮罩效果不夠細膩。尤其頭像邊緣處理較爲嚴重。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、端側識別時手機 cup 消耗增大,即耗電量會增大,同時可能也影響到播放器卡頓率,整個內存壓力也很大。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"五、棘手的問題","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"1:蒙版文件太大","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一個2分鐘視頻按照每秒32幀進行抽針、圖像分割的話,將會得到3840張蒙版文件,目前從圖像分割算子獲取的蒙版文件是一張二值圖(PNG格式),大約在100多K,那麼一個2分鐘視頻生成的蒙版文件總計375M。按照這樣的規格去設計,彈幕蒙版文件有可能比視頻本身還大,佔用更多的帶寬,這樣肯定是不能落地的。另外,由於彈幕蒙版文件過大,下載也需要花費比較長的時間,勢必會造成視頻已經播放了很長一段時間,但彈幕蒙版還處於下載中,用戶體驗也會非常糟糕。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對這個問題,我們主要從兩方面入手:第一、把二值圖轉爲svg文件,因爲svg文件就是一個純粹的XML,並且可壓縮性非常強。只需要把二值圖中人形輪廓記錄到svg文件裏面即可(也就是一些點的集合)。另外還可以靈活調整記錄人形輪廓的粒度,從而進一步調整svg文件的大小。最終我們把一張二值圖大小從100多K壓縮爲幾百字節。第二、蒙版文件集合採用分段壓縮存儲,這樣可以達到邊播變下載的效果。而且,在第一段下載完成以後就可以渲染,提升用戶體驗;另外,視頻播到哪裏,彈幕蒙版文件下載到哪裏,這樣也節省了帶寬。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Svg壓縮包格式:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"▎","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"zip文件命名規則:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"{vid}_{interval} _{index}.zip","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"示例:4752528107223374247_10_0.zip","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"▎","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"svg文件命名規則:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"{index}.svg 示例:0000001.svg","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"▎","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"索引文件結構:","attrs":{}},{"type":"text","text":"index.json","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"     ","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4c/4cc877f200b290f92f43bb415926701e.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"2: 手機端內存消耗過大","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不同手機內存是有限制的,尤其一些低端機對內存的消耗更是捉襟見肘,而視頻類app相對來說更耗內存,所以彈幕sdk的耗內存程度,直接決定了其可用程度。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於每個遮罩文件大約在100~200kb, 我們的遮罩是1分鐘會產生32幀,即使經過合併處理也還是很大。按照這個計算1分鐘的內存佔用:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Memory Total = 32 * 100kb = 3.125MB","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在ios側還算可以,在Android側每一分鐘產生3mb的內存佔用即使進行內存回收也會產生很差的性能體驗,平凡的內存回收會很耗性能,導致app卡頓明顯。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"解決思路:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根據視頻時長分配固定的本地內存,該本地內存循環利用,這樣減少了內存的頻繁回收,限制了內存的無止盡使用。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"六、未來展望","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"人臉數據的產生過程是很耗成本的,單服務跑一個腳本5分鐘視頻,大約需要2小時,但是產生的數據是複雜的,不僅可以產產生人臉二值圖,還可以產生人體的其他數據, 如人的運動軌跡等,下一步我們準備把帶有人臉數據和人體數據的腳本,人像運動軌跡做爲基本腳本,基於這些基本腳本,可以做很多創新的案例,比人彈幕互動,彈幕跟隨的玩法,也可以對視頻中不同人物的頭像進行變臉等等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"推薦閱讀:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://mp.weixin.qq.com/s?__biz=Mzg5MjU0NTI5OQ==&mid=2247502056&idx=1&sn=d198187b97dd01246256214c548ad953&chksm=c03ef894f7497182b8a7bd24e8ece6d44596b873805e3bb34857ca04d3d5305c9ae0d4e2d943&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"|","attrs":{}}]},{"type":"link","attrs":{"href":"http://mp.weixin.qq.com/s?__biz=Mzg5MjU0NTI5OQ==&mid=2247503500&idx=1&sn=f84fd6f9e62feeb7b91dfd6f1f0d145d&chksm=c03efef0f74977e66f26f287912dc37c1f3cb0124c57483205345ad7645ea59bc3518e1c82e2&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"iOS簽名校驗那些事兒","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://mp.weixin.qq.com/s?__biz=Mzg5MjU0NTI5OQ==&mid=2247498745&idx=1&sn=88ab93caf2e8a3662b307206954bbcdb&chksm=c03ecb85f749429346766d992c69ffbe58037d33ae55642693efe497a5da722b5ae484b38d11&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"|","attrs":{}}]},{"type":"link","attrs":{"href":"http://mp.weixin.qq.com/s?__biz=Mzg5MjU0NTI5OQ==&mid=2247502740&idx=1&sn=499a704b05ff86891250ac96a538cc7d&chksm=c03efbe8f74972fef5f901768fca78430b65bf0aee686b479b7cf1d0ede31b89077015e60726&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"信息流推薦系統智能交付解決方案探索","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://mp.weixin.qq.com/s?__biz=Mzg5MjU0NTI5OQ==&mid=2247502056&idx=1&sn=d198187b97dd01246256214c548ad953&chksm=c03ef894f7497182b8a7bd24e8ece6d44596b873805e3bb34857ca04d3d5305c9ae0d4e2d943&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"|","attrs":{}}]},{"type":"link","attrs":{"href":"http://mp.weixin.qq.com/s?__biz=Mzg5MjU0NTI5OQ==&mid=2247502687&idx=1&sn=8953092fe93d4f3299982bc48cf6211e&chksm=c03efb23f74972353d8f71802a256f0f79905552ffdad3fe694eb4914142c1fb8070e1f4586c&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"圖譜相關技術在風控反作弊中的應用和探索","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"---------- END ----------","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"百度 Geek 說","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"百度官方技術公衆號上線啦!","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"技術乾貨 · 行業資訊 · 線上沙龍 · 行業大會","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"招聘信息 · 內推信息 · 技術書籍 · 百度周邊","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"歡迎各位同學關注","attrs":{}}]}]}
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章