视频精修一帧要花2小时？AI只要5.3毫秒

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"进入全民短视频时代，人像视频的拍摄也正在迈向专业化。随着固化审美的瓦解，十级磨皮的网红滤镜被打破，多元化的高级质感成为新的风向标，“美”到每一帧是人们对动态视频提出的更高要求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，大部分手机均可记录主流的24fps、25fps、30fps、50fps和60fps（frame per second，FPS），以常见的30FPS为例，1分钟的视频就需要处理1800帧左右，如何保证处理过程中帧与帧之间的效果连续性是算法面临的关键突破点。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"事实上，传统磨皮算法是一般实时美颜算法设计的优先选项，其本质是由各类高通滤波算法和图像处理算法组合而成，通过滤波核的大小来实现人像的瑕疵祛除和肤质光滑，经过优化后也能够达到移动端的实时性能要求，但经传统磨皮算法处理后导致的五官与皮肤纹理细节缺失容易形成明显的“假脸”效果。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/a6\/40\/a6b9e28d78226a596e212808cce30940.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图1：传统磨皮算法VS美图美颜算法"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/cf\/b3\/cfc95c9435b5b1faa990b977103905b3.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图2：原图VS美图美颜算法"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"围绕用户更具个性化的“变美”需求，美图影像研究院（MT Lab）自研基于深度学习的实时视频美容方案。通过设计轻量的神经网络生成式模型，结合强大的模型优化推理框架（Manis）和千万级人像图库训练优势，实现对动态视频人脸的瑕疵修复与暗沉祛除，同时最大程度地保留了皮肤的真实纹理细节。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/87\/e9\/873baa475526e9054bd2bdcb8d7f32e9.gif","alt":null,"title":"","style":[{"key":"width","value":"25%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/77\/71\/775c7fddf72fcf4fbc47ce722eb22171.gif","alt":null,"title":"","style":[{"key":"width","value":"25%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"原视频VS美图实时美颜算法效果"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对比之下美图的实时美颜算法既没有弱化面部结构，对细微瑕疵也进行了精细化处理，脸部皮肤呈现干净通透、清晰自然的高级质感。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/2a\/14\/2a8969156c2c5b73ed644e9bbb713514.gif","alt":null,"title":"","style":[{"key":"width","value":"25%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/77\/71\/775c7fddf72fcf4fbc47ce722eb22171.gif","alt":null,"title":"","style":[{"key":"width","value":"25%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"传统磨皮算法VS美图实时美颜算法效果"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，为了兼顾更好的使用感受，轻量级的网络能在低、中、高端不同档位的移动端产品上实现更大范围地部署，满足移动端的实时性能要求，平均1秒钟能够美化处理视频142帧，为更多用户带来更好的“变美”体验。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"轻量级模型设计，提升生成效果"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"轻量级结构设计策略"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在进行网络结构设计时，首要考虑如何实现效果和速度的均衡。因此在保证不损失过多效果的前提下，模型结构尽量遵循了并行度高的设计原则，轻量级结构设计（如图3）的具体策略如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"不使用大于3x3的卷积核，下采样也使用stride=2的3x3卷积替代，因为3x3卷积的计算速度远高于其他大核卷积。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"模型中最大通道数不大于64，以减少大尺寸feature map的计算量。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"网络输入尺寸在不影响效果的前提下尽可能地缩小。同时，一定程度上减少输入宽度，而不是使用1:1的输入比例，因为人像两侧存在与美颜无关的背景区域，要避免增加额外的计算量。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"上采样使用最近邻插值加3x3卷积替代反卷积和双线性插值，以便于加速。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"非必要情况下尽量采用简单的单路架构，只在stride=2卷积后加入Concate分支，因为Add或者Concate操作虽然计算量很小，但是MAC很高；同时，网络不使用ResBlock，以节省内存占用。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/57\/c4\/57d37e01d3485610b5e451359d2147c4.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图3：美图轻量级实时美化模型结构"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"模型生成效果提升方案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了获得更好的实时生成效果，MT Lab借鉴了RepVGG的重参数等价转换思路，来进一步优化轻量级模型的组件重组流程（如图4）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"该流程在模型训练阶段，对每个3x3卷积增加并行的1x1卷积分支和恒等映射分支；而在模型实际推理阶段，则把对应的1x1卷积分支和恒等映射分支通过padding操作分别等价转换成特殊的3x3卷积，根据卷积的线形可加性，再将参数合并到主分支的3x3卷积里面。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这个方式相当于只增加模型训练阶段的网络消耗以提升网络生成效果，而在实际模型部署时增加的分支参数等价合并，并不会给网络增加任何额外的计算量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/9f\/aa\/9f77cf4aa5f34ca10bfd270b33790caa.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图4:模型组件重参数优化流程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同时，为了大幅提升网络训练效率，除了使用常规的重建感知Loss和像素级Loss外，MT Lab还借鉴对抗生成网络的思路，设计相应的判别Loss来监督网络，在微调（fine-tunning）阶段对实时美化网络进行修正，从而进一步优化模型的生成效果。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"判别Loss设计流程（如图5）先对训练数据标定出对应的斑痘、暗沉等瑕疵区域，作为瑕疵mask。再使用参数多、结构深的大型网络训练出一个精准的瑕疵mask分割模型，作为实时美化模型的判别网络。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在训练实时美化网络时，固定判别网络的参数，将实时美化网络输出的结果作为判别网络的输入，同时用一张全“0”mask作为监督，要求判别网络监督实时美化网络不能生成有瑕疵区域的结果，从而达到提升美化效果的目的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/2b\/d0\/2b36f2a075973d19a2154251f8b210d0.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图5:判别Loss设计流程"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"优化实时体验效果"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"众所周知，影响模型实时执行的因素包括图片帧率、分辨率和功耗。视频人像美化需要保持实时的高分辨率，模型的FeatureMap就会相应增大，再叠加美化模型内部的高计算量，导致整个推理过程帧率低且耗时长；同时，大量的图像前后处理增加了整体的效果耗时和设备功耗，实时处理难以长时间维持稳定。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MT Lab基于自研的全平台AI推理框架Manis，通过整合模型智能分发、纹理数据推理加速、效果叠加优化等多种技术方案，来完成美图美化模型在移动端App的顺利落地应用，为用户带来最优的实时效果体验。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"基于算力配置定制化模型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为保证不同档位的移动端产品均能获得最佳体验效果，MT Lab通过Manis的天枢平台系统为不同机型的设备能力下发定制的美化模型与AI配置，再通过AI推理框架（Manis）调度选择最优算力执行推理过程，从而既能保证低端算力设备达到实时效果，也能实现高端算力设备更优品质的画质表现。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"模型分发流程以不同设备最优性能的实现为原则，在模型设计之前就与包括华为、MTK、高通、苹果在内的AI芯片厂商达成深度交流与合作，从而保证训练后的模型结构和参数完全符合AI芯片的计算特性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"GPU推理方面，Manis针对高通的GPU架构在纹理内存上的访存能力较优的特点，选择GL texture纹理推理计算方式；针对MTK设备在普通内存上的多种加速特性能力，选择GL buffer纹理推理计算方式；而针对支持OpenCL规范的共享特性的高通GPU设备，则通过OpenCL和OpenGL上下文关联，将GL texture与CL texture、GL buffer与CL buffer进行映射，实现OpenGL\/OpenCL混合执行，再利用渲染和计算方式的优势，从而达到AI算法在GPU的最优调度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/0e\/96\/0ec30dd5a96b13e666860330057eb996.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图6:美图天枢解决方案模型分发流程"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"实时美化模型优化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"事实上，CPU和GPU数据交互同步是一件非常损耗性能的操作，功耗增加导致长时间的处理下容易出现掉帧现象。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对此，MT Lab在人脸检测环节采用极速轻量的CPU推理，快速获取人脸区域，通过局部的数据操作，降低FeatureMap大小的同时保留关键特征图信息，避免大数据量下GPU带宽受限带来的性能掉点问题；在图像处理环节通过GPU数据流并发推理，弱化了高计算量带来的负面影响。最后，基于双通道数据流在局部区域上进行效果叠加优化，从而保证了视频中每帧数据的高分辨率，呈现高品质的实时画质。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/f4\/d1\/f42d362a7c46f62a263aa9bb6f16c3d1.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图7:纹理数据流加速策略"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"美图优化加速器— —AI推理框架Manis"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基于Manis的模型优化是视频美化算法得以顺利落地的核心环节，与此同时Manis还在美图产品应用场景中扮演着更为重要的角色。它既实现了移动端上极致性能优化，还服务于加速AI项目的落地生态打造。通过与主流开源框架的性能数据对比（如图8），可以很明显地感受到Manis所具备的高水平推理能力与性能提升能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/6a\/42\/6a9a721a939f774902489a73974de642.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图8:Manis与某开源框架第三季度最新版本的性能数据对比"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在实际应用中，Manis包含AI服务、天枢系统、运维监控等在内的多项功能，主要通过以下三个体系模块来实现对算法的优化加速："}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"模型转换模块"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以快速实现各主流模型结构向Manis模型结构转换，以便算法顺利接入。同时，通过图优化技术简化模型结构，为各种执行设备如CPU、GPU、AI芯片添加优化控制手段，达到模型层面的性能优化。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"模型测试模块"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基于 Manis在主流手机设备上的部署，能够在线测试输出模型在各种算力使用场景下的性能表现和评估信息，对模型算法进行快速验证，从而帮助模型不断迭代优化，同时缩短优化的开发周期。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"模型推理模块"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Manis高度适配CPU、GPU、DSP、NPU、ANE、APU等多种硬件设备，其中GPU支持OpenGL、OpenCL、Metal、CUDA等多种技术方案，CPU支持fp32、fp16、bf16、int8等多种精度方案。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中，针对移动端设备的性能优化包括汇编级CPU neon优化、图优化、Auto-Tuning、多线程优化以及算子融合；针对移动端的精度优化包含fp32\/fp16浮点计算方式、bf16格式计算策略以及8位整型量化计算方案，能够结合推理的设备能力，进行动态图切分及混合精度计算，释放设备的最大算力。而针对类似实时美化这样的复杂应用，则采用定制化的优化策略，包括内存复用策略、内存池、模型共享以及数据排布优化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/f7\/ae\/f7c1c8e8b253a047eff29a758d4e75ae.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图9:美图AI推理框架（Manis）架构图"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，美图全部产品的应用落地场景都有着Manis的身影，它为美图核心AI算法在不同平台和硬件上实现了低延迟、低内存、低功耗的应用落地，未来Manis也会进一步迭代和优化，拓展实时化应用上的更优性能。"}]}]}