实时音视频通讯过程中声音的那些事儿

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"前言","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由于新冠疫情的影响，视频会议和线上教育迎来了飞速的发展。而让这一切成为现实的基础就是实时音视频通讯技术，但在实时音视频通讯过程中，会面临各种各样的问题，有可能是网络问题，也有可能是产品问题，在一定程度上左右了用户体验（QoE）。尽管服务质量（QoS）是一个产品或者服务非常重要的参考标准，但是对于用户而言，他们更关心是QoS指标。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"正文","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"众所周知，一个产品或者服务的价值，很大程度上体现在用户的口碑上。如果用户都说这个产品或者服务好，那么这个产品或者服务一定能够赢得市场。这就不得不提一个和用户口碑相关的指标——用户体验（QoE）。在实时音视频通讯领域，用户的音频体验占有非常重要的地位。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"说到QoE，有很多评价的方法，通用的评价方法可以分为有参考客观评价方法、无参考客观评价方法和主观评价方法三种。其中，有参考客观评价方法有P.861、P.862、P.863等，无参考客观评价方法有P.563、ANIQUE+、P.1201、xxNet等。它们都为音频QoE指标的量化对比提供了理论依据。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天，我们主要围绕音频QoE指标在实际项目中遇到的问题进行展开。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"噪声问题","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"噪声问题应该是所有实时音视频产品不得不面临的问题，降噪处理（NS）可以说是产品必备的基础功能之一。但是，产生噪音的原因有很多，比如设备噪声、环境噪声、声音信号溢出、算法问题等。其中，对于设备噪声，常见的形式有风扇声音、键盘声音、异常电流声音等。对于环境噪声，常见的形式有鸣笛声音、周围人的说话声音、走路的声音、电视的声音、闹铃的声音等。对于声音信号溢出，大多和音频源有关系。对于算法问题，有可能是算法设计本身的问题，比如回声残留，还有就是算法适用范围的问题。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b0/b08880451cad4016a1d2953a0cffe351.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下来，通过一个典型的案例来分析一下实际项目中的噪声问题。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这个噪声问题是在科大讯飞语音识别服务对接过程中遇到的，由于项目需要，我司的移动端（安卓和苹果）SDK需要集成科大讯飞的语音识别功能，并做成一个可选功能对外提供。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对接科大讯飞语音识别服务的关键一步就是将移动端设备采集的音频PCM数据，每四十毫秒回调一次云端接口。由于安卓和苹果底层是用一套C++代码实现的，对外接口单独封装了Java层和OC层，所以在音频PCM数据的组织上，我在C++层实现了数据采集、存储和处理操作。最开始的时候，我将音频数据保存为16位短整型，安卓端SDK通过JNI层的数据转换，转换为8比特的音频原始数据，再由Java层回调科大讯飞的语音识别接口，是没有问题的，语音内容能够以文字的形式返回，并且正确率能够保证在95%以上；但是到了苹果端就出问题了，苹果端SDK在OC层将数据转化为8比特的音频原始数据，再由OC层回调科大讯飞的语音识别接口，返回的文字内容总是词不达意，正确率都不到50%。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"于是，我们展开了问题排查的排查工作，首先通过将C++层回调的音频PCM原始数据保存下来进行播放，声音是没有问题的，说明采集模块正常。然后，我们又将OC层转换前的16位短整形（注意：OC语言是没有短整形的概念的，这样讲是为了方便大家理解）数据保存下来，播放也是没有问题的，说明C++层到OC层的数据转换逻辑正常。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最后，我猜测只有一种可能，问题出在了16位短整形转换成8位的字节数据上。为了验证我的想法，我将转换后的8位音频数据保存下来，播放时果然发现了问题，存在","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"严重的噪音","attrs":{}},{"type":"text","text":"！通过观察声音的波形图发现，这段音频中存在有规律性的等间隔噪音波形。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"好了，问题定位了，那就解决吧！分析问题的原因可能是iOS平台在处理16位短整形数据时存在某种自动截取机制，会导致数据丢失。为了避免音频数据在OC层和JNI层的转换问题，我在C++层处理数据时，直接将音频PCM原始数据处理成8位字节类型，再进行向上回调。通过验证，安卓端和iOS端的语音识别表现都正常了。","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"至此，噪音问题解决","attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"声音偏小","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"声音偏小问题的原因也有很多，大致可以分为四类，设备采集能力弱、设备播放能力弱、模拟增益小、数字增益小。其中，设备采集能力弱是比较常见的原因，当然和用户说话声音小也有一定的关系。设备播放能力弱是从声音的接收端进行分析得到的结果，有可能用户的播放设备，比如耳机、音响存在一定硬件问题，导致声音输出音量小。模拟增益和数字增益是从算法的角度出发，对声音的增益程度有差异。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fa/fa4fbef61910cba5a55527023f7592e0.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下来，通过一个典型的案例来分析一下实际项目中音量偏小的问题。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我司对外提供的实时音视频SDK，第三方客户对接后，反映锤子手机在进入直播间后，声音特别小，别的安卓手机都正常。问题抛出后，让我方去排查。最终，这个重担又落到了我身上。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"拿到有问题的锤子手机，我开始了问题排查工作。声音偏小的问题很容易复现，只要进入直播间，基本上100%必现。因此，我断定这可能不是一个偶然现象，和自己最初的判断不符。后来通过深入分析发现，这款锤子手机的语音通话模式的声音本身就非常小，而WebRTC在直播推流和拉流过程中默认使用语音通话模式，因此，导致了直播间内播放声音非常小的问题。【老罗确实做手机的年头有些短，因为后来陆陆续续发现，几乎所有型号的锤子手机都存在这个问题，真替老罗着急】","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那么，这个声音偏小的问题有没有解决方法呢？方法肯定是有的，但是个折中的方案。因为我后来发现，锤子手机的媒体模式声音非常大，于是，我在SDK底层增加了黑名单，只要是黑名单中的手机型号都默认使用媒体模式，而不是通话模式。","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"至此，声音偏小问题解决","attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"回声问题","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"回声问题也是实时音视频通讯中比较常见的问题，形成的原因也有很多，基本上也能分为四大类，延时抖动、大混响环境、采集信号溢出、双讲。其中，延时抖动可能是由于线程繁忙导致的，也有可能是双设备导致的。大混响环境多半是混响长度超出了滤波器的长度。采集信号溢出很有可能是滤波器不收敛造成的。双讲，比较依赖自然语言处理技术，在内部处理过程中容易顾此失彼。其实，WebRTC在处理双讲时，本身就有一定的问题，所以对双讲支持的不好。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/97/97d52de905221e42c5077adf47f09593.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下来，通过一个典型的案例来分析一下实际项目中的回声问题。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在视频会议产品中，我司采购了一批安卓盒子，用做视频会议设备终端。安装了我司的移动端版本的客户端后，遇到了一个问题，发现讲话时声音总是忽大忽小，甚至消失。后来排查发现，原来是安卓盒子本身就支持硬件的回声消除，移动端安卓APP的软件回声消除和安卓盒子的硬件回声消除作用叠加了，导致了主讲人的声音被循环消除。后来关闭了硬件设备的回声消除，主讲人的声音就正常了。为了对比验证，我们关闭软件的回声消除，同时打开安卓盒子的硬件回声消除，主讲人的声音也是正常的。","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"至此，回声消除问题解决","attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"结尾","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"直播过程中音频的用户体验，是直播服务最后的一道保障。用户允许视频画面在一定程度上的卡顿，但是对于声音的卡顿是零容忍的。守好最后一道防线非常重要，我们要重视音频的QoE。音频好了，才能进一步追求视频的最佳表现。好了，今天关于音频QoE指标在实际项目中的介绍就结束了，欢迎大家赞点评论。关注我，分享更多音视频直播内容。","attrs":{}}]}]}