提升RTC音频体验 - 从搞懂硬件开始

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"前言","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RTC（实时音视频通信）技术的快速发展，助力了直播、短视频等互动娱乐形式的普及；在全球疫情持续蔓延的态势下，云会议需求呈现爆发式增长，进一步推动了RTC行业的快速发展。为了给客户提供稳定可靠的服务，网络系统方面需要不断提升频道连通率，降低会议过程中的断流率，增强抗弱网能力；视频方面需要提升视频清晰度，降低视频卡顿率等，音频方面在追求端到端MOS的同时，也要重点关注音频3A算法的效果，这些都是各厂家必须修炼的“内功”，也是最终沉淀下来的核心竞争力。本文将重点阐述硬件设备采集的音频质量对RTC端到端音频体验的重要性。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4b/4b64020cf464565a5b22fee2fc0bf58c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"图片来源于网络","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"采集质量不佳，会有什么影响？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在RTC架构中，端到端的音频信号处理流程大致如下图，上行分别经过了音频信号的采集，音频3A（AEC:回声消除、ANS:自适应降噪和AGC:自动增益控制）和编码；下行分别经过丢包恢复，解码，混音和播放。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ef/ef3f189e4bfe72d79099aad4294cf12a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"端到端的音频信号处理流程","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不难看出，音频信号经过模数转换，再经过设备集成的音频信号处理芯片，最后才传递给RTC SDK。由于硬件厂商的不同，音频采集解决方案参差不齐，因此采集到的音频质量的好坏直接影响着3A算法拿到的生产资料的可用性，同时也决定这最终用户接收到音频信号质量的上限。根据实际工作中遇到的音频问题，因为设备采集引起的问题基本可以归纳为如下几类：","attrs":{}}]},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

采集问题	现象与影响
无回调，音频异常	无声；直接影响可用性
音频异常	不可听；直接影响可用性
抖动	采集到播放延时抖动，引发回声；严重的会导致语义缺失，影响交流；
音量过小	声音过低，sdk如果数字增益不足，都是对端听感上比较吃力；
音量过大，爆音	音量巨大刺耳，影响听感体验；回采非线性失真较重，影响回声消除效果；
频谱缺失	无法满足高音质需求

"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/25/2567044f44396076ce37d52800d0fa6a.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"举几个例子：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"（1）采集异常","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"采集异常主要体现在频谱“模糊”，严重的会导致无法听懂语义，影响正常交流。如下语谱图。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/6a/6a368b967727f97d682a55f722e44847.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，采集异常后，播放的信号被麦克风采集后也会表现出异常，从而引起严重的非线性失真，影响回声消除效果，如下图。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e3/e3af83425334b8bb9daaf91c7fc949d8.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"(2) 采集抖动","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"常见的就是采集丢数据，听感上会听到有很多高频噪点（下图为上图中噪点放大后的局部图），严重的会影响AEC算法中对延时估计准确性和远近端非因果问题，严重的会导致漏回声。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/78/78cec19e53246f405bc526160f9c191b.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9c/9c78ec0f5da24f9e01a9d8f2c5635260.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"（3）爆音和音量小问题","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"采集爆音问题主要发生在PC，也是PC端设备最应该避免的问题，影响较大，除了截顶导致的频谱失真之外，严重的非线性失真会影响回声消除效果。爆音问题需要AGC算法通过自适应调节PC端模拟增益以及麦克风加强解决。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0a/0a0d1c3410e7c30204eb73c40492cb15.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"（4）频谱缺失","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"频谱缺失主要是硬件回调的音频采样率与实际的频谱分布不一致，即使编码器给到很高的编码码率，听感上也没有高音质的效果，如下图，采集信号采样率为48kHz，但是频谱上限却只有8k。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/94/94a2bb9fdf6d7e53c2f5b5d78e5f25eb.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"改善采集音质，硬件层面我们能做什么？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具备RTC能力的硬件设备早已渗透我们生活的方方面面，常见的如移动端手机和PC，现在甚至连儿童电话手表，天猫精灵以及各种高端的指纹密码锁等设备都支持了RTC。然而，设备的多样性直接决定这采集能力的差异性，抛开声学元器件设计差异这一因素，就Android端而言，芯片和软件系统的差异使得同一品牌的手机，也没办法用同一种配置适配所有型号的手机。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，现在绝大多数的移动端设备都自带硬件音频信号处理（后称硬件3A）能力，不同芯片效果方面也是千差万别的同时，更严重的是经过硬件处理的音频信号频谱往往会有缺失，如开启硬件3A后回调到RTC SDK的音频信号频谱上限仅支持到8k，相当于16kHz采样的音频信号，尤其在娱乐方面根本无法满足我们对高音质的追求。因此，做好硬件层的适配工作，是保障RTC高质量音频体验的基础。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Android端","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（1）需要搞清楚javaaudioclass和opensles这两种模式的差异，以及各自需要适配的参数，掌握关闭硬件3A的配置。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（2）采集抖动或音频音量异常，可以试试更改请求的采样率，通常设置的48k采样不会适用于所有的android设备。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Windows端","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（1）当前很多Windows设备会在屏幕顶端内置麦克风阵列，提供音频增强功能，开启方式如下图。这个功能默认屏幕正前方夹角区域为拾音区域，通过麦克风阵列技术可以有效的增强拾音区域内发言人语音，“隔离”拾音区域以外的“噪声”，其主要的弊端就在于开启此功能后仅支持8k频谱，且各厂家增强算法存在差异，效果也参差不齐。因此，软件需要具备能够bypass硬件自带音频增强功能的能力，为高音质做保障。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/de/de7f9eaf163f290e5175cd96dfedbcb1.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"Windows设备自带的双麦阵列（图片来源于网络）","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ae/ae4b064dbe0379d4cd6d4b6b16cc2041.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"音频设置中的增强功能开关","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d9/d9b45bfa78d4fb961eb66cdc3f381980.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"开启音频增强后，带来的频谱缺失","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（2）音量方面，PC端设备都支持模拟增益调节，大多数带有阵列的Windows设备都有额外的麦克风加强（如下图）。软件算法层面（3A中的AGC）需要具备自适应调节他们的能力，保障音频采集音量的平稳以控制采集底噪水平。初值设置或自适应调节不当都会导致音量小和爆音等问题，严重的会影响回声消除和降噪的效果，带来影响可用性的风险。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/55/55d527edf97309f27ed61270e1c18c1e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"模拟增益与麦克风加强","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"苹果设备","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（1）ios端适配工作较少，需要熟悉关闭硬件3A的配置，因为ios设备自带的硬件3A频谱也只能支持到10k-12k。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（2）Mac笔记本设备比较简单，仅提供了模拟增益调节。但是有一点需要注意，RTC在支持双声道播放时，由于麦克风会与某个扬声器在同一侧，导致播放音频时附近的麦克风采集爆音问题，一般只能优化软件AEC算法解决。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"总结","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"当48k高音质成了刚需，为了保障采集环节的高质量，一方面需要投入时间去掌握Android参数适配的规律，同时市面上出现的越来越多的定制化的android设备（手表，智能音箱等），也必不可少的需要先确定好配置参数；另一方面关闭硬件设备自带的音频处理功能，启用RTC自带的纯软3A算法也是一种趋势，前提是要优化好软件3A算法整体效果以及控制好功耗，这也是客户评测各厂家之间音频体验的必测项，也是各厂家的核心竞争力之一。","attrs":{}}]}]}