全球最大多领域中文语音识别数据集 WenetSpeech 正式发布并开放下载

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"日前，西北工业大学音频语音和语言处理研究组(ASLP Lab)、出门问问、希尔贝壳联合发布1万小时多领域中文语音识别数据集 WenetSpeech，在腾讯会议天籁实验室、华为升思 MindSpore、 西安未来人工智能计算中心等机构大力支持下，该数据集目前已经开放下载。数据申请入口: "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/wenet-e2e.github.io\/WenetSpeech\/ "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前该工作已经投稿语音研究顶级会议ICASSP2022，详见："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/arxiv.org\/pdf\/2110.03370.pdf"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/9a\/9afa84ea8e19f0012f88163ba3b82e8c.webp","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"WenetSpeech 介绍"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"近十年以来，在深度学习的推动下，语音识别技术和应用均取得了突飞猛进的发展，搭载语音识别技术的相关产品和服务，诸如语音搜索、语音输入法、智能音箱、智能电视、智能穿戴、智能客服、机器人等已经广泛应用到我们生活的方方面面。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但在现有的中文语音识别研究中，由于开源中文语音数据集数据量少，场景单一，缺乏挑战性，不能反映研究模型在大数据量和复杂场景下的泛化能力，例如，当前最大的中文普通话开源数据集 AIShell-2，包含1000小时的朗读风格录制数据，主流识别系统在该数据的测试集上获得的错误率低至5.3%左右。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"工业界往往使用更大规模的内部数据进行研究，而学术界无法获取这些数据进行研究，这导致了中文语音识别研究在学术界和工业界的严重割裂。另一方面，当下研究的热点无监督学习和自学习，在中文语音识别领域，也缺乏公开标准的大数据集的支持。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今年以来，Facebook 发布面向监督学习的5万小时的英文 audiobook 数据集 Multilingual LibriSpeech；SpeechColab 发布1万小时的多领域英文数据集 GigaSpeech。受这些工作的启发，同时中文语音识别研究也迫切需要一个标准的大规模多领域的数据集，为此我们设计开发了 WenetSpeech 数据集。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"WenetSpeech 除了含有 10000+ 小时的高质量标注数据之外，还包括2400+ 小时弱标注数据和 22400+ 小时的总音频，覆盖各种互联网音视频、噪声背景条件、讲话方式，来源领域包括有声书、解说、纪录片、电视剧、访谈、新闻、朗读、演讲、综艺和其他等10大场景，领域详细统计数据如下图所示。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/05\/05a4ed6d2c46a35e1785f4e9e4c44083.webp","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"数据收集"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"WenetSpeech 所有的数据均来源于网络，其中三分之二的数据来自 Youtube，三分之一来自 Podcast。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对于 Youtube 数据，我们人工选择含有嵌入式硬字幕（字幕嵌入在视频流中，非外挂字幕）的视频资源，并构建了如下图的基于 OCR 的系统进行数据挖掘，流程如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"文本检测，在当前视频帧上进行文本检测。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"字幕位置校验，判断检测到的文本区域是否为合法的字幕区域。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"字幕切换检测，已得到字幕位置和区域，在连续的视频帧上对该区域进行检测，直至该区域的字幕变化为止，得到字幕的起始和结束时间。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"文本识别，将字幕区域进行 OCR 识别，得到文本。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"将3中对应时间的音频提取出来，结合4中的文本，即得到字幕文本和该文本对应的音频，也就是语音识别训练中所需的文本和语音的候选平行数据。"}]},{"type":"paragraph","attrs":{"indent":0,"number":6,"align":null,"origin":null}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7a\/7a6cce3ab0ce15e83d48fc53603aab2e.webp","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下图中给出该 OCR 系统在不同场景下的几个典型示例。图中绿色的框为检测到的所有文字区域，红色的框为判定为字幕的文字区域，红色框上方的文本为 OCR 的识别结果。 可以看到，该系统正确的判定了字幕区域，并准确的识别了字幕文本，同时经过我们测试，发现该系统也可以准确判定字幕的起始和结束时间。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/6a\/6a5ec439169b2995d905e38f5a073c07.webp","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对于 Podcast 数据，我们使用国内最好的商业语音识别系统之一，对 Podcast 数据进行切分，并生成切分后音频和其所对应的文本作为候选平行数据。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"数据校验"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"OCR 字幕识别和 ASR 语音转写生成的候选平行数据中不可避免的存在一些错误，如人工字幕本身有错误，字幕时间不准，OCR 识别错误，转写错误等。为了检测该错误，WenetSpeech 中提出一种基于端到端的自动标注错误检测算法，如下图所示。该算法首先根据候选平行数据的文本(ref)构建一个一个强制对齐图，该图中允许在任意位置进行删除、插入和替换操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然后将候选平行数据的语音输入到该图进行解码得到识别结果(hyp)，最终计算 ref 和 hyp 的编辑距离并做归一化从而得到该候选平行数据的置信度。当候选语音和文本一致性高时，ref 和 hyp 一致性高，置信度高，反之，当候选语音和文本一致性低时，置信度低。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3b\/3bb96ebae567e2990e9e9d5bf04b4fd6.webp","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"WenetSpeech 中选取置信度>=95%的数据作为高质量标注数据，选取置信度在0.6和0.95之间的数据作为弱监督数据。关于该算法的详细内容，请参考我们的论文。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"排行榜"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了训练中校验用途的 Dev 集外，我们还设计了两个人工精标测试集，互联网测试集 Test_Net 和会议测试集 Test_Meeting，作为“匹配”和“不匹配”测试，同时提供三个语音识别主流工具包（Kaldi，ESPNet，WeNet）上搭建的基线系统，方便大家复现。在 10000+ 小时的高质量标注数据上，目前三个系统的语音识别率如下表所示（结果为 MER%，中文算字错误，英文算词错误）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/6c\/6c8d6a33fb6569976e24fd9f1548677e.webp","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"WenetSpeech 2.0"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"虽然 WenetSpeech 将开源中文语音识别训练数据规模提升到一个新的高度，然而我们希望进一步进行扩展和完善："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"从领域角度，现有数据集在口音、中英文混合、会议、远场、教育、电话、语音助手等场景仍覆盖不足。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.从数据量角度，现有的2万+小时的总数据，对于无监督学习仍然远远不够。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，WenetSpeech 在设计之初，就考虑到了未来做进一步扩展。目前我们已经开始 WenetSpeech 2.0 的工作，并且在 2.0 中，我们希望更多的行业机构和开发者能参与进来，能够集行业之力更好、更快的去做一个更大更泛化的数据集，从而进一步反哺和造福整个行业。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者介绍"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"张彬彬，出门问问 WeNet 项目负责人，2018 年加入出门问问负责端到端语音识别系统的研发和落地，包括 WeNet 的开源推进，车载和 toB 项目等。2017 年硕士毕业于西北工业大学音频语音与语言处理研究组，曾在微软、百度、地平线等公司工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}