国际酒店聚合算法优化

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1. 背景介绍"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"『聚合解决的是“让数据有的比”的问题，聚合的成功率和准确率直接奠定了用户在网站进行比价体验的基调。』"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在酒店频道，聚合一直被认为是业务的基础和核心。因为无论是Qunar最开始定位的报价搜索，还是现在转型的比价平台，业务模式决定了我们要从众多代理商和渠道获取大量的酒店数据并对其进行整合， "},{"type":"text","marks":[{"type":"strong"}],"text":"所以聚合解决的是一个“让数据有的比”的问题"},{"type":"text","text":" ，不夸张的说，聚合的成功率和准确率直接奠定了用户在网站进行比价体验的基调。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"酒店聚合的职责简单归纳就是："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"将不同代理商来源的酒店数据（酒店tree）统一到Qunar酒店下，为比价平台提供酒店对应关系的映射。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"例如以下这组聚合关系："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/56\/c1\/56b1889d8286a784d73331854b06d3c1.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前酒店聚合算法主要参考 "},{"type":"text","marks":[{"type":"strong"}],"text":"酒店名、地址、城市、"},{"type":"text","text":" 座标、电话这几类数据进行判定，名称和地址作为重点参考内容，在大多数场景下可直接决定聚合结果；座标和电话由于来源数据规范等问题（如座标系不一致，酒店电话和代理商电话掺杂提供），仅做辅助判定。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2. 痛点难点"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"『各国地址信息命名本地化、差异大；基于对文本相似度的计算，对于酒店名和地址中的信息解析能力有限。』"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"国际酒店在Qunar起步较晚，各类基础数据，特别是聚合层面的数据积累有限，且 "},{"type":"text","marks":[{"type":"strong"}],"text":"国际代理商数据参差不齐，数据的规范化程度较差"},{"type":"text","text":" ，再加上运营资源有限，依靠 "},{"type":"text","marks":[{"type":"strong"}],"text":"人工为数据建立聚合关系成本高、效率低、不现实"},{"type":"text","text":" 。基于以上这些情况，我们急需提高国际酒店自动聚合算法的能力。本文将介绍过去一年中针对国际几个重点国家进行的聚合算法优化的情况，希望能给到有类似业务痛点的同学一些解题思路和参考。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前国际酒店聚合的痛点与难点集中在以下两个方面："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"A. 不同国家间地址信息的本地化差异"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/4e\/ca\/4e76218331b691b34ba7a540e031feca.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上图所示，地址的组成格式在各个国家之间都有差异且其中夹杂着本地化的信息（图中标红的部分）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"B. 原有算法针对文本相似度计算，对于酒店名和地址中的信息解析能力有限"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"酒店聚合算法的本质是针对两个酒店信息进行文本相似度的计算，满足一定的相似度分值后即可判定为二者存在聚合关系。但对于类似名称，地址类的长文本数据，其本身就包含多种成分，如品牌，分店，酒店行业词，路名，门牌号，城市等，且这些成分的重要程度是有所区别的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"比如“永正”作为品牌信息几乎可以直接锁定“北京永正商务酒店”，然而“商务酒店”作为行业词，由于信息太过模糊，几乎不能定位到任何具体的酒店。如果不能把这些成分区分清楚而直接进行文本相似度计算，就可能出现把下列两家酒店聚合到一起的错误。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/f0\/c0\/f07b4cd7fbb90578e3f44a20b31214c0.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下图所示，目前的国际酒店聚合算法对于酒店名和地址中的详细成分几乎无法拆分。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/69\/51\/69ee2a7de731e89bbf4ab41fd22dee51.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3. 优化思路"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"『整理高频次出现的名称及地址格式，形成各种分词结果；从文本匹配变为将分词结果与可能模式进行匹配，针对相似度进行加权评分计算。』"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上文提到，酒店名和地址可细分出多种不同成分，其中重要的部分可归纳为："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/43\/27\/4309e7140f27064a6d89966db2b8c927.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中 "},{"type":"text","marks":[{"type":"strong"}],"text":"城市层级，品牌，poi字典"},{"type":"text","text":" 以qunar酒店基础信息为基底，配合代理商数据，酒店官网扩充覆盖及同义词；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过对酒店名分词并进行反向高频统计，配合运营人工筛选，可整理出酒店 "},{"type":"text","marks":[{"type":"strong"}],"text":"描述词"},{"type":"text","text":" 和 "},{"type":"text","marks":[{"type":"strong"}],"text":"行业词"},{"type":"text","text":" 字典；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"酒店地址中的路名信息由于存在大量本地化内容，这部分关键字词典需要分地区（一般是按国家维度）对地址分词并统计高频结果，配合官方数据（google，Wikipedia等）筛选整理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基于目前库内已有的酒店数据，我们对名称和地址的组成格式进行统计，整理出高频率出现的格式，具体如下图截取的片段所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/a8\/fa\/a8d1e1195a3d974782774d315103b7fa.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过引入这种模式匹配的概念，就可以很容易的从给定的酒店名和地址中抽取出各类成分信息，举例如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"酒店名拆分示例"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/38\/47\/38e948762a5cde26568ceb9fe9e89b47.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"酒店地址拆分示例"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/80\/d8\/808a8f68a3fc864b9ab7f251221271d8.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"【划重点】"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"优化后的一次完整聚合流程如下"},{"type":"text","text":"："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"一条待聚合的酒店数据进入；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"基于预先收集的各类型词汇（品牌，城市，国家）的词典，对酒店名和地址数据进行解析，找出其中所有可能的分词结果；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"将分词结果与所有可能的模式进行匹配，选出合法匹配中的最优解作为解析结果；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"根据解析出的关键成分（品牌+分店名，路名+门牌号）在已聚合数据中进行全文检索，初筛出相似度最高的n条结果作为候选集；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"在待聚合和各候选酒店之间，针对各个成分进行对比，对比出的基于字符串的相似度（匹配的可能性有：完全一致，包含，前缀，后缀等）结果配合各个成分本身的权重（如酒店名中：品牌>分店>行业词>城市），最终算出一个综合的分数；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":6,"align":null,"origin":null},"content":[{"type":"text","text":"在某些特殊情况下进行打分的调整：如路名相同，门牌号不同，减分；如联系电话相同或座标距离相近，加分；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":7,"align":null,"origin":null},"content":[{"type":"text","text":"选择相似度得分最高的候选酒店作为最终候选者，判断得分是否高于给定的聚合分数线，是则可认为聚合成功。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4. 优化成果"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"针对国际重点的八个国家，我们按照模式匹配的方式优化了聚合算法，收效成果如下图所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/e9\/bc\/e99e7891dc159da89719e2c41d463abc.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"后续还将继续分享国际酒店房型的聚合经验，欢迎各位多多交流，共同进步。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者介绍"},{"type":"text","text":"："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"王刚，2013年加入Qunar，目前主要负责国际酒店供应链系统，专注于基础数据集成，竞对分析，搜索 & 聚合算法等领域。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"本文转载自公众号Qunar技术沙龙（ID：QunarTL）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文链接"},{"type":"text","text":"："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s?__biz=MzA3NDcyMTQyNQ==&mid=2649263854&idx=1&sn=ce65fb27b7d3cc0e8422ec64f8c6f41d&chksm=87675f10b010d606822f4586ae3dcdd07ab7e1d872861e72fa608991e31982bf4d2368406a79&token=1438974945&lang=zh_CN#rd","title":"","type":null},"content":[{"type":"text","text":"国际酒店聚合算法优化"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}