作业帮PB级低成本日志检索服务
{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"日志是服务观察的主要方式,我们依赖日志去感知服务的运行状态、历史状况;当发生错误时,我们又依赖日志去了解现场,定位问题。日志对研发工程师来说异常关键,同时随着微服务的流行,服务部署越来越分散化,所以我们需要一套日志服务来采集、传输、检索日志。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基于这个情况,诞生了以ELK为代表的开源的日志服务。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"需求场景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在我们的场景下,高峰日志写入压力大(每秒千万级日志条数);实时要求高:日志处理从采集到可以被检索的时间正常1s以内(高峰时期3s);成本压力巨大,要求保存半年的日志且可以回溯查询(百PB规模)。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"ElasticSearch的不足"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"ELK"},{"type":"text","text":"方案里最为核心的就是"},{"type":"text","marks":[{"type":"strong"}],"text":"ElasticSearch"},{"type":"text","text":", 它负责存储和索引日志, 对外提供查询能力。"},{"type":"text","marks":[{"type":"strong"}],"text":"Elasticsearch"},{"type":"text","text":" 是一个搜索引擎, 底层依赖了"},{"type":"text","marks":[{"type":"strong"}],"text":"Lucene的倒排索引技术"},{"type":"text","text":"来实现检索, 并且通过"},{"type":"text","marks":[{"type":"strong"}],"text":"shard"},{"type":"text","text":"的设计拆分数据分片, 从而突破单机在存储空间和处理性能上的限制。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"• 写入性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ElasticSearch写入数据需要对日志索引字段的倒排索引做更新,从而能够检索到最新的日志。为了提升写入性能,可以做聚合提交、延迟索引、减少refersh等等,但是始终要建立索引, 在日志流量巨大的情况下(每秒20GB数据、千万级日志条数), 瓶颈明显。离理想差距过大,我们期望写入近乎准实时。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"• 运行成本"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ElasticSearch需要定期维护索引、数据分片以及检索缓存, 这会占用大量的 CPU 和内存,日志数据是存储在机器磁盘上,在需要存储大量日志且保存很长时间时, 机器磁盘使用量巨大,同时索引后会带来数据膨胀,进一步带来成本提升。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"•对非格式化的日志支持不好"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ELK需要解析日志以便为日志项建立索引, 非格式化的日志需要增加额外的处理逻辑来适配。存在很多业务日志并不规范,且有收敛难度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"总结:日志检索场景是一个"},{"type":"text","marks":[{"type":"strong"}],"text":"写多读少"},{"type":"text","text":"的场景, 在这样的场景下去维护一个庞大且复杂的索引, 在我们看来其实是一个性价比很低的事情。如果采用ElasticSearch方案,经测算我们需要几万核规模集群,仍然保证不了写入数据和检索效率,且资源浪费严重。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"日志检索设计"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"面对这种情况, 我们不妨从一个不同的角度去看待日志检索的场景, 用一个更适合的设计来解决日志检索的需求, 新的设计具体有以下三个点:"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1. 日志分块"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同样的我们需要对日志进行采集,但在处理日志时我们不对日志原文进行解析和索引,而是通过日志时间、日志所属实例、日志类型、日志级别等日志元数据对日志进行分块。这样检索系统可以"},{"type":"text","marks":[{"type":"strong"}],"text":"不对日志格式做任何要求"},{"type":"text","text":",并且因为没有解析和建立索引(这块开销很大)的步骤, 写入速度也能够达到极致(只取决于磁盘的 IO 速度)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/03\/21\/03df897547bb9a63ab8f10980a53f821.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"简单来说, 我们可以将一个实例产生的同一类日志按时间顺序写入到一个文件中, 并按时间维度对文件拆分. 不同的日志块会分散在多台机器上(我们一般会按照实例和类型等维度对日志块的存储机器进行分片), 这样我们就可以在多台机器上对这些日志块并发地进行处理, 这种方式是支持横向扩展的. 如果一台机器的处理性能不够, 横向再扩展就行。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那如何对入日志块内的数据进行检索呢?这个很简单, 因为保存的是日志原文,可以直接使用 grep 相关的命令直接对日志块进行检索处理。对开发人员来说, grep是最为熟悉的命令, 并且使用上也很灵活, 可以满足开发对日志检索的各种需求。因为我们是直接对日志块做追加写入,不需要等待索引建立生效,在日志刷入到日志块上时就可以被立刻检索到, 保证了检索结果的"},{"type":"text","marks":[{"type":"strong"}],"text":"实时性"},{"type":"text","text":"。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2. 元数据索引"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下来我们看看要如何对这么一大批的日志块进行检索。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先我们当日志块建立时, 我们会基于日志块的元数据信息搭建索引, 像服务名称、日志时间, 日志所属实例, 日志类型等信息, 并将日志块的存储位置做为value一起存储。通过索引日志块的元数据,当我们需要对某个服务在某段时间内的某类日志发起检索时,就可以快速地找到需要检索的日志块位置,并发处理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/16\/06\/16fd67096b1b93e91ed5b4eb7914c706.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"索引的结构可以按需构建, 你可以将你关心的元数据信息放入到索引中, 从而方便快速圈定需要的日志块。因为我们只对日志块的元数据做了索引, 相比于对全部日志建立索引, 这个成本可以说降到了极低, 锁定日志块的速度也足够理想。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3. 日志生命周期与数据沉降"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"日志数据以时间维度的方向可以理解为一种时序数据, 离当前时间越近的日志会越有价值, 被查询的可能性也会越高, 呈现一种冷热分离的情况。而且冷数据也并非是毫无价值,开发人员要求回溯几个月前的日志数据也是存在的场景, 即我们的日志需要在其生命周期里都能够对外提供查询能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对于这种情况,如果将生命周期内的所有日志块都保存在本地磁盘上, 无疑是对我们的机器容量提了很大的需求。对于这种日志存储上的需求,我们可以采用压缩和沉降的手段来解决。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/75\/16\/75f67436650eedd2191c05ae2311d216.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"简单来说,我们将日志块存储分为本地存储(磁盘)、远程存储(对象存储)、归档存储三个级别; 本地存储负责提供实时和短期的日志查询(一天或几个小时), 远程存储负责一定时期内的日志查询需求(一周或者几周), 归档存储负责日志整个生命周期里的查询需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"现在我们看看日志块在其生命周期里是如何在多级存储间流转的, 首先日志块会在本地磁盘创建并写入对应的日志数据, 完成后会在本地磁盘保留一定时间(保留的时间取决于磁盘存储压力), 在保存一定时间后, 它首先会被"},{"type":"text","marks":[{"type":"strong"}],"text":"压缩"},{"type":"text","text":"然后被上传至远程存储(一般是对象存储中的标准存储类型), 再经过一段时间后日志块会被迁移到归档存储中保存(一般是对象存储中的归档存储类型)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这样的存储设计有什么好处呢? 如下面的多级存储示意图所示, 越往下存储的数据量越大, 存储介质的成本也越低, 每层大概为上一层的 1\/3 左右, 并且数据是在压缩后存储的, 日志的数据压缩率一般可以达到"},{"type":"text","marks":[{"type":"strong"}],"text":"10:1"},{"type":"text","text":", 由此看归档存储日志的成本能在本地存储的"},{"type":"text","marks":[{"type":"strong"}],"text":"1%"},{"type":"text","text":"的左右, 如果使用了 SSD 硬盘作为本地存储, 这个差距还会更大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"价格参考:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.