如何有效缩短闲鱼消息处理时长

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"随着用户数的快速增长，闲鱼的IM也迎来了前所未有的挑战。多年的业务迭代，端侧IM的代码已经因为多年的迭代层次结构不足够清晰，之前消息一些隐藏起来的数据同步问题，也随着用户数的增大而被放大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这里面的具体流程在于，后台需要同步到用户端侧的数据包，后台会根据数据包的业务类型划分成不同的数据域，数据包在对应域里面存在唯一且连续的编号，每一个数据包发送到端侧并且被成功消费后，端侧会记录当前每一个数据域已经同步过的版本编号，下一次数据同步就以本地数据域的编号开始，不断的同步到客户端。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"当然用户不会一直在线等待消息，所以之前这里端侧采用了推拉结合的方式保证数据的同步。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在线时使用ACCS实时的将最新的数据内容推送到客户端。ACCS是淘宝无线向开发者提供全双工、低延时、高安全的通道服务。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"离线启动后，根据本地的数据域编号，拉取不在线时候的数据差。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"当数据获取出现黑洞（数据包Version不连续的状态）时，触发数据同步拉取。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"分析"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"现存的一个同步策略是可以基本保障IM的数据同步的，但是也伴随着一些隐含的问题："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"短时间密集数据推送时，会快速的触发多次数据域同步。域同步回来的数据如果存在问题，又会触发新一轮的同步，造成网络资源的浪费。冗余数据包\/无效的数据内容会占用有效内容的处理资源，又对CPU和内存资源造成浪费。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"数据域中的数据包客户端是否正常消费，服务端侧无感知，只能被动地根据当前数据域信息返回数据。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"数据收取\/消息数据体解析\/存储落库逻辑拆分不够清晰，无法针对性的对某一层的代码拆分替换进行ABTest。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"针对遇见的这些问题，对闲鱼IM进行分层改造，抽离数据同步层。除了希望以后这个数据的同步内容可以用在IM之外，也希望随着稳定性的增加，赋能其他的业务场景。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文重点来看下端侧闲鱼IM数据同步的一些解决思路。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"数据同步优化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"拆分&分层"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对于服务端来说，业务侧产出数据包后，会拼接上当前的数据域信息，然后通过数据同步层将数据推送到端侧。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}