如何将数亿MySQL数据无缝迁移到MongoDB？

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在好大夫在线内部，S3系统负责各业务方操作日志的集中存储、查询和管理。目前，该系统日均查询量数千万次，插入量数十万次。随着日志量的不断累积，主表已经达到数十亿，单表占用磁盘空间400G+。S3是业务早期就存在的系统，当时为了简单快速落地，使用了Mysql来存储，随着业务的不断增长，同时也要兼顾性能和可扩展性，到了必须要重新选型的时候了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新项目命名为：LogStore。"}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"目标"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、安全性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"S3系统在设计之初，没有按业务系统考虑数据隔离，而是直接采用 key(系统 + 类名 + id) + 有限固定字段 + 序列化value 的方式进行存储，这种方式显然不便于后续集群拆分和管理。LogStore系统要在逻辑上进行数据区域划分，业务方在接入时要指定app进行必要的权限验证，以区分不同业务数据，进而再进行插入和查询操作。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、通用性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"S3主要提供一种3层结构，采用MySQL固定字段进行存储，这就不可避免的会造成字段空间的浪费。LogStore系统需要提供一种通用的日志存储格式，由业务方自行规定字段含义，并且保留一定程度的可查询维度。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3、高性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"S3系统的QPS在300+，单条数据最大1KB左右。LogStore系统要支持当前QPS 10倍以上的写入和读取速度。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4、可审计"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"要满足内部安全审计的要求，LogStore系统不提供对数据的更新，只允许数据的插入和查询。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5、易扩展"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"LogStore系统以及底层存储要满足可扩展特性，可以在线扩容，满足公司未来5年甚至更长时间的日志存储需求，并且要最大化节省磁盘空间。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"方案选型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了达成改造目标，本次调研了四种存储改造方案，各种方案对比如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/resource\/image\/18\/c0\/1825bc742e3f0174b0274ff1ee8ccbc0.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、我们不合适—分库分表"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}