数据采集方案设计与实践

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一. 数据采集是什么？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所谓“数据采集”，指的是针对特定用户行为或事件进行捕获、处理和发送的相关技术及其实施过程。比如用户某个icon点击次数、观看某个视频的时长等等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"数据采集的技术实质，是先监听软件应用运行过程中的事件，当需要关注的事件发生时进行判断和捕获。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在数据分析的整个体系中，通常是由数据采集、数据传输、数据建模、数据统计\/分析、数据可视化反馈5个步骤组成，我们认为，第一个步骤，也即数据采集是最核心的问题。数据采集是否丰富，采集的数据是否准确，采集是否及时，都直接影响整个数据分析的效果。所以如何选择正确的数据采集方式，采集哪些数据对做好数据分析至关重要。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、数据采集的现状"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"企业在数据采集的道路上经常会遇到各种各样的问题，如何采？采哪些？用什么手段？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前现有企业的数据采集工作，通常会选择三种途径，分别是第三方统计工具、通过业务数据库做统计分析和、通过后端接口去做代码打点并结合业务数据库做精细化统计分析。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、第三方统计工具"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中，友盟、百度统计等第三方统计工具，通过嵌入 APP SDK 或 JS SDK 来直接查看统计数据。这种方式简单、免费，基本满足宏观基础数据分析需求，如访问量、活跃用户量等。但使用这类统计工具的用户很快便会发现简单免费的同时存在一些问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"a．"},{"type":"text","marks":[{"type":"strong"}],"text":"由于数据采集不够完整，无法实现深度分析"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这种方式的 SDK 只能采集到一些基本的用户行为数据，如设备的基本信息、用户执行的基本操作等数据，无法采集到一些精细化的维度。例如，在一些提交操作中，提交对应的个人信息、事项等信息无法采集，导致后续的分析成了“巧妇难为无米之炊”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"b. "},{"type":"text","marks":[{"type":"strong"}],"text":"安全顾虑"},{"type":"text","text":"，云模式的数据分析平台让不少企业不愿意将核心数据放在第三方平台上。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"c. "},{"type":"text","marks":[{"type":"strong"}],"text":"基于第三方平台，缺乏灵活度，无法满足定制化的需求，"},{"type":"text","text":"比如数据采集的各种采集策略动态控制等等。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、通过业务数据库做统计分析"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过业务数据库实现统计分析时，一些基于业务数据库中存储的事项、用户信息等数据，进行常规的统计分析需求，实时且准确，但也有不足之处。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，性能较差，无法进行批量数据操作。业务数据表设计针对高并发、低延迟的小操作，而数据分析常常针对大数据进行批量操作，导致性能很差。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其次，缺少必要的数据字段。业务数据库是为满足正常的业务运转服务的，而有些分析需求用到的信息并不会在业务数据库中出现。比如浏览器版本信息，设备信息等，我们在进行数据分析时就会用到，分析不同设备版本的用户转化情况，但是正常的业务流程并不使用，这时我们就无法进行对应的分析。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3、通过后端接口去做代码打点"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在某个控件操作发生时通过预先写好的代码来发数据的代码埋点，能够做到精细化的获取用户行为数据，但包括用户的绑定、用户行为路径的分析，页面时长的统计、以及后续数据的流转并没有成体系，往往还是需要结合业务数据库去做最终的数据统计分析，所以与通过业务数据库做统计分析存在同样的性能问题。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、数据采集系统总体方案设计"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基于上述现状，结合政务项目自身特点（我们希望能保证采集数据的安全问题，又能携带用户属性和事件属性实现精细化的数据运营，还需要保证批量分析操作的性能，同时又能灵活方便的动态修改数据采集的采集策略，满足需求的多变性），我们搭建了一套自己的数据采集系统。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们整个数据集采系统共包括5个模块："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"前端sdk、后端服务、etl解析模块、可视化埋点websocket服务、后台管理端。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们的数据采集系统支持多个前端渠道（包括PC、移动端（iOS、android）、小程序）的sdk集成，同时支持多种数据采集方式，包括代码埋点、全埋点、可视化埋点三种。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"a、代码埋点"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"是目前常用的数据采集方式，主要包括web、h5页面的JS埋点、移动端的iOS、Android埋点、微信小程序等.通过代码方式进行埋点，优点是数据采集比较全面、准确"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"b、全埋点"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相对于传统的采集方式，全埋点的采集方式更加简单、快捷，并且可以看到页面元素点击的情况，更加了解自身的产品特点。缺点是采集的数据过于多，只要是可点击元素都会采集，上传数据多，消耗流量多。无法采集到更深维度的信息，如事件的属性，用户的属性等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"c、可视化埋点"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可视化埋点是基于全埋点之上，需要业务同事对页面的元素进行圈选，被选择的元素才会采集。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可视化埋点基本和全埋点相同，具有同样的优缺点，虽然解决了全埋点数据杂乱的问题，但是每次页面的结构变化，都会使选择失效，需要重新圈选才可以，业务人员工作量较大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以根据不同的业务需求去选择不同的数据采集方式组合，从而真正实现精细化的数据采集。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同时我们还支持以云配置的方式，满足不同的数据采集策略实时变更的需求，省去了传统的变更采集策略需要发版本的问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b6\/b65cc99dae6318fdd81dde74d2f829ee.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、核心技术实现"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、全埋点"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"a、代码拦截系统事件"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以 iOS 为例。动态地在函数调用前后插入相应的代码，在 Objective-C 中我们可以利用 Runtime 特性，用 Method Swizzling 来 hook 相应的函数，为了给所有类方便地 hook，我们可以给 NSObject 添加个 Category，名字叫做 NSObject+MethodSwizzling，"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"+(void)swizzleMethod:(SEL)originalSelector swizzledSelector:(SEL)swizzledSelector {\nClassclass=[selfclass];\n\/\/原有方法\nMethod originalMethod = class_getInstanceMethod(class, originalSelector);\n\/\/替换原有方法的新方法\nMethod swizzledMethod = class_getInstanceMethod(class, swizzledSelector);\n\/\/先尝试给源SEL添加IMP，这里是为了避免源SEL没有实现IMP的情况\n BOOL didAddMethod = class_addMethod(class,originalSelector,\n method_getImplementation(swizzledMethod),\n method_getTypeEncoding(swizzledMethod));\nif(didAddMethod){\/\/添加成功：表明源SEL没有实现IMP，将源SEL的IMP替换到交换SEL的IMP\n class_replaceMethod(class,swizzledSelector,\n method_getImplementation(originalMethod),\n method_getTypeEncoding(originalMethod));\n}else{\/\/添加失败：表明源SEL已经有IMP，直接将两个SEL的IMP交换即可\n method_exchangeImplementations(originalMethod, swizzledMethod);\n}\n}\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"b、全量收集"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"全量收集采用 hook AppDelegate 代理、UIViewController 生命周期、按钮点击事件、手势事件、各种系统控件的点击回调方法、应用状态切换等实现。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

动作	事件
UIViewController 生命周期函数	给 UIViewController 添加分类，hook 生命周期
UIButton 等点击	UIButton 添加分类，hook 点击事件
手势事件 UITapGestureRecognizer、UIControl、UIResponder	相应系统事件"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以统计PV 事件为例，我们对 UIViewController 进行 hook，"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\/\/ load 方法里面添加 dispatch_once 是为了防止手动调用 load 方法。\n+(void)load {\nstaticdispatch_once_t onceToken;\n dispatch_once(&onceToken,^{\n@autoreleasepool{\n[[selfclass] swizzleMethod:@selector(viewWillAppear:) swizzledSelector:@selector(zg_viewDidAppear:)];\n[[selfclass] swizzleMethod:@selector(viewWillDisappear:) swizzledSelector:@selector(zg_viewDidAppear:)];\n}\n});\n}\n\n-(void)zg_viewDidAppear:(BOOL)animated {\n\/\/ do something\n CCBFT * CCBFT =[CCBFT sharedInstance];\nNSMutableDictionary*data =[NSMutableDictionary dictionary];\n[data setObject:@\"pv\" forKey:@\"$eid\"];\n[data setObject:isNil([selfCCBFTScreenName]) forKey:@\"$url\"];\n[data setObject:isNil([selfCCBFTScreenTitle]) forKey:@\"$page_title\"];\n[data setObject:isNil(CCBFT.ref) forKey:@\"$ref\"];\n[CCBFT autoTrack:data];\n[CCBFT startTrack:@\"test\"];\n[self zg_viewDidAppear:animated];\n}\n-(void)zg_viewDidDisappear:(BOOL)animated {\n\/\/ do something\n CCBFT * CCBFT =[CCBFT sharedInstance];\n[CCBFT endTrack:@\"test\" properties:@{@\"\":@\"\"}];\n[self zg_viewDidDisappear:animated];\n}"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、可视化埋点"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/69\/69696b6a422abedfe4d9c338f60c0857.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"可视化埋点方案图示"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根据标识来识别每一个事件， 针对指定的事件进行取参埋点。而事件的标识与参数信息都写在配置表中，通过动态下发配置表来实现埋点统计。设置状态通过特定动作来与前端建立socket连接，并传递当前应用的界面信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"工作状态从服务器后台获取当前的配置信息，依据路径表来查找想要监测的view，并添加代理来统计行为，利用一份配置表来管理这个“事件唯一标识符“。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这里主要分为两个部分 ："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"a、事件的锁定"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"事件的锁定主要是靠 “事件唯一标识符”来锁定，而事件的唯一标识是由我们写入配置表中的。这里分为两种，本地配置表和线上下载的配置表。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"b、埋点数据的上报。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"埋点数据的数据又分为两种类型：固定数据与可变的业务数据， 而固定数据我们可以直接写到配置表中，通过唯一标识来获取。而对于业务数据，我是这么理解的：数据是有持有者的，例如我们Controller的一个属性值，又或者数据再Model的某一个层级。这么的话我们就可以通过KVC的的方式来递归获取该属性的值来取到业务数据。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"五、总结"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"总而言之，要做好用户数据行为分析，数据源很重要，我们要更“全”、更“细”地采集数据。无论选取什么样的数据采集方式，这些都是手段，需要根据不同的应用场景，灵活设计数据采集方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上，就是我们针对北京事业群政务项目需要选择的数据采集方式的介绍，希望能对你们了解数据采集与埋点有帮助！"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文转载自：金科优源汇（ID：jkyyh2020）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文链接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/xFymhdKaTlbjyHn7n9Z5CQ","title":"xxx","type":null},"content":[{"type":"text","text":"数据采集方案设计与实践"}]}]}]}