推理性能提升一倍，TensorFlow Feature Column性能优化实践

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"写在前面"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在CTR（Click Through Rate）点击率预估的推荐算法场景，TensorFlow Feature Column被广泛应用到实践中。这一方面带来了模型特征处理的便利，另一方面也带来了一些线上推理服务的性能问题。为了优化推荐业务性能，提升线上服务效率，爱奇艺深度学习平台团队在实践中总结了一些性能优化方法。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"经过这些优化，推荐业务的线上推理服务性能效率可以提升一倍以上，p99延迟降低达到50%以上。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景介绍"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Feature Column是TensorFlow提供的用于处理结构化数据的工具，是将样本特征映射到用于训练模型特征的桥梁。它提供了多种特征处理方法，让算法人员可以很容易将各种原始特征转换为模型的输入，来进行模型实验。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/1d\/1ddccd183fe4a4ee2ae9d83b0ba9d45d.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上图所示，所有Feature Column都源自FeatureColumn类，并继承了三个子类CategoricalColumn、DenseColumn和SequenceDenseColumn，分别对应"},{"type":"text","marks":[{"type":"strong"}],"text":"稀疏特征"},{"type":"text","text":"、"},{"type":"text","marks":[{"type":"strong"}],"text":"稠密特征"},{"type":"text","text":"、"},{"type":"text","marks":[{"type":"strong"}],"text":"序列稠密特征"},{"type":"text","text":"。算法人员可以按照样本特征的类型找到对应的接口直接适配。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而且 Feature Column 和 TF Estimator 接口有很好的集成，通过定义好对应的特征输入就可以直接在预定义的 Estimator 模型中使用。TF Estimator 在推荐算法的使用非常普遍，特别是它封装了分布式训练的功能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下图是一个使用Feature Column处理特征，进入到 Estimator DNN Classifier的示例："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ce\/cef236beccb60b10188d4fac68b8615f.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"虽然Feature Column使用起来很方便，模型代码编写比较快，但是在爱奇艺推荐类业务的线上服务落地过程中，一些性能问题逐渐凸显，下面将逐个介绍我们在实际中碰到的一些问题，以及如何优化。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"整型特征哈希优化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"推荐类模型通常都会将ID类特征哈希到一定数量的bucket 分桶，然后转换成 Embedding再作为神经网络的输入，比如视频类ID特征，用户ID特征，商品ID特征等。示例如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d4\/d4df0bbc6059d58f9584c77d8ecf6d6d.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在` categorical_column_with_hash_bucket `的文档[2]里面说到：对于String类型的输入，会执行`output_id = Hash(input_feature_string) % bucket_size`做哈希操作，而对于整数类型的输入会先转成String类型然后再进行同样的哈希操作。通过查看源代码[3]可以看到这样的逻辑："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/8b\/8b71d83a6eb268f121ba99dd6f7d8182.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在推荐业务中，通常这类ID都是已经过某种方式的哈希，形成64bit的整型特征放到样本里面，因此必然要执行整型转化成String的操作。但是在TensorFlow 的 Timeline中可以看到函数`as_string`所对应的TF内部的`AsString` OP其实是一个比较耗时的操作，经过分析对比发现`AsString` OP的耗时通常是后面的哈希操作的3倍以上，如下图所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/c9\/c9352da2198db292bf3be01647ceee97.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"进一步分析`AsString` OP内部的代码，可以发现这个OP内部还涉及到了内存分配和拷贝操作，因此比纯哈希计算慢就可以理解了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"很自然，团队考虑去掉相关操作来做优化，因此专门编写了一个给整型做哈希的函数来做优化，示例代码如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/66\/66b14e2e747d53f924833f4ad05b461e.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"经过这样做区分类型的哈希方式，完全优化了原先耗时长的类型转换操作。这里需要注意的是新加的哈希函数对应的新OP同样需要加到 TF Serving 中。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"定长特征转换优化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"定长特征是指使用接口`tf.io.FixedLenFeature`来解析的特征，比如用户的性别，年龄等，这类特征的长度通常都是定长的，并且固定为 1 维或多维。这类特征经过接口`tf.io.parse_example` 解析成 Dense Tensor，然后经过Feature Column处理，再进入到模型的输入层。常见的代码示例如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/45\/4552c745e90cfc7362037f742e4bd19a.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上面的代码为例子，举例解析一下 TensorFlow 内部的Tensor 转换逻辑。如下图所示，两个样本user_name分别为bob和wanda，经过样本解析成shape为2的Dense Tensor，然后经过`categorical_column_with_vocabulary_list`转换，查找词表分别转成0和2，再经过`indicator_column`转换成One hot编码的Dense输入。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/30\/300e41f6fc6e1faa9ce83cdcb6650e63.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从上面的样本处理来看没有什么问题，然后再来看一下Feature Column代码内部的转换处理逻辑："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/91\/911fa3c1a490ea969f41ab1aa5c56d5e.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上图所示，在代码中Vocabulary Categorical Column 会先去除掉一些非法值，然后把输入的Dense Tensor 转换成 Sparse Tensor，在Indicator Column中会再次把Tensor从Sparse转成Dense，最后转成需要的One Hot Tensor。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先来思考一下上面两个转换操作的目的，一方面是为了去除样本数据中一些异常的值，另外一方面是这样的处理其实是同时兼顾了输入是 Sparse Tensor 的情况，如果输入是Spare Tensor就直接做Vocabulary词表查找，然后再转成Dense Tensor。这样转换虽然达到代码复用的作用，但是在性能上却有损失。如果能直接将原始的Input Tensor转换成One Hot Tensor，就可以省去两个转换过程，而且Sparse Tensor 和Dense Tensor之间的转换其实是非常耗时的操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"再回到定长特征的原始性质，对于这类定长特征来讲，在样本处理的时候如果没有值，会被填充成默认值，而且在生成样本的时候都会被保证不会出现有空值或者 -1的情况，因此异常值的处理其实是可以被省略的。最后优化后的内部转换逻辑如下图，省去了两次Sparse Tensor 和Dense Tensor之间的转换。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/eb\/eb5a910a6b0af5f966a6afdaca0aadcd.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了上面的Vocabulary Categorical Column，还有别的类似Feature Column也有同样的问题，因此针对这类特征，平台专门开发了一套优化的Feature Column接口提供给业务使用，优化性能效果还不错。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"用户特征去重优化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"推荐类算法模型都有个很典型的特点，那就是模型中会包含用户侧特征和要推荐的Item侧特征，比如视频的特征、商品的特征等。模型在线上服务部署的时候，会给一个用户推荐多个视频或商品，模型会返回这多个视频或商品的打分，然后按照打分的大小推荐给用户。由于是给单个用户做推荐，这个时候该用户的特征会根据推荐Item的数量重复多次，再发送给模型。如下是一个典型的推荐算法排序模型线上推理的示意图："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/bb\/bb13d2e6da7de0ad049a8500173dd8c8.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"图示的模型输入有3个User特征，3个Item特征，假定在对某个用户做推荐，该用户的3个特征分别对应为u1，u2和u3。这时要对两个不同的Item做推荐评分请求，也就是一个请求里面有两个Item，这两个Item分别为I1和I2，这两个Item分别有三个特征，I1对应I11，I12，I13，以此类推，这样构成一个batch size为2的推理请求。从图中可以看到，因为是给同一个用户推荐两个不同的Item，Item侧的特征是不同的，但是用户的特征被重复了两次。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面的例子只以2个Item为例，但是实际线上的服务一个推理请求会带100个Item甚至更多，因此用户的特征也会被重复100次甚至更多，重复的用户特征不仅增加了传输的带宽，而且增加了特征处理时的计算量，因此业务非常希望能解决这个问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这个问题的本源要从TensorFlow 的模型训练代码说起。TensorFlow 训练时的每一条样本是某个用户对某个Item的行为，然后经过shuffle和batch后进入到训练模型，这时候一个batch里面的数据肯定包含了多个用户行为的样本，这个和线上推理服务的输入数据格式是完全不同的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如何解决这个问题？最简单的想法，如果在线上服务就只发送一条用户特征会怎么样？快速的尝试就可以知道特征数据进入到模型输入层的时候会 concat失败。这是因为Item特征的batch size是多个，而用户特征的batch size只有1，示例如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ad\/ada8d11ef6abf5b307a8c02790084793.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了解决concat失败的问题，单纯先从模型的角度来看，可以考虑在进入到输入层之前把用户特征还原到和Item特征同样的batch size，如下图所示。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/45\/45f5ac25b7969795cda23091d51ee92a.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"显然这个从数学模型上是可行的，接下来就是怎么在TensorFlow 的代码里面实现这个想法。这里需要注意的是复制的操作只能在线上服务的模型里面，不能在训练的模型里面。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前TF Estimator 接口在推荐类算法的应用比较常见，而Estimator 接口提供了很好的模型区分方法，通过判断ModeKeys为`tf.estimator.ModeKeys.PREDICT`时是线上的服务模型，ModeKeys为`tf.estimator.ModeKeys.TRAIN`时是训练模型，下面是示例代码："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7a\/7a4b7a6a4b330d71d619739ef12809d1.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在实际的模型上，需要将User和Item的feature column区分开来分别传入，这个对原来的模型代码改动比较大，batch size的获取可以通过判断Item特征的长度来获取，这里不再赘述。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在实际的上线过程中，团队经历了两个阶段，第一个阶段是只对算法模型代码做修改，在处理用户特征时只取第一维，但是实际发送的推理请求还是会把用户特征重复多次；第二个阶段才把发送的推荐请求优化成只发送一份用户特征，这个时候模型代码不需要再做修改，已经自动适配。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/86\/86593cf987a81ce7ea29676c4bb2e96c.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上图所示，第一阶段的时候用户特征的输入还是重复多次，在模型中，对用户特征只取第一维再进行特征处理，示例代码如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/eb\/ebb733a2c8231ea212fca67e6eb9f28a.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面的模型代码可以同时适配推理请求发送重复的用户特征，或者只发送一条用户特征。因此在第二阶段的时候，不需要再修改模型代码，只需要优化发送推理请求的引擎侧代码。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"经过这样的优化，线上推理服务不需要重复发送用户特征，不仅节约了带宽，而且减少了序列化的消耗。对一个 batch 中的用户特征只做一份数据的Feature Column转换，然后做复制操作，复制消耗的时间远远小于转换的时间。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这里其实还可以做进一步的优化，将复制操作延后到第一层神经网络的矩阵乘后面，这样可以减少第一个矩阵乘的部分计算消耗。如果用户特征的维度占比比较高，优化的效果会比较明显。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"总结"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文介绍了爱奇艺深度学习平台在实践过程中总结的一些TensorFlow Feature Column优化。经过这些优化，线上的推理服务性能效率提升一倍以上，p99延迟降低达到50%以上。而且相比较于做op fuse，模型图修改等优化，这些优化在业务实际中也比较容易去落地。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最后，我们还是要肯定TensorFlow Feature Column给推荐类算法带来的特征处理便利性，它将整个特征的处理抽象出来，算法只要稍微适配一下样本特征就可以很快的做迭代和实验。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"参考文献"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1. https:\/\/www.tensorflow.org\/tutorials\/structured_data\/feature_columns"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. https:\/\/www.tensorflow.org\/api_docs\/python\/tf\/feature_column"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3. https:\/\/github.com\/tensorflow\/tensorflow"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文转载自：爱奇艺技术产品团队（ID：iQIYI-TP）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文链接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/2pV38VbvwCJkNA44HfcPuA","title":"xxx","type":null},"content":[{"type":"text","text":"推理性能提升一倍，TensorFlow Feature Column性能优化实践"}]}]}]}