基于 TensorFlow.js 和 MoveNet 的下一代姿态检测

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"近日，我们非常高兴地发布了最新的姿态检测模型 MoveNet，并在 TensorFlow.js 中推出了新的姿态检测 API。MoveNet 是一个超快速和准确的模型，能够检测出身体的 17 个关键点。这个模型在 TF Hub 上提供了两种变体，即 Lightning 和 Thunder。Lightning 用于延迟关键型应用，而 Thunder 用于需要高精度的应用。在大多数现代台式机、笔记本电脑和手机上，这两种模型都比实时要快（30+FPS），这对于实时性的健身、运动和健康应用都是至关重要的。通过在浏览器中使用 TensorFlow.js 在客户端完全运行模型，无需调用服务器，也无需在初始页面加载之后安装任何依赖关系，就可以实现这一点。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/9c\/9c1bc1e678048465414f4c747f09c6a5.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"MoveNet 可以通过快速运动和非典型姿势来追踪关键点"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在过去的五年里，人体姿势追踪技术有了很大进展，但在许多场景中仍未得到广泛应用。那是因为人们更加关注与让姿势模型变得更大、更准确，而不是为了让它们能够迅速部署到任何地方。对于 MoveNet 来说，我们的任务是设计并优化一个模型，在保持尽可能少的推理时间的同时，利用先进架构的最佳方面。这样的模型可以在不同的姿势、环境和硬件设置中提供准确的关键点。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了解 MoveNet 是否有助于解锁患者的远程医疗，我们与数字健康和性能公司 IncludeHealth 进行了合作。IncludeHealth 开发了一个交互式网络应用程序，指导患者在自己舒适的家中（使用手机、平板电脑或笔记本电脑）完成各种日常操作。这些程序是由物理治疗师以数字方式设定的，用来检验平衡、力量和活动范围。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这项服务需要一个基于网络和本地运行的隐私姿势模型，它能提供高帧率下的精确关键点，然后用来量化和定量分析人体姿势和运动。一种典型的现成检测器，对于肩部外展或全身下蹲等简单动作来说是足够的，而更为复杂的姿势，比如坐姿膝盖伸展或仰卧姿势（躺下），甚至对于针对错误数据进行训练的最先进的检测器来说，也会带来麻烦。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/ad\/adb2cb47257d840227283b57c03a7631.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"传统探测器（上图) 和 MoveNet（下图) 在高难度姿势下的对比"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们为 IncludeHealth 提供了早期版本的 MoveNet，可以通过新的 pose-detection API 进行访问。这个模型是通过健身、舞蹈和瑜伽姿势训练得到的（详细信息见下面的训练数据集）。IncludeHealth 在其应用程序中集成了这个模型，并用 MoveNet 和其他可用的姿势检测器做了基准测试："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“MoveNet 模式注入了强大的组合，提供治疗所需的速度和准确性。在其他模式相互交换时，这种独特的平衡使得下一代护理服务得以展开。在这个过程中，谷歌团队是一个杰出的合作伙伴。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"—— Ryan Eder，IncludeHealth 公司的创始人兼首席执行官。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作为下一步，IncludeHealth 将与医院系统、保险机构等合作，将传统的护理和培训延伸到实体医院之外。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/b1\/b1a5a5bfa0160db75ad565ac36efe920.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"在浏览器中运行的 IncludeHealth 演示应用程序，使用由 MoveNet 和 TensorFlow.js 支持的关键点估计对平衡和运动进行量化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"安装"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有两种方法来使用 MoveNet 和新的姿势检测 api："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过 NPM："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"null"},"content":[{"type":"text","text":"import * as poseDetection from '@tensorflow-models\/pose-detection';\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过脚本标记："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"null"},"content":[{"type":"text","text":"