教你用5步检测出企业网络中的恶意爬虫

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"恶意爬虫（bot）在企业网络的安全漏洞中起到至关重要的作用。这已经不是什么秘密。爬虫经常被恶意软件利用，在企业网络中传播。但检测和移除恶意爬虫却很复杂，这是由于操作环境中的许多日常进程，诸如软件更新，用的都是爬虫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"直到最近，一直都没有什么有效的方式让安全团队能区分出“好爬虫”和“坏爬虫”。开源源码和社区规则声称它们可以辨别爬虫，但收效甚微；误报太多。最后，安全分析专家会因为追踪分析“好爬虫”触发的无关紧要的安全警报而疲于奔命。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Cato，我们保护客户网络时也面临相同问题。为解决这个问题，我们发明了一种新方法，在我们的安全即服务中实施的多维度方法论，可以比单纯使用开源源码或社区规则多鉴别72%以上的恶意事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最重要的是，你可以在自己的网络中采用类似策略。您的工具将会是任何网络工程师的交易手段：访问您的网络，像分接传感器一样捕获流量，保留足够的磁盘空间来存储一周的数据包。下面是如何分析这些捕获的数据包，以便更好地保护你的网络。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"辨别恶意爬虫和流量的五项原则"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在这篇文章中，我们将引入多维度的方法来分辨恶意爬虫。仅凭一条规律或许并不能让我们准确辨别出恶意爬虫，但综合多条规则并加以分析利用，将让这些爬虫无所遁形。方法并不难懂，是常见的逐步缩小狩猎范围，从人们日常产生的会话缩小到危害网络安全的那部分会话。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具体步骤如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"区分人类与爬虫"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"区分浏览器与其他用户客户端"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"找出浏览器中的爬虫"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分析payload"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"确定目标威胁程度"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面，让我们一步一步地看："}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"根据对话频率区分人类与爬虫"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爬虫的本质决定了它们会更倾向于与对象进行连续不断的会话。这是由于它们需要接收命令、发送KeepAlive信号，或者是渗出数据。区分爬虫与人类行为的第一步便是要揪出这些与目标机械性的重复对话。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们需要找到那些与多个目标有规律且不间断对话的主机，过往经验告诉我们，收集一周左右的流量便足以判断这些客户端与目标间对话的本质。从统计学的角度来看，这些会话越是有规律，他们越有可能是由爬虫生成的（见图一）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/cf\/cf54b98bceb0c7849991f4554369b3b7.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"图中所示为今年五月中旬时收集的爬虫对话频率，对话产生的流量均匀分布，基本可以确定是爬虫流量。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"区分浏览器与其他用户客户端"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"仅仅是分辨出爬虫寄居的主机并没有太大的用处，如我们之前提到的，大多数的主机都可能产生爬虫流量，因此我们还需要搞清楚网络中客户端的通信类型。一般来说，良性爬虫存在于浏览器之中，而恶意爬虫则相反。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"操作系统中有很多种客户端和库都会产生流量，像是“Chrome”、“WinINet”，以及“Java Runtime Environment”都是客户端类型。乍一看这些客户端产生的流量可能大同小异，但通过一些方法，我们还是可以将他们区分开来的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先是应用层的头。因为大多数防火墙都会在设置里允许HTTP和TLS访问任意IP，许多爬虫都会利用这些协议与他们的目标进行对话。我们只需要辨别客户端设置中的HTTP和TLS特征群组，便可轻易揪出在浏览器外运行的爬虫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每一个HTTP session都有请求头来定义请求，以及服务器端的应对方法。这些头的序列、值，都是在生成HTTP请求的时候就设定好的（见图二）。类似的，TLS的session属性，例如加密套件、扩展列表、ALPN（应用层协议协商）和椭圆曲线等，这些都是在初始的TLS消息中确定的。这条初始的消息又被称作“ClientHello”消息，也是未经过加密处理的。根据HTTP和TLS属性种类的不同进行聚类，可以在一定程度上将爬虫进行分类。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"举例来说，如果我们捕获到了加密套件不相符的TLS流量，那么我们基本就可以确定这些流量是在浏览器外生成的了，这种非人为行为也就意味着是爬虫在制造流量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/4d\/4d2ac8dc7ae3137410f8b46a668d3b36.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"图中的例子是某条由Windows加密库生成的消息的头。通过查看与设置不相符的seq、key以及value可以辨别出爬虫的存在。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"找出浏览器中的爬虫"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"判定恶意爬虫的另一种方法则是通过观察HTTP头中的一些特定信息：网络浏览器的HTTP头通常都是规范且清晰的。在正常的浏览session中，点击浏览器中的链接后生成的请求头中会包含Referer，标明这个链接的来源。爬虫流量则相反，直接访问链接的请求头中Referer会为空，部分恶意访问甚至可以仿造Referer。因此，在所有信息流中都长得一模一样的请求，大概率是爬虫流量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/58\/58440ec2b7fcce0b0eab6a92d59cb6a2.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图中显示的是在浏览session中包含Referer的头的示例"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"User-Agent（UA）这个字符段代表了程序在发起对话请求，一些诸如fingerbank.org这类的第三方服务会将UA里的程序版本号与已知版本相对应，并试图通过这条信息检测异常爬虫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"举例来说，最新版本浏览器会在UA字段使用“Mozilla\/5.0”，低于这个版本的Mozilla或者缺失这条信息的请求基本意味着会是异常爬虫。有信誉的浏览器产生都流量不会不携带UA值。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Payload分析"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是，我们并不希望异常爬虫的检查被局限在HTTP和TLS协议上，所有我们需要将更多的协议纳入考虑范围。以IRC协议为例，IRC上的爬虫为僵尸网络贡献了不少有生力量。我们可以通过观察在已知端口上，使用固定未知协议的现有恶意软件样本，利用应用识别来标记恶意爬虫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而流量的方向（入方向或出方向）在鉴别爬虫时同样有重要价值。直连到互联网的设备通常会被暴露在扫描操作中，因此，我们可以将这些爬虫看作是入站扫描器。另一方面，出站扫描行为则代表了该设备已被扫描器感染。感染的后果则是目标可能会被袭击，降低企业的IP地址信誉。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下图显示了在某一时间段内的流量图，这样的活动轨迹很可能是扫描机器人的杰作。这种类型的图可以通过计算流量\/秒来进行数据分析。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a9\/a9586e9e279b69eb0f86a9ebbee8d3e5.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"高频率出站扫描活动示例"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"目标分析：了解你的目的地是什么"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在了解过从客户端-服务器的通信频率以及客户端类型判断恶意爬虫后，我们将引入另一判定维度：爬虫的目标或者说目的地。判定恶意爬虫的目标需要考虑两种因素："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一是目标的信誉"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"二是目标的受欢迎度"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目标信誉可以从各种渠道中收集来的经验，判断一个域受到恶意攻击的可能性而计算得出。信誉的判定需要由第三方服务给出，或者是通过收集用户反馈的受到攻击的报告得出。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然而，通常情况下，仅仅是凭借着对目标信誉、URL信誉的判断，并不足以标记恶意爬虫。每个月都要有数以百万计的新域名注册成功，在没有充足背景调查下的域名信誉判定系统并不能很好地给出该域名是否可信的判断，由此带来的高误判率也证明了这一点。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"总而言之"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"将文中的方法总结后，可以发现，如果session满足以下条件，其为恶意爬虫的可能性很高："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由计算机生成而非人为"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"产生于浏览器之外，或是带有异常元数据的浏览器流量"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"与低人气目标沟通，目标如果被标记为恶意或未归类，那么将更可疑。正常或良性爬虫不应当与低人气目标对话。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"练习：揭开仙女座恶意软件的网络面纱"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过综合这几种判定手段，我们可以找出网络中的各类危险因素。光说不练假把式，下面我们将通过“仙女座爬虫”这个经典例子来练习。“仙女座”是其他恶意软件的常用下载器，而通过文中介绍的四种手段进行数据分析，我们将揭露“仙女座”爬虫的真面目。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"目标声誉"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"多家高信誉的网站均判断“disorderstatus.ru”为恶意域名，他们给这个域名打上的标签大多是：“已知感染源”，“僵尸网络”之类。然而，仅凭如此，我们并不能直接判定与该域名对话的主机感染了仙女座病毒：用户可能只是访问过这个网站而已。更何况，这个URL大概率只会被归类为“未知”或“非恶意”。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"目标受欢迎度"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"访问某个目标的用户万中无一，这个“一”很不寻常，也为该贡献了“低人气”分数。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"对话频率"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在一周的数据收集中，我们注意到了客户端与目标在连续三天内均有流量产生。重复的对话意味着爬虫的存在。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/10\/10ec4a0698534eea07066898b3354e3c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"用户与disorderstatus.ru在三天内均有持续一小时的client-target对话流量产生"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"头分析"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"请求头中的“User-Agent”值为“Mozilla\/4.0”，无效的服务器版本信息意味着该UA多半是爬虫制造。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/83\/831a6707b1a40ef3d92afb485fe8e710.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"图中是我们在与disorderstatus.ru对话时捕获的HTTP头。值得注意的是，在我们捕获的所有请求中，没有一个头包含“Referer”。另外，UA值为“Mozilla\/4.0”。这两点都指向了Andromeda会话。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"总结"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"IP网络中的爬虫检测正在逐渐成为检测恶意软件的基本组成部分之一，虽然不易，但我们相信，通过文中介绍的这五种手段的组合变种，爬虫检测将会更加有效。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文链接："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/www.catonetworks.com\/blog\/how-to-identify-malicious-bots-on-your-network-in-5-steps"}]}]}