Scrapy中间件如何使用?Scrapy如何设置代理和更换请求头?Scrapy进阶使用[IP池、随机请求头](基于scrapy2.0+编写) ๑乛◡乛๑ Scrapy框架使用方法

中间件简介

我们先来看一下scrapy爬取数据的大致流程,中间件出现在下载器和爬虫中。设置代理和更换请求头需要在调度器到下载器之间的环节进行修改。

下载中间件

class ScrapyTextDownloaderMiddleware(object):
    # Not all methods need to be defined. If a method is not defined,scrapy acts as if the downloader middleware does not modify the passed objects.
    # 中间件并非必须定义,根据所需定义即可。

    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        # Scrapy使用此方法创建爬虫(spider)。
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
        return s

    def process_request(self, request, spider):
        # 当每个请求(request)通过下载中间件时，该方法被调用

        # 返回值必须是下列方式之一:
        # - 返回值为None(return None): 继续处理此请求
        # - 返回一个响应(Response)对象
        # - 返回一个请求(Request)对象
        # - 抛出异常(raise IgnoreRequest): 将调用已安装的下载中间件(DownloadMiddleware)的process_exception()方法
        return None

    def process_response(self, request, response, spider):
        # 当下载器完成http请求，传递响应(response)给引擎的时候调用

        # 返回值必须是下列方式之一:
        # - 返回一个响应(Response)对象
        # - 返回一个请求(Request)对象
        # - 抛出异常(raise IgnoreRequest)
        return response

    def process_exception(self, request, exception, spider):
        # 当下载器或process_request()(来自其他下载中间件)引发异常时调用.

        # 返回值必须是下列方式之一:
        # - 返回值为None(return None):  继续处理此异常
        # - 返回一个响应(Response)对象: 停止异常处理链(process_exception() chain)
        # - 返回一个请求(Request)对象: 停止异常处理链(process_exception() chain)
        pass

    def spider_opened(self, spider):
        spider.logger.info('启动爬虫(spider): %s' % spider.name)

下载中间件中process_request()方法在发送请求前执行,所以,更换代理和请求头就在process_request()方法中设置即可。

更换请求头

相关工具性网站

1. 查看当前请求头(http://httpbin.org/user-agent)
2. 请求头网站,内涵大量请求头

我们先做一个简单的请求,查看在默认情况下,我们发送的请求头为"user-agent": "Scrapy/2.0.1 (+https://scrapy.org)"

在中间件中,我们可以自行添加一个随机请求头类

class 随机请求头(object):
    def process_request(self, request, spider):
        user_agent = random.choice(spider.settings.get("PC_请求头"))
        request.headers['User-Agent'] = user_agent

在设置中,我们需要在DOWNLOADER_MIDDLEWARES中启动我们自定义的中间件并且添加请求头。

请求头

下方为我自行总结的一些比较新的PC端请求头。

# 常见PC端请求头
PC_请求头 = [
   # 谷歌
   # windows
   "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36",
   "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36",
   # Linux
   "Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2919.83 Safari/537.36",
   "Mozilla/5.0 (X11; Ubuntu; Linux i686 on x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2820.59 Safari/537.36",
   # mac
   "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2866.71 Safari/537.36",
   "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2762.73 Safari/537.36",
   "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2656.18 Safari/537.36",

   # 火狐
   # windows
   "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0",
   "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:77.0) Gecko/20190101 Firefox/77.0",
   "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:77.0) Gecko/20100101 Firefox/77.0",
   "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/75.0",
   "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:71.0) Gecko/20100101 Firefox/71.0",
   "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:69.2.1) Gecko/20100101 Firefox/69.2",
   "Mozilla/5.0 (Windows NT 6.1; rv:68.7) Gecko/20100101 Firefox/68.7",
   "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0",
   # linux
   "Mozilla/5.0 (X11; Linux ppc64le; rv:75.0) Gecko/20100101 Firefox/75.0",
   "Mozilla/5.0 (X11; Linux; rv:74.0) Gecko/20100101 Firefox/74.0",
   "Mozilla/5.0 (X11; Linux i686; rv:64.0) Gecko/20100101 Firefox/64.0",
   # mac
   "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.10; rv:75.0) Gecko/20100101 Firefox/75.0",
   "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:61.0) Gecko/20100101 Firefox/73.0",
   "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.10; rv:62.0) Gecko/20100101 Firefox/62.0",
   "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:10.0) Gecko/20100101 Firefox/62.0",
   "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.13; ko; rv:1.9.1b2) Gecko/20081201 Firefox/60.0"
]

设置代理

查看当前ip(http://httpbin.org/ip)
同样先写一个测试用的爬虫,和上述爬虫基本一致

# -*- coding: utf-8 -*-
import scrapy


class TestIpSpider(scrapy.Spider):
    name = 'test_ip'
    allowed_domains = ['httpbin.org/ip']
    start_urls = ['http://httpbin.org/ip']

    def parse(self, response):
        print(response.text)

想用更换代理,代理存储在request的meta中request.meta['proxy'],根据此我们可以添加代理,如下所示。

class 代理池(object):
    def process_request(self, request, spider):
        request.meta['proxy'] = 'http://123.207.57.145:1080'
        print(request.meta['proxy'])

简单代理池制作

免费代理由于稳定性和实效性都较短,如果你对代理有需求,我建议还是购买一个收费代理使用。免费代理中,小幻代理我个人用上相对较好,而且有相应的接口。下方代理池,我就将基于小幻代理制作,如果购买了正版代理,能拿到json格式的数据处理起来会更加方便!

# -*- coding: utf-8 -*-
import scrapy


class A代理提取Spider(scrapy.Spider):
    name = '代理提取'
    allowed_domains = ['ip.ihuan.me']
    start_urls = ['https://ip.ihuan.me/address/5Lit5Zu9.html']

    def parse(self, response):
        ip_tr = response.xpath('//*[@class="table-responsive"]/table/tbody/tr')
        for td in ip_tr:
            item = {}
            item['ip'] = td.xpath('./td[1]/a/text()').extract_first()
            item['端口'] = td.xpath('./td[2]/text()').extract_first()
            item['位置'] = td.xpath('./td[3]/a[1]/text()').extract_first()
            print(item)
            yield item

        ip_page = response.xpath('//*[@class="pagination"]/li/a[@aria-label="Next"]/@href').extract_first()
        ip_page = self.start_urls[0] + ip_page
        print(ip_page)
        yield response.follow(
            url=ip_page
        )

我们拿到ip后可以传入需要使用代理的地方进行使用,也可以存放起来,但免费ip一般失效会很快,存起来意义不大。