scrapy-redis的settings文件配置

提前安裝scrapy_redis庫，下載redis數據庫，打開redis服務端，然後直接創建scrapy文件，最後添加以下常量至settings文件即可實現分佈式

DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
SCHEDULER_PERSIST = True    # 爲false Redis關閉了 Redis數據也會被清空
REDIS_URL = "redis://127.0.0.1:6379"

直接使用scrapy框架的方式，以下爲spider文件

import scrapy

class JdSpider(scrapy.Spider):
    name = 'jd'
    allowed_domains = ['jd.com', 'p.3.cn']
    start_urls = ['https://book.jd.com/booksort.html']

    def parse(self, response):
        # 大分類列表
        dt_list = response.xpath("//div[@class='mc']/dl/dt")
        for dt in dt_list:
            item = {}
            item['b_cate'] = dt.xpath("./a/text()").extract_first()
            yield item

創建rediscrawlspider的時候需要繼承導入的RedisCrawlSpider，並且添加start_url的時候需要用redis客戶端使用lpush壓入redis_key才能啓動，如果是繼承RedisSpider的話，也是需要壓入start_urls的。下面是spider文件

from scrapy.spiders import Rule
from scrapy.linkextractors import LinkExtractor
from scrapy_redis.spiders import RedisCrawlSpider

class DangdangSpider(RedisCrawlSpider):
    name = 'mycrawler_redis'
    redis_key = 'mycrawler:start_urls'
    allowed_domains = ['dangdang.com']
    
    rules = (
        # follow all links
        # 列表頁面
        Rule(LinkExtractor(restrict_xpaths="//ul[@class='title-state-ul']/li"), callback='parse_page'),
        # 列表頁面翻頁
        Rule(LinkExtractor(restrict_xpaths="//a[@class='arrow-page prov_rota']"), follow=True),
    )

    def parse_page(self, response):
        item = {}
        item['content'] = response.xpath("//div[@class='details-box']/pre/text()").extract()
        print(item)