【爬蟲】Scrapy配合Selenium爬取京東動態加載的商品信息

【原文鏈接】https://www.cnblogs.com/cnkai/p/7570116.html

在之前的一篇實戰之中，我們已經爬取過京東商城的數據，但是前面的那一篇其實是有一個缺陷的，不知道你看出來沒有，下面就來詳細的說明和解決這個缺陷。

我們在京東搜索頁面輸入關鍵字進行搜索的時候，頁面的返回過程是這樣的，它首先會直接返回一個靜態的頁面，頁面的商品信息大致是30個，之所以說是大致，因爲有幾個可能是廣告商品，之後，當我們鼠標下滑的使用，京東後臺使用Ajax技術加載另外的30個商品數據，我們看上去是60個數據，其實這60個數據是分兩次加載出來的，而且只是在你鼠標下滑到一定的位置纔會加載那另外的30個數據。

當你點擊頁面最後的第二頁的時候，仔細觀察新的url你會發現它的頁面顯示是第三頁。下面是初始第一面和點擊第二頁之後的url：

# 第一頁的url
https://search.jd.com/Search?keyword=%E6%89%8B%E6%9C%BA&enc=utf-8&qrst=1&rt=1&stop=1&vt=2&wq=%E6%89%8B%E6%9C%BA&cid2=653&cid3=655&page=1&s=1&click=0

# 點擊第二頁的url
https://search.jd.com/Search?keyword=%E6%89%8B%E6%9C%BA&enc=utf-8&qrst=1&rt=1&stop=1&vt=2&wq=%E6%89%8B%E6%9C%BA&cid2=653&cid3=655&page=3&s=59&click=0

page參數一個是1，一個是3。因此可以得知，京東的第二頁的信息是使用Ajax加載出來的，而不是直接請求url的形式。如果我們想要拿到另外的30個信息，一方面是需要js渲染，另一方面是實現滾動條下拉，觸發Ajax請求。

知道了過程，下面就是着手解決這個問題，由於Scrapy框架只能加載靜態數據，因此我們需要另外的工具來配合Scrapy實現爬取頁面的完整信息。

我們的技術路線是這樣的，使用selenium加Firefox來實現目的。實現的過程是這樣的，將selenium作爲scrapy的下載中間件（Downloader Middleware），執行js腳本實現滾動條的下拉，並且實現js的渲染。

下面就來演示。

scrapy startproject Jingdong

spider.py文件

# -*- coding: utf-8 -*-
"""
Created on Tue Jul 24 17:17:24 2018

@author: Administrator
"""

from scrapy import Spider,Request
from selenium import webdriver

class JingdongSpider(Spider):
    name = 'jingdong'

    def __init__(self):
        #NotADirectoryError: [WinError 267] 目錄名稱無效。: 'E:\\software\\python\\geckodriver-v0.21.0-win64\\geckodriver.exe'
        #self.browser = webdriver.Firefox('E:\software\python\geckodriver-v0.21.0-win64\geckodriver.exe')
        self.browser = webdriver.Firefox(executable_path='E:\software\python\geckodriver-v0.21.0-win64\geckodriver.exe')
        self.browser.set_page_load_timeout(30)

    def closed(self,spider):
        print("spider closed")
        self.browser.close()

    def start_requests(self):
        start_urls = ['https://search.jd.com/Search?keyword=%E6%95%B0%E7%A0%81%E7%9B%B8%E6%9C%BA&enc=utf-8&wq=%E6%95%B0%E7%A0%81%E7%9B%B8%E6%9C%BA&pvid=86343bd54f6a4ed3baa79001538b52e7'.format(str(i)) for i in range(1,2,2)]
        for url in start_urls:
            yield Request(url=url, callback=self.parse)


    def parse(self, response):
        selector = response.xpath('//ul[@class="gl-warp clearfix"]/li')
        print(len(selector))
        print('---------------------------------------------------')

這裏將webdriver定義在spider文件的好處是，不需要每一次請求url都打開和關閉瀏覽器。
其中的closed()方法，是在爬蟲程序結束之後，自動關閉瀏覽器。
由於這裏是演示之用，我們就以一個頁面來測試，看一下最後的結果是不是返回60條數據，如果是60條左右就證明我們的selenium起作用了，如果僅僅是30條左右的數據，就證明失敗。

middlewares.py

下面的這個文件是主要邏輯實現。在程序中，我們執行了js腳本，來實現滾動條下拉，觸發Ajax請求，之後我們簡短的等待，來加載Ajax。在scrapy官方文檔中，對下載中間件有着比較詳細的說明，當某個下載中間件返回的是response對象的時候，之後的下載中間件將不會被繼續執行，而是直接返回response對象。

# -*- coding: utf-8 -*-

# Define here the models for your spider middleware
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/spider-middleware.html

from scrapy import signals
from scrapy.http import HtmlResponse
from selenium.common.exceptions import TimeoutException
import time

class SeleniumMiddleware(object):
    def process_request(self, request, spider):
        if spider.name == 'jingdong':
            try:
                spider.browser.get(request.url)
                spider.browser.execute_script('window.scrollTo(0, document.body.scrollHeight)')
            except TimeoutException as e:
                print('超時')
                spider.browser.execute_script('window.stop()')
            time.sleep(2)
            return HtmlResponse(url=spider.browser.current_url, body=spider.browser.page_source,
                                encoding="utf-8", request=request)

class JingdongSpiderMiddleware(object):
    # Not all methods need to be defined. If a method is not defined,
    # scrapy acts as if the spider middleware does not modify the
    # passed objects.

    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
        return s

    def process_spider_input(self, response, spider):
        # Called for each response that goes through the spider
        # middleware and into the spider.

        # Should return None or raise an exception.
        return None

    def process_spider_output(self, response, result, spider):
        # Called with the results returned from the Spider, after
        # it has processed the response.

        # Must return an iterable of Request, dict or Item objects.
        for i in result:
            yield i

    def process_spider_exception(self, response, exception, spider):
        # Called when a spider or process_spider_input() method
        # (from other spider middleware) raises an exception.

        # Should return either None or an iterable of Response, dict
        # or Item objects.
        pass

    def process_start_requests(self, start_requests, spider):
        # Called with the start requests of the spider, and works
        # similarly to the process_spider_output() method, except
        # that it doesn’t have a response associated.

        # Must return only requests (not items).
        for r in start_requests:
            yield r

    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)


class JingdongDownloaderMiddleware(object):
    # Not all methods need to be defined. If a method is not defined,
    # scrapy acts as if the downloader middleware does not modify the
    # passed objects.

    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
        return s

    def process_request(self, request, spider):
        # Called for each request that goes through the downloader
        # middleware.

        # Must either:
        # - return None: continue processing this request
        # - or return a Response object
        # - or return a Request object
        # - or raise IgnoreRequest: process_exception() methods of
        #   installed downloader middleware will be called
        return None

    def process_response(self, request, response, spider):
        # Called with the response returned from the downloader.

        # Must either;
        # - return a Response object
        # - return a Request object
        # - or raise IgnoreRequest
        return response

    def process_exception(self, request, exception, spider):
        # Called when a download handler or a process_request()
        # (from other downloader middleware) raises an exception.

        # Must either:
        # - return None: continue processing this exception
        # - return a Response object: stops process_exception() chain
        # - return a Request object: stops process_exception() chain
        pass

    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)

之後，我們只需要在設置中將當前的下載中間件添加到settings.py文件中，就可以實現了。

# -*- coding: utf-8 -*-

# Scrapy settings for Jingdong project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://doc.scrapy.org/en/latest/topics/settings.html
#     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://doc.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'Jingdong'

SPIDER_MODULES = ['Jingdong.spiders']
NEWSPIDER_MODULE = 'Jingdong.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'Jingdong (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'Jingdong.middlewares.JingdongSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#    'myproject.middlewares.CustomDownloaderMiddleware': 543,
DOWNLOADER_MIDDLEWARES = {
    'Jingdong.middlewares.SeleniumMiddleware': 543,
}

# Enable or disable extensions
# See https://doc.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
#    'Jingdong.pipelines.JingdongPipeline': 300,
#}

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

To put our spider to work, go to the project’s 最高一層的目錄 and run:

scrapy crawl jingdong

控制檯結果：

2018-07-24 18:13:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://search.jd.com/Search?keyword=%E6%95%B0%E7%A0%81%E7%9B%B8%E6%9C%BA&enc=utf-8&wq=%E6%95%B0%E7%A0%81%E7%9B%B8%E6%9C%BA&pvid=86343bd54f6a4ed3baa79001538b52e7> (referer: None)
60

大功告成！

【爬蟲】Scrapy配合Selenium爬取京東動態加載的商品信息

spider.py文件

middlewares.py

測試人員都是畫畫大神，讓我看看誰還不會用代碼圖？

Object.values()對象遍歷

我拍了拍Redis，被移出了羣聊···

網絡現代化通向雲原生應用的高速公路

面試官：說說你對序列化的理解

【Sqoop】Export data into RDBMS using Sqoop 及其調優

【NLP】Python中文文本聚類

【NLP】Python英文文本聚類

【NLP】Jieba中文分詞

【Python】解決matplotlib圖例中文亂碼問題——win10版本

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結