scrapy示例爬取數據並存入mongodb

目的：爬取最新的美劇裏面的基本信息
總共5條信息 url：https://www.meijutt.com/new100.html

環境：win10
廢話不說，開始~

1.創建工程
scrapy startproject meiju100
scrapy gensipider meiju(爬蟲名稱) url(上面給的url)

2.填寫item.py 定義你需要爬取的信息
名字可隨意填寫

3.定義 爬蟲.py 這裏就是要給出解析

import scrapy
from meiju100.items import Meiju100Item


class MeijuSpider(scrapy.Spider):
    name = 'meiju'
    allowed_domains = ['www.meijutt.com/new100.html']
    start_urls = ['http://www.meijutt.com/new100.html']

    def parse(self, response):
        items = []
        selector = response.xpath('//ul[@class="top-list  fn-clear"]/li')
        for div in selector:
            item = Meiju100Item()
            item['storyName'] = div.xpath('./h5/a/text()').extract()
            item['storyState'] = div.xpath('./span[1]/font/text()').extract()
            item['category'] = div.xpath('./span[2]/text()').extract()
            item['tvStation'] = div.xpath('./span[3]/text()').extract()
            item['updateTime'] = div.xpath('./div[2]/font/text()').extract()
            if item['updateTime']:
                pass
            else:
                item['updateTime'] = div.xpath('./div[2]/text()').extract()
            items.append(item)
        return items

加上if 是因爲有些信息可能不能完全匹配

4.定義下載器中間件，這裏寫了隨機使用UA，防止被封

from meiju100.UA import ua_list
import random


class UserAgentmiddleware():
    def process_request(self, request, spider):
        agent = random.choice(ua_list)
        request.headers['User-Agent'] = agent

UA.py 就定義了一個list

5.定義數據管道（存入mongodb數據庫）
具體參考官方API 這裏修改了一下

from scrapy.conf import settings
import pymongo



class Meiju100Pipeline(object):
    def __init__(self):
        host = settings['MONGODB_HOST']
        port = settings['MONGODB_PORT']  # 端口
        db_name = settings['MONGODB_DBNAME']
        client = pymongo.MongoClient(host=host, port=port)
        db = client[db_name]
        self.post = db[settings['MONGODB_DOCNAME']] # collections的名字

    def process_item(self, item, spider):
        movie = dict(item)
        self.post.insert(movie)
        return item

6.配置好setting文件
DOWNLOADER_MIDDLEWARES = { 'meiju100.middlewares.UserAgentmiddleware': 543, }

ITEM_PIPELINES = {
   'meiju100.pipelines.Meiju100Pipeline': 301,
}

MONGODB_HOST = '127.0.0.1'
MONGODB_PORT = 27017
MONGODB_DBNAME = 'text2'
MONGODB_DOCNAME = 'meiju'

根據需要的情況自己定義好

7.運行
scrpay crawl meiju

最後通過命令行或者可視化工具RoBo 3T
c成功！！！