參考:https://docs.scrapy.org/en/latest/topics/item-pipeline.html#topics-item-pipeline
架構圖:
Item Pipeline
就是一些簡單的處理Item的類,輸入是Item輸出也是Item,多個類就組成一個管道。
典型用法:
- 清洗數據
- 驗證數據的有效性
- 去重
- 排序
Writing your own item pipeline
process_item
(self, item, spider)
必需實現,必需返回dict或者Item或者Twisted Deferred或者觸發DropItem異常,DropItem異常將導致Item停止在Pipeline中的流動。
Parameters: |
---|
可選擇性實現的方法:
open_spider
(self, spider)
當Spider打開時調用此方法,可以做一些初始化工作,出個日誌什麼的。
Parameters: | spider (Spider object) – the spider which was opened |
---|
close_spider
(self, spider)
SPIDER關閉的時候調用,清理工作。
Parameters: | spider (Spider object) – the spider which was closed |
---|
from_crawler
(cls, crawler)
不知道什麼時候調用,通過crawler可以訪問Scrapy架構中的所有組件。
Parameters: | crawler (Crawler object) – crawler that uses this pipeline |
---|
Item pipeline example
from scrapy.exceptions import DropItem
class PricePipeline(object):
vat_factor = 1.15
def process_item(self, item, spider):
if item.get('price'):
if item.get('price_excludes_vat'):
item['price'] = item['price'] * self.vat_factor
return item
else:
raise DropItem("Missing price in %s" % item)
加一點對數據進行整形的邏輯。
Write items to a JSON file
import json
class JsonWriterPipeline(object):
def open_spider(self, spider):
self.file = open('items.jl', 'w')
def close_spider(self, spider):
self.file.close()
def process_item(self, item, spider):
line = json.dumps(dict(item)) + "\n"
self.file.write(line)
return item
Write items to MongoDB
import pymongo
class MongoPipeline(object):
collection_name = 'scrapy_items'
def __init__(self, mongo_uri, mongo_db):
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db
@classmethod
def from_crawler(cls, crawler):
return cls(
mongo_uri=crawler.settings.get('MONGO_URI'),
mongo_db=crawler.settings.get('MONGO_DATABASE', 'items')
)
def open_spider(self, spider):
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db]
def close_spider(self, spider):
self.client.close()
def process_item(self, item, spider):
self.db[self.collection_name].insert_one(dict(item))
return item
Activating an Item Pipeline component
Item pipeline寫好了以後,需要配置一下才能生效,配置項爲ITEM_PIPELINES,示例如下:
ITEM_PIPELINES = {
'myproject.pipelines.PricePipeline': 300,
'myproject.pipelines.JsonWriterPipeline': 800,
}