關於Scrapy: 如何分別從Spider、Middleware、Pipeline獲取settings.py裏的參數

運行環境

PyCharm 2018.1
Scrapy 1.5.1
Python 3.5.4
Windows 10

---

settings的優先級

官方文檔中scrapy中settings參數有四個級別：

1. 命令行選項(Command line Options)(最高優先級)
2. 項目設定模塊(Project settings module)
3. 命令默認設定模塊(Default settings per-command)
4. 全局默認設定(Default global settings) (最低優先級)

---

Spider

1. 在parse()函數中獲取settings

def parse(self, response):
	print(self.settings.get('CONFIG_KEY'))

2. 在實例化spider時獲取settings:

class MySpider:
	def __init__(self, settings, *args, **kwargs):
		super(MySpider, self).__init__( *args, **kwargs)
		print(settings.get('CONFIG_KEY'))
	@classmethod
	def from_crawler(cls, crawler, *args, **kwargs):
		spider = cls(crawler.settings, *args, **kwargs)
		spider._set_crawler(crawler)
		return spider

---

Middleware & Pipeline

1. 通過處理方法裏傳入的spider參數 獲取：
   比如Middleware中的process_spider_input方法：

def process_spider_input(response, spider):
	print(spider.setting.get('CONFIG_KEY'))

2. 在實例化時獲取settings:

class MyMiddleware:
	def __init__(self, settings):
		print(setting.get('CONFIG_KEY'))
	@classmethod
	def from_crawler(cls, crawler):
		return cls(crawler.settings)

---

一個清晰簡單但有風險的方法：get_project_setting()

from scrapy.utils.project import get_project_settings
...
def parse(self, response):
	settings = get_project_settings()
	print(settings.get('CONFIG'))

• pros: 簡單明瞭
• cons: 不能識別從command line 中傳入的參數， command line 傳入的參數具有最高優先級
  
  

---

參考文檔：

• scrapy中settings參數的使用詳解
• Using settings in spider init
• Reading settings in spider scrapy