關於Scrapy: 運行"scrapy crawl" 發生 "TypeError: cannot deepcopy this pattern object"

運行環境

PyCharm 2018.1
Scrapy 1.5.1
Python 3.5.4
Windows 10

Scrapy爬蟲已經編寫完成,運行(scrapy crawl <spider_name> ) 發生如下錯誤

Traceback (most recent call last):
  ...
  File "d:\pycharmprojects\commercialcrawler\venv\lib\site-packages\scrapy\cmdline.py", line 149, in execute
    cmd.crawler_process = CrawlerProcess(settings)
  File "d:\pycharmprojects\commercialcrawler\venv\lib\site-packages\scrapy\crawler.py", line 249, in __init__
    super(CrawlerProcess, self).__init__(settings)
  File "d:\pycharmprojects\commercialcrawler\venv\lib\site-packages\scrapy\crawler.py", line 137, in __init__
    self.spider_loader = _get_spider_loader(settings)
  File "d:\pycharmprojects\commercialcrawler\venv\lib\site-packages\scrapy\crawler.py", line 336, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
  File "d:\pycharmprojects\commercialcrawler\venv\lib\site-packages\scrapy\settings\__init__.py", line 372, in frozencopy
    copy = self.copy()
  File "d:\pycharmprojects\commercialcrawler\venv\lib\site-packages\scrapy\settings\__init__.py", line 354, in copy
    return copy.deepcopy(self)
  ...
  File "D:\Python3.5\lib\copy.py", line 166, in deepcopy
    y = copier(memo)
TypeError: cannot deepcopy this pattern object

查閱網上發現並沒有出現和我類似的錯誤,於是從頭開始檢查traceback。
發現scrapy.crawler._get_spider_loader()裏調用了settings.frozencopy()方法,
找到frozencopy()

import copy

def frozencopy(self):
	"""
	Return an immutable copy of the current settings.
	
	Alias for a :meth:`~freeze` call in the object returned by :meth:`copy`.
	"""
	copy = self.copy()
	copy.freeze()
	return copy
	
def copy(self):
	"""
	Make a deep copy of current settings.
	
	This method returns a new instance of the :class:`Settings` class,
	populated with the same values and their priorities.
	
	Modifications to the new object won't be reflected on the original
	settings.
	"""
	return copy.deepcopy(self)
	
def freeze(self):
	"""
	Disable further changes to the current settings.
	
	After calling this method, the present state of the settings will become
	immutable. Trying to change values through the :meth:`~set` method and
	its variants won't be possible and will be alerted.
	"""
	self.frozen = True

可見此方法‘’凍結‘’了現有的settings,使用了copy.deepcopy()深拷貝。
此時再在網上搜索深拷貝類似的錯誤,並且檢查settings.py裏面是否有無法深拷貝的對象。
發現我在settings.py裏面調用了configparser,並且以變量名CONFIG實例化了一個ConfigParser對象

import configparser
CONFIG= configparser.ConfigParser()

由於scrapy 只會讀取變量名爲大寫的變量,將CONFIG更名爲config

Problem solved!

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章