https://www.cnblogs.com/yunlongaimeng/p/11526466.html
1. 在項目文件夾中新建一個commands文件夾
2. 在command的文件夾中新建一個文件 crawlall.py
3.在crawlall.py 中寫一個command類,該類繼承 scrapy.commands
from scrapy.commands import ScrapyCommand
class Command(ScrapyCommand):
requires_project = True
def syntax(self):
return '[options]'
def short_desc(self):
return 'Runs all of the spiders 運行多個爬蟲文件'
def run(self, args, opts):
spider_list = self.crawler_process.spiders.list()
print('*'*100)
print(spider_list)
print('*'*100)
for name in spider_list:
self.crawler_process.crawl(name, **opts.__dict__)
self.crawler_process.start()
到這裏還沒完,settings.py配置文件還需要加一條。
COMMANDS_MODULE = ‘項目名稱.目錄名稱’
COMMANDS_MODULE = 'news_spider.commands'
命令行執行:啓動所有爬蟲 scrapy crawlall