scrapyd官方文檔,scrapyd-client官方文檔,spiderkeeper官方文檔
Scrapyd是一個服務,用來運行scrapy爬蟲;scrapyd-client提供scrapyd-deploy工具,方便部署scrapy項目到scrapyd中。spiderkeeper可視化管理工具。
C:\Windows\system32>pip install scprayd scrapyd-client spiderkeeper
window環境需要在C:\Program Files\Python36\Scripts目錄下添加scrapyd-deploy.bat文件:
@echo off
"C:\Program Files\Python36\python.exe" "C:\Program Files\Python36\Scripts\scrapyd-deploy" %*
此時scrapyd-deploy命令纔可用。
新建一個目錄,然後啓動scrapyd服務:
D:\ksy\scrapyd>scrpyd
訪問http://localhost:6800,可得:
啓動spiderkeeper服務:
D:\ksy\scrapyd>spiderkeeper
SpiderKeeper startd on 0.0.0.0:5000 username:admin/password:admin with scrapyd servers:http://localhost:6800
訪問http://localhost:5000,默認賬號和密碼爲admin
- 點擊Create Project按鈕創建一個項目,
- 進入scrapy項目的根目錄D:\ksy\proxy,修改scrapy.cfg
-
[deploy] url = http://localhost:6800/ project = proxy
然後執行命令:
D:\ksy\proxy>scrapyd-deploy
Server response (200):
{"node_name": "zhiguo-PC", "status": "ok", "project": "proxy", "version": "1568792829", "spiders": 14}
運行一個scrapy項目: