SpiderKeeper 添加 Stats 鏈接過濾 log 最新信息

0.參考

https://github.com/DormyMo/SpiderKeeper

 

1.Job Dashboard 頁面添加 Stats 鏈接

python3.6/site-packages/SpiderKeeper/app/templates/job_dashboard.html

搜索 /log 定位

1.1 添加 Stats 表格列

1.2 添加 Stats 鏈接

 

2. 頁面效果

 

3. 過濾最新 log 信息

python3.6/site-packages/SpiderKeeper/app/spider/controller.py

本質上是通過 requests 請求 scrapyd 的 log 頁面，再重新排版，注意 escape

搜索 /log 定位

 

3.1 添加 python 代碼

import re
from html import escape
@app.route("/project/<project_id>/jobexecs/<job_exec_id>/stats")
def job_stats(project_id, job_exec_id):
    job_execution = JobExecution.query.filter_by(project_id=project_id, id=job_exec_id).first()
    res = requests.get(agent.log_url(job_execution))
    res.encoding = 'utf-8'
    #return res.text
    m = re.findall(r'\n.*?Redirecting\s+\(.*', res.text)
    a = m[-1] if m else ''
    m = re.findall(r'\n.*?Crawled\s+\(.*', res.text)
    b = m[-1] if m else ''
    m = re.findall(r'\n.*?Scraped\s+from\s+<.*', res.text)
    c = m[-1] if m else ''
    m = re.findall(r'\n.*?{.*?}', res.text)
    d = m[-1] if m else ''
    m = re.findall(r'\n.*?Crawled\s+\d+\s+pages\s+\(.*', res.text)
    e = m[-1] if m else ''
    return '<br>'.join([escape(i) for i in [a, b, c, d, e]])

 

4. log 過濾結果