前菜
之前跟大家分享了許多爬蟲的經驗，當然很多我也是邊學習邊覆盤這些掌握的知識，基本上來說我們已經可以上至九天攬月，下入五洋捉鱉
咳咳，事實上我們已經試過爬取的形式有文本、圖片、音頻、視頻這些，而且基本上都是從網頁上獲取的，那麼今天會帶大家來看看如何從App來獲取數據！
湯
工具：Fiddler、Android手機
環境：Python3、Scrapy
主菜
這裏我們主要通過Fiddler對手機進行抓包來分析（當然有的人可能用的Charles，這裏就不同時介紹了，因爲方法很多，但思路是相似的），正常情況下是無法抓到手機上的數據的，所以在這之前我們需要配置一下
安裝Fiddler，這個網上也可以下，或者在公衆號回覆【pye】裏面也有
對Fiddler配置，打開Fiddler, Tools-> Fiddler Options (配置完後記得要重啓Fiddler)，選中"Decrpt HTTPS traffic", Fiddler就可以截獲HTTPS請求，選中"Allow remote computers to connect"，是允許別的機器把HTTP/HTTPS請求發送到Fiddler上來，這裏注意下面的端口號 8888
- 再次注意重啓Fiddler，接着打開命令行窗口，輸入
ipconfig查看本機的IP，這裏我就不展示出來了，此時我們已經知道了IP以及端口號
- 接着打開手機，進入WLAN設置，修改已連接的網絡（確保一下手機和PC是連接在同一個局域網中），設置代理爲手動，填入本機的IP於服務器主機名，服務器端口爲8888，然後保存即可
- 接着我們要下載安全證書，使用Android手機的瀏覽器打開：http://本機IP:8888， 點"FiddlerRoot certificate" 然後下載證書
- 之後點擊下載的證書可以命名證書便於區別（額...你可能覺得...沒什麼意義）
- 到這裏我們的配置就結束了，接着打開Fiddler來看看，可以點擊你手機上的一個App，就會看到Fiddler已經抓到了數據包
這裏我首先試了下打開Boss直聘，搜索Python爬蟲崗位的信息，看看抓到的包中的數據之怎樣的，結果
可以看到雖然抓到了包，但是其中的數據是加密了的，沒法獲取對我們有用的信息，然後看了幾個其他的url也是加密的，於是換了一個App，這次打開的是兼職貓
然後我在全部兼職中，給定範圍，臨時工，地點徐匯
那麼我們來看看Fiddler是否抓到了這些兼職的數據
答案是肯定的，事實上你只要點擊App的某個按鈕，Fiddler都會抓到數據包，你也可以嘗試下點擊“我的”、“簡歷”這些就可以看到對應的操作在Fiddler中抓到的包，可能看的不太清晰，我們把這個鏈接拿出來放到瀏覽器上看能不能看到數據
首先通過右側的Headers，我們可以知道這是一個Post請求，但是當我們在瀏覽器直接打開這個鏈接，也同樣能看到我們想看的數據
圖上所顯示的就是對應的第一條兼職的信息，來點進去看看職位詳情
同樣這時候我們看看Fiddler最新抓到的
額...是不是看不太清...但是這就是對應的包，我發四！
我們來總結一下App抓包的流程：
1、首先需要配置Fiddler以及下載安全證書到你的Android手機
2、接着打開你想要獲取數據的App，隨意滑動、點擊，Fiddler就能抓到發送的請求
3、再從這些剛剛獲取的請求中篩選，就能找到所對應的數據
4、當然一般都會進行加密，這時候你要再想獲取數據可能就得換種方式，從網頁等入手
5、正常的話就可以拿到這個請求對應的鏈接，這時候再通過代碼根據對應的請求方式，測試需要帶上的請求參數，最後發起請求，就可以獲得我們想要的數據
甜點
上面已經講述了App抓包的流程，而這裏我們來看一部分代碼的演示，當然用的是Scrapy這個爬蟲框架，雖然我平時喜歡寫爬蟲直接上，相當於一個腳本，這樣對我來說直接簡單，但是提高速度什麼的也就需要自己開多進程、協程等
而用Scrapy就很好的解決了速度問題，其內部是基於Twisted框架，所以是以異步的方式發送請求，但是還需要你根據自己的要求改動某些參數
當然這也是你要做爲一名蟲師所必須要掌握的，所以之後的案例，代碼我會盡量克服自己，多用Scrapy寫，當然除了一些小東西，畢竟殺雞焉用牛刀
首先是創建一個Scrapy項目，可以在命令行中運行如下指令
python3 -m scrapy startproject projectname
cd projectname
python3 -m scrapy genspider spidername "爬取網站的域名，like : xxx.com"
這樣就創建了一個Scrapy項目，至於我爲什麼要加上 python3 -m 是因爲我的機子同時裝了Python2跟3，爲了區別調用，若你只有Python3，可以不用加
這是Scrapy項目的基本結構
接着是基本的配置，一般都要在settings.py中令
ROBOTSTXT_OBEY = False # 默認爲True
這是不遵守網站的機器人協議，一般搜索引擎在訪問一個網站的資源時都會首先訪問這個協議，協議中規定了搜索引擎只能訪問哪些內容，而我們的爬蟲一般不會去遵守，你可以試試在網站的主網址下跟上/robots.txt，就像 ： https://www.bilibili.com/robots.txt 跑偏了跑偏了......
還需要修改開啓這裏的User-Agent，改爲手機的請求頭
USER_AGENT = 'Dalvik/2.1.0 (Linux; U; Android 7.0; KNT-AL10 Build/HUAWEIKNT-AL10)'
其他暫時可以不用動，接下來是 jzm.py 部分
# -*- coding: utf-8 -*-
import scrapy
from scrapy import Request
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
class JzmSpider(scrapy.Spider):
name = 'jzm' # 爬蟲名稱
allowed_domains = ['jianzhimao.com'] # 爬蟲允許的域名範圍
# 重寫了start_url 的內容，yield回調parse來解析返回的響應
def start_requests(self):
# 這是兼職貓，徐匯臨時工兼職的數據的URL，從Fiddler抓包獲得
start_url = 'http://api.yh.jianzhimao.com/v35http/job/pageNewJob?order=10&token=f4b68d09a22bf4ee14f96623673b4a54&city=%E4%B8%8A%E6%B5%B7&mc=69DDU16519018674&system=1×tamp=1548567219385&versions=5.8.1&request_tag=e6fc923518c9c5b313eddc45beaa2ab5&userid=13821503&lng=121.611041&usedRecommend_data=&area=%E5%BE%90%E6%B1%87&statistics_data=%7B%22userid%22%3A%2213821503%22%2C%22sys_info_sign%22%3A%222fdfc484a2b33b26874ac15e62246a66%22%2C%22app_version%22%3A%225.8.1%22%7D&sm=f694185c7ecbfffda31e734e91fc0dd4&channel=huawei&type=%E4%B8%B4%E6%97%B6%E5%B7%A5&page=1&lat=31.149018'
yield Request(url=start_url, callback=self.parse)
def parse(self, response):
# 打印返回的結果
print(response.text)
if __name__ == "__main__":
# 這裏這樣做的好處是不用通過命令行 crawl 來運行Scrapy項目，可以直接右鍵Run
process = CrawlerProcess(get_project_settings())
# 參數爲爬蟲的名稱
process.crawl('jzm')
process.start()
結果，注意右滑
C:\Python3\python3.exe D:/Python/PycharmProject/Work/Jzmao/Jzmao/spiders/jzm.py
2019-01-27 19:57:05 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: Jzmao)
2019-01-27 19:57:05 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.5, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.6.3 (v3.6.3:2c5fed8, Oct 3 2017, 17:26:49) [MSC v.1900 32 bit (Intel)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0h 27 Mar 2018), cryptography 2.2.2, Platform Windows-8.1-6.3.9600-SP0
2019-01-27 19:57:05 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'Jzmao', 'NEWSPIDER_MODULE': 'Jzmao.spiders', 'SPIDER_MODULES': ['Jzmao.spiders'], 'USER_AGENT': 'Dalvik/2.1.0 (Linux; U; Android 7.0; KNT-AL10 Build/HUAWEIKNT-AL10)'}
2019-01-27 19:57:05 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2019-01-27 19:57:15 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-01-27 19:57:15 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-01-27 19:57:15 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2019-01-27 19:57:15 [scrapy.core.engine] INFO: Spider opened
2019-01-27 19:57:15 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-01-27 19:57:15 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2019-01-27 19:57:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://api.yh.jianzhimao.com/v35http/job/pageNewJob?order=10&token=f4b68d09a22bf4ee14f96623673b4a54&city=%E4%B8%8A%E6%B5%B7&mc=69DDU16519018674&system=1×tamp=1548567219385&versions=5.8.1&request_tag=e6fc923518c9c5b313eddc45beaa2ab5&userid=13821503&lng=121.611041&usedRecommend_data=&area=%E5%BE%90%E6%B1%87&statistics_data=%7B%22userid%22%3A%2213821503%22%2C%22sys_info_sign%22%3A%222fdfc484a2b33b26874ac15e62246a66%22%2C%22app_version%22%3A%225.8.1%22%7D&sm=f694185c7ecbfffda31e734e91fc0dd4&channel=huawei&type=%E4%B8%B4%E6%97%B6%E5%B7%A5&page=1&lat=31.149018> (referer: None)
{"total":56,"entity":{"feedFlowRecommend":[]},"status":0,"msg":"ok","list":[{"id":2970091,"jobType":0,"days":180,"deadline":1563465599,"sex":"不限","tab":"1","type":"臨時工","title":"愛建園3期連鎖超市日結小時工寒假工","city":"上海","area":"徐匯","longitude":"121.438984","latitude":"31.182868","salary":"250元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":804,"isshow":1,"mi":16785,"userId":10963508,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548574990,"work_time":1547913600,"createtime":"2019-01-20 10:14:32","open_addr":"上海市徐匯愛建園","clearing_cycle":"日結","miToString":"16785.0","tabList":[{"tab_name":"日結","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2979916,"jobType":0,"days":181,"deadline":1564070399,"sex":"不限","tab":"1","type":"臨時工","title":"徐家彙接聽客服240","city":"上海","area":"徐匯","longitude":"121.443481","latitude":"31.200208","salary":"240元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":166,"isshow":1,"mi":16914,"userId":10739109,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548569959,"work_time":1548432000,"createtime":"2019-01-26 13:39:01","open_addr":"上海市徐匯徐家彙地鐵附近","clearing_cycle":"月結","miToString":"16914.0","tabList":[{"tab_name":"客服","tab_color":"#866437"}],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2973256,"jobType":0,"days":38,"deadline":1551369599,"sex":"不限","tab":"1","type":"臨時工","title":"1號線徐家彙服務員丶洗碗180/天日結","city":"上海","area":"徐匯","longitude":"121.443476","latitude":"31.200206","salary":"18元/小時","welfare":"1,2,3,4,5,","isrecommended":0,"is_filter_ios":0,"click_times":344,"isshow":1,"mi":16914,"userId":13341593,"is_filter_home_page":1,"privilege_tag":0,"start_time":1548555683,"work_time":1548086400,"createtime":"2019-01-22 03:38:19","open_addr":"上海市徐匯徐家彙地鐵5號口","clearing_cycle":"日結","miToString":"16914.0","tabList":[{"tab_name":"日結","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2980235,"jobType":0,"days":6,"deadline":1548950399,"sex":"不限","tab":"2,3","type":"臨時工","title":"日結200/天","city":"上海","area":"徐匯","longitude":"121.446234","latitude":"31.16915","salary":"200元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":537,"isshow":1,"mi":15829,"userId":13523363,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548549860,"work_time":1548432000,"createtime":"2019-01-26 19:30:51","open_addr":"上海市徐匯","clearing_cycle":"完工結","miToString":"15829.0","tabList":[{"tab_name":"日結","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2963613,"jobType":0,"days":44,"deadline":1551369599,"sex":"不限","tab":"1","type":"臨時工","title":"1號線徐家彙洗碗工180/天日結","city":"上海","area":"徐匯","longitude":"121.443476","latitude":"31.200206","salary":"18元/小時","welfare":"1,2,3,4,5,","isrecommended":0,"is_filter_ios":0,"click_times":970,"isshow":1,"mi":16914,"userId":13341593,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548508604,"work_time":1547568000,"createtime":"2019-01-16 23:57:47","open_addr":"上海市徐匯徐家彙地鐵5號口","clearing_cycle":"日結","miToString":"16914.0","tabList":[{"tab_name":"日結","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2980217,"jobType":0,"days":3,"deadline":1548777599,"sex":"男","tab":"2,3","type":"臨時工","title":"順豐短派200一天","city":"上海","area":"徐匯","longitude":"121.446234","latitude":"31.16915","salary":"200元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":375,"isshow":1,"mi":15829,"userId":10189553,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548501268,"work_time":1548518400,"createtime":"2019-01-26 19:14:28","open_addr":"上海市徐匯","clearing_cycle":"完工結","miToString":"15829.0","tabList":[{"tab_name":"日結","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2976450,"jobType":0,"days":32,"deadline":1550937599,"sex":"不限","tab":"1","type":"臨時工","title":"溫泉管包吃住5000","city":"上海","area":"徐匯","longitude":"121.443476","latitude":"31.200206","salary":"5000元/月","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":379,"isshow":1,"mi":16914,"userId":10189553,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548383539,"work_time":1548172800,"createtime":"2019-01-23 22:07:19","open_addr":"上海市徐匯徐家彙","clearing_cycle":"完工結","miToString":"16914.0","tabList":[],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2976793,"jobType":0,"days":180,"deadline":1563811199,"sex":"不限","tab":"1","type":"臨時工","title":"兼職一手機日結一在家可做","city":"上海","area":"徐匯","longitude":"121.446234","latitude":"31.16915","salary":"300元/天","welfare":"1,2,3,4,5,","isrecommended":0,"is_filter_ios":0,"click_times":277,"isshow":1,"mi":15829,"userId":11286085,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548297528,"work_time":1548259200,"createtime":"2019-01-24 10:38:48","open_addr":"上海市徐匯加微信18170621123","clearing_cycle":"日結","miToString":"15829.0","tabList":[{"tab_name":"日結","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2908763,"jobType":0,"days":6,"deadline":1549641599,"sex":"不限","tab":"2,3","type":"臨時工","title":"管吃住160/天服務員上海就近分配","city":"上海","area":"徐匯","longitude":"121.446234","latitude":"31.16915","salary":"160元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":1518,"isshow":1,"mi":15829,"userId":6888856,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548236183,"work_time":1549123200,"createtime":"2018-12-28 17:30:11","open_addr":"上海市徐匯就近分配","clearing_cycle":"日結","miToString":"15829.0","tabList":[],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2974897,"jobType":0,"days":180,"deadline":1563638399,"sex":"不限","tab":"1","type":"臨時工","title":"裕德路連鎖超市高價日結小時工寒假工","city":"上海","area":"徐匯","longitude":"121.441868","latitude":"31.191101","salary":"250元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":366,"isshow":1,"mi":16749,"userId":10963508,"is_filter_home_page":1,"privilege_tag":0,"start_time":1548160028,"work_time":1548086400,"createtime":"2019-01-22 20:27:08","open_addr":"上海市徐匯裕德路58號","clearing_cycle":"日結","miToString":"16749.0","tabList":[{"tab_name":"寒假工","tab_color":"#4756ec"}],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2794265,"jobType":0,"days":141,"deadline":1554047999,"sex":"不限","tab":"1","type":"臨時工","title":"大型餐飲酒店每天要人日結18一個小時","city":"上海","area":"徐匯","longitude":"121.443125","latitude":"31.203978","salary":"180元/天","welfare":"1,2,3,4,5","isrecommended":0,"is_filter_ios":0,"click_times":13205,"isshow":1,"mi":17091,"userId":13808431,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548132357,"work_time":1541865600,"createtime":"2018-11-10 15:44:46","open_addr":"上海市上海徐匯匯銀廣場","clearing_cycle":"完工結","miToString":"17091.0","tabList":[{"tab_name":"日結","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"認證企業","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2946398,"jobType":0,"days":42,"deadline":1550678399,"sex":"不限","tab":"1","type":"臨時工","title":"宜芝多136-200/天寒假工","city":"上海","area":"徐匯","longitude":"121.446235","latitude":"31.169152","salary":"200元/天","welfare":"5","isrecommended":0,"is_filter_ios":0,"click_times":664,"isshow":1,"mi":15829,"userId":1006298,"is_filter_home_page":1,"privilege_tag":0,"start_time":1548062002,"work_time":1547049600,"createtime":"2019-01-10 19:33:00","open_addr":"上海市徐匯全上海各地鐵站就近分配","clearing_cycle":"月結","miToString":"15829.0","tabList":[{"tab_name":"寒假工","tab_color":"#4756ec"}],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2971579,"jobType":0,"days":39,"deadline":1551369599,"sex":"不限","tab":"1","type":"臨時工","title":"上海宜芝多門店就近安排","city":"上海","area":"徐匯","longitude":"121.446234","latitude":"31.16915","salary":"5000元/月","welfare":"5,","isrecommended":0,"is_filter_ios":0,"click_times":224,"isshow":1,"mi":15829,"userId":14353534,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548036331,"work_time":1548000000,"createtime":"2019-01-21 10:05:31","open_addr":"上海市徐匯","clearing_cycle":"月結","miToString":"15829.0","tabList":[],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2970422,"jobType":0,"days":26,"deadline":1550246399,"sex":"不限","tab":"1","type":"臨時工","title":"急招寒假工餐廳服務員","city":"上海","area":"徐匯","longitude":"121.487899","latitude":"31.249162","salary":"150元/天","welfare":"1,2","isrecommended":0,"is_filter_ios":0,"click_times":167,"isshow":1,"mi":16149,"userId":3549404,"is_filter_home_page":0,"privilege_tag":0,"start_time":1547961776,"work_time":1548000000,"createtime":"2019-01-20 13:22:56","open_addr":"上海市上海徐匯","clearing_cycle":"完工結","miToString":"16149.0","tabList":[{"tab_name":"寒假工","tab_color":"#4756ec"}],"vip_tab_list":[{"tab_name":"認證企業","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2970334,"jobType":0,"days":182,"deadline":1563638399,"sex":"不限","tab":"1","type":"臨時工","title":"兼職一手機日結一在家可做","city":"上海","area":"徐匯","longitude":"121.446235","latitude":"31.169152","salary":"300元/天","welfare":"1,2,3,4,5","isrecommended":0,"is_filter_ios":0,"click_times":759,"isshow":1,"mi":15829,"userId":11286085,"is_filter_home_page":0,"privilege_tag":0,"start_time":1547957457,"work_time":1547913600,"createtime":"2019-01-20 12:10:57","open_addr":"上海市上海徐匯加微信18170621123","clearing_cycle":"日結","miToString":"15829.0","tabList":[{"tab_name":"日結","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2960246,"jobType":0,"days":180,"deadline":1563119999,"sex":"不限","tab":"1","type":"臨時工","title":"徐彙區連鎖超市日結小時工","city":"上海","area":"徐匯","longitude":"121.441015","latitude":"31.191503","salary":"250元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":738,"isshow":1,"mi":16839,"userId":10963508,"is_filter_home_page":0,"privilege_tag":0,"start_time":1547864800,"work_time":1547568000,"createtime":"2019-01-16 01:04:01","open_addr":"上海市徐匯裕德路","clearing_cycle":"日結","miToString":"16839.0","tabList":[],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2948610,"jobType":0,"days":49,"deadline":1551369599,"sex":"不限","tab":"1","type":"臨時工","title":"711便利店寒假兼職","city":"上海","area":"徐匯","longitude":"121.443476","latitude":"31.200206","salary":"20元/小時","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":2222,"isshow":1,"mi":16914,"userId":10189553,"is_filter_home_page":0,"privilege_tag":0,"start_time":1547819343,"work_time":1547136000,"createtime":"2019-01-11 16:54:47","open_addr":"上海市徐匯徐家彙","clearing_cycle":"完工結","miToString":"16914.0","tabList":[{"tab_name":"可長期","tab_color":"#3a45eb"}],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2964534,"jobType":0,"days":0,"deadline":1550764799,"sex":"不限","tab":"2","type":"臨時工","title":"高薪百麗鞋子營業員就近分配","city":"上海","area":"徐匯","longitude":"121.427217","latitude":"31.191935","salary":"185元/天","welfare":"2,","isrecommended":0,"is_filter_ios":0,"click_times":216,"isshow":1,"mi":18115,"userId":10258217,"is_filter_home_page":0,"privilege_tag":0,"start_time":1547704536,"work_time":1550678400,"createtime":"2019-01-17 13:55:36","open_addr":"上海市徐匯柳州路928號","clearing_cycle":"完工結","miToString":"18115.0","tabList":[{"tab_name":"包住","tab_color":"#b71144"}],"vip_tab_list":[{"tab_name":"認證企業","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2906710,"jobType":0,"days":31,"deadline":1548950399,"sex":"男","tab":"1","type":"臨時工","title":"順豐短派兼職","city":"上海","area":"徐匯","longitude":"121.443476","latitude":"31.200206","salary":"180元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":2358,"isshow":1,"mi":16914,"userId":10189553,"is_filter_home_page":0,"privilege_tag":0,"start_time":1547651147,"work_time":1546272000,"createtime":"2018-12-28 07:21:18","open_addr":"上海市徐匯徐家彙","clearing_cycle":"完工結","miToString":"16914.0","tabList":[{"tab_name":"包住","tab_color":"#b71144"}],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2947376,"jobType":0,"days":39,"deadline":1550764799,"sex":"不限","tab":"1","type":"臨時工","title":"羅森便利店寒假工187/ 天","city":"上海","area":"徐匯","longitude":"121.455922","latitude":"31.195215","salary":"187元/天","welfare":"5","isrecommended":0,"is_filter_ios":0,"click_times":617,"isshow":1,"mi":15615,"userId":195218,"is_filter_home_page":0,"privilege_tag":0,"start_time":1547646860,"work_time":1547395200,"createtime":"2019-01-11 11:04:15","open_addr":"上海市徐匯羅森便利店就近分配","clearing_cycle":"月結","miToString":"15615.0","tabList":[],"vip_tab_list":[{"tab_name":"熱點招聘","tab_color":"#ff9c27"}],"is_nationwide":0}]}
2019-01-27 19:57:15 [scrapy.core.engine] INFO: Closing spider (finished)
2019-01-27 19:57:15 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 789,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 14370,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2019, 1, 27, 11, 57, 15, 591689),
'log_count/DEBUG': 2,
'log_count/INFO': 7,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2019, 1, 27, 11, 57, 15, 177682)}
2019-01-27 19:57:15 [scrapy.core.engine] INFO: Spider closed (finished)
Process finished with exit code 0
以上就是今天的內容，之後也會更多的來講解Scrapy，敬請期待
print('微信公衆號搜索 "猿獅的單身日常" ，Java技術升級、蟲師修煉，我們 不見不散!')
print('也可以掃下方二維碼哦~')
