環境

win 10 + pycharm + python 3.6 + scrapy 3.2.3

Scrapy安裝

pip install scrapy

Scrapy抓取步驟

第一步：新建項目

第二步：明確目標

第三步：製作爬蟲

第四步：存儲內容

第一步：新建項目

創建scrapy項目

scrapy startproject douban

設置settings.py

settings.py：定義項目的全局設置。

爬蟲協議設置：

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

默認 ROBOTSTXT_OBEY = True ，即遵守此協議；當爬取內容不符合該協議且仍要爬取時，設置 ROBOTSTXT_OBEY = False
（引自：https://www.jianshu.com/p/19c1ea0d59c2）

抓取速度設置：

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
DOWNLOAD_DELAY = 0.5
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

創建爬蟲文件（douban_spider.py）

此文件用於編寫爬蟲的xpath和正則表達式。

控制檯中，進入項目douban/spiders，輸入：

scrapy genspider douban_spider movie.douban.com

爬蟲名稱：douban_spider（不能與項目名重複）

所爬域名：movie.douban.com

第二步：明確目標

此步驟主要對所爬網站及所爬數據進行分析，並定義數據結構。

打開網站

如：https://movie.douban.com/top250

分析抓取內容

我所需要的內容包括序號、電影名字、電影介紹、評分、描述等6個部分。

以xpath工具分析獲取節點路徑：

    # 序號
//div[@class='article']//ol[@class='grid_view']//li//div[@class='item']//div[@class='pic']//em/text()
    # 電影名稱    
//div[@class='article']//ol[@class='grid_view']//li//div[@class='item']//div[@class='info']//div[@class='hd']//a//span[@class='title'][1]/text()
    # 電影介紹    
//div[@class='article']//ol[@class='grid_view']//li//div[@class='item']//div[@class='info']//div[@class='bd']//p[1]/text()
    # 星級    
//div[@class='article']//ol[@class='grid_view']//li//div[@class='item']//div[@class='info']//div[@class='bd']//div[@class='star']//span[@class='rating_num']/text()
    # 電影評論數    
//div[@class='article']//ol[@class='grid_view']//li//div[@class='item']//div[@class='info']//div[@class='bd']//div[@class='star']//span[4]/text()
    # 電影描述    
//div[@class='article']//ol[@class='grid_view']//li//div[@class='item']//div[@class='info']//div[@class='bd']//p[@class='quote']//span[@class='inq']/text()
    # 下一頁
//div[@class='article']//ol[@class='grid_view']//li//span[@class='next']/link/@href

實現數據結構（items.py）

import scrapy

class DoubanItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    # 序號
    serial_number = scrapy.Field()
    # 電影名稱
    movie_name = scrapy.Field()
    # 電影介紹
    introduce = scrapy.Field()
    # 星級
    star = scrapy.Field()
    # 電影評論數
    evaluate = scrapy.Field()
    # 電影描述
    describe = scrapy.Field()

第三步：製作爬蟲

測試

測試是否正常請求回數據（douban_spider.py）：

# -*- coding: utf-8 -*-
import scrapy

class DoubanSpiderSpider(scrapy.Spider):
    # 爬蟲的名字（不能與項目名字相同）
    name = 'douban_spider'
    # 允許的域名
    allowed_domains = ['movie.douban.com']
    # 入口url
    start_urls = ['http://movie.douban.com/top250']
    
    def parse(self, response):
        print(response.text)

查詢並設置user-agent（settings.py）（默認設置需要修改）：

# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'

控制檯運行：

scrapy crawl douban_spider

設置啓動文件main.py（在pycharm直接輸出，方便運行）。在項目douban目錄新建.py，命名爲main.py：

from scrapy import cmdline

# 運行
cmdline.execute('scrapy crawl douban_spider'.split())

編寫解析文件（douban_spider.py的parse()方法）

具體步驟：

通過xpath解析目標節點；
xpath規則抽取需要的內容（不要忘記打點(.//)）；
yield到douban_item中去翻頁；
解析“下一頁” 實現自動翻頁。

# 默認的解析方法
    def parse(self, response):
        # 查看是否請求回數據
        # print(response.text)
        # 循環電影的條目
        movie_list = response.xpath("//div[@class='article']//ol[@class='grid_view']//li")
        for i_item in movie_list:
            # item文件導進來
            douban_item = DoubanItem()
            # 寫詳細的xpath，進行數據的解析
            douban_item['serial_number'] = i_item.xpath(".//div[@class='item']//div[@class='pic']//em/text()").extract_first()
            douban_item['movie_name'] = i_item.xpath(".//div[@class='item']//div[@class='info']//div[@class='hd']//a//span[@class='title'][1]/text()").extract_first()
            content = i_item.xpath(".//div[@class='item']//div[@class='info']//div[@class='bd']//p[1]/text()").extract()
            # 數據的處理
            for i_content in content:
                content_s = "".join(i_content.split())
                douban_item['introduce'] = content_s
            # print(douban_item)
            douban_item['star'] = i_item.xpath(".//div[@class='item']//div[@class='info']//div[@class='bd']//div[@class='star']//span[@class='rating_num']/text()").extract_first()
            douban_item['evaluate'] = i_item.xpath(".//div[@class='item']//div[@class='info']//div[@class='bd']//div[@class='star']//span[4]/text()").extract_first()
            douban_item['describe'] = i_item.xpath(".//div[@class='item']//div[@class='info']//div[@class='bd']//p[@class='quote']//span[@class='inq']/text()").extract_first()
            # 你需要講數據yield到pipelines裏面去
            yield  douban_item
        # 解析下一頁規則，取後一頁的xpath
        next_link = response.xpath(".//span[@class='next']/link/@href").extract()
        if next_link:
            next_link = next_link[0]
            yield scrapy.Request("https://movie.douban.com/top250" + next_link, callback=self.parse)

第四步：保存數據

存到文件

存爲json：

scrapy crawl douban_spider -o test.json

存爲csv：

scrapy crawl douban_spider -o test.csv

如有亂碼：

Notepad++打開csv文件，編碼選擇utf-8 bom，保存。

main.py實現方式：

from scrapy import cmdline

# 運行
# cmdline.execute('scrapy crawl douban_spider'.split())

# 數據導出
cmdline.execute('scrapy crawl douban_spider -o test.json'.split())
cmdline.execute('scrapy crawl douban_spider -o test.csv'.split())

存到數據庫

如Mongodb、mysql等。

需要開啓item_piplines（settings.py裏）：

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'douban.pipelines.DoubanPipeline': 300,
}

其他部分：爬蟲的僞裝

Ip代理中間件編寫（middlewares.py）

引自：https://www.imooc.com/learn/1017

以阿布雲代理爲例：

request.meta['proxy'] = '代理ip:端口'
proxy_name_pass = b'用戶名:密碼'

如：

用戶名、密碼分號分隔：

agent = random.choice(USER_AGENT_LIST)

使用base64加密：

proxy_name_pass = b'用戶名:密碼'
encode_pass_name = base64.b64encode(proxy_name_pass)

如：

設置ip頭（Basic後有空格）：

requst.headers['Proxy-Authorization'] = 'Basic ' + encode_pass_name.decode()

中間件設置到settings.py裏（優先級保持默認即可，數字越小優先級越高）：

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
    # 'douban.middlewares.DoubanDownloaderMiddleware': 543,
    'douban.middlewares.my_proxy': 543,
}

運行（有my_proxy即代表運行代理成功，有效地隱藏了ip）

代碼：

class my_proxy(object):
    def process_request(self, request, spider):
        request.meta['proxy'] = '代理ip:端口'
        proxy_name_pass = b'用戶名:密碼'
        encode_pass_name = base64.b64encode(proxy_name_pass)
        requst.headers['Proxy-Authorization'] = 'Basic ' + encode_pass_name.decode()

user-agent中間件編寫（middlewares.py）

常見user-agent：（引自：https://blog.csdn.net/weixin_42144379/article/details/85639397）

# 各種PC端
        user_agent_list_2 = [
            # Opera
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 OPR/26.0.1656.60",
            "Opera/8.0 (Windows NT 5.1; U; en)",
            "Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.8.1) Gecko/20061208 Firefox/2.0.0 Opera 9.50",
            "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 9.50",
            # Firefox
            "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0",
            "Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10",
            # Safari
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2",
            # chrome
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36",
            "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11",
            "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.133 Safari/534.16",
            # 360
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36",
            "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko",
            # 淘寶瀏覽器
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11",
            # 獵豹瀏覽器
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER",
            "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)",
            "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E; LBBROWSER)",
            # QQ瀏覽器
            "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; QQBrowser/7.0.3698.400)",
            "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",
            # sogou瀏覽器
            "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0",
            "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; SE 2.X MetaSr 1.0)",
            # maxthon瀏覽器
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Maxthon/4.4.3.4000 Chrome/30.0.1599.101 Safari/537.36",
            # UC瀏覽器
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 UBrowser/4.0.3214.0 Safari/537.36",
        ]
        # 各種移動端
        user_agent_list_3 = [
            # IPhone
            "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5",
            # IPod
            "Mozilla/5.0 (iPod; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5",
            # IPAD
            "Mozilla/5.0 (iPad; U; CPU OS 4_2_1 like Mac OS X; zh-cn) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5",
            "Mozilla/5.0 (iPad; U; CPU OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5",
            # Android
            "Mozilla/5.0 (Linux; U; Android 2.2.1; zh-cn; HTC_Wildfire_A3333 Build/FRG83D) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            "Mozilla/5.0 (Linux; U; Android 2.3.7; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            # QQ瀏覽器 Android版本
            "MQQBrowser/26 Mozilla/5.0 (Linux; U; Android 2.3.7; zh-cn; MB200 Build/GRJ22; CyanogenMod-7) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            # Android Opera Mobile
            "Opera/9.80 (Android 2.3.4; Linux; Opera Mobi/build-1107180945; U; en-GB) Presto/2.8.149 Version/11.10",
            # Android Pad Moto Xoom
            "Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13",
            # BlackBerry
            "Mozilla/5.0 (BlackBerry; U; BlackBerry 9800; en) AppleWebKit/534.1+ (KHTML, like Gecko) Version/6.0.0.337 Mobile Safari/534.1+",
            # WebOS HP Touchpad
            "Mozilla/5.0 (hp-tablet; Linux; hpwOS/3.0.0; U; en-US) AppleWebKit/534.6 (KHTML, like Gecko) wOSBrowser/233.70 Safari/534.6 TouchPad/1.0",
            # Nokia N97
            "Mozilla/5.0 (SymbianOS/9.4; Series60/5.0 NokiaN97-1/20.0.019; Profile/MIDP-2.1 Configuration/CLDC-1.1) AppleWebKit/525 (KHTML, like Gecko) BrowserNG/7.1.18124",
            # Windows Phone Mango
            "Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0; HTC; Titan)",
            # UC瀏覽器
            "UCWEB7.0.2.37/28/999",
            "NOKIA5700/ UCWEB7.0.2.37/28/999",
            # UCOpenwave
            "Openwave/ UCWEB7.0.2.37/28/999",
            # UC Opera
            "Mozilla/4.0 (compatible; MSIE 6.0; ) Opera/UCWEB7.0.2.37/28/999"
        ]
        # 一部分 PC端的
        user_agent_list_1 = [
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
            "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
            "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
            "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
            "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
            "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
            "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
            "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
            "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
        ]

USER_AGENT_LIST：

class my_useragent(object):
    def process_request(self, request, spider):
        USER_AGENT_LIST = [
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
            "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
            "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
            "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
            "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
            "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
            "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
            "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
            "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
        ]

設置隨機：

agent = random.choice(USER_AGENT_LIST)

設置http頭：

request.headers['User-Agent'] = agent

settings.py設置（與其他中間件的優先級不能相同）：

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
    
    'douban.middlewares.my_proxy': 543,
    'douban.middlewares.my_useragent': 544,
}

注意事項

中間件定義完要在settings.py中啓用；
爬蟲項目不能與爬蟲文件名稱相同，spiders目錄內不能存在相同爬蟲名稱的項目文件；
文明守法，尊重公民隱私，不違反法律法規。

參考資料

教程及內容：https://www.imooc.com/learn/1017

settings部分內容：https://www.jianshu.com/p/19c1ea0d59c2

user-agent部分：https://blog.csdn.net/weixin_42144379/article/details/85639397

scrapy框架爬蟲初體驗——豆瓣評分top250

環境

Scrapy安裝

Scrapy抓取步驟

第一步：新建項目

創建scrapy項目

設置settings.py

創建爬蟲文件（douban_spider.py）

第二步：明確目標

打開網站

分析抓取內容

實現數據結構（items.py）

第三步：製作爬蟲

測試

編寫解析文件（douban_spider.py的parse()方法）

第四步：保存數據

存到文件

存到數據庫

其他部分：爬蟲的僞裝

Ip代理中間件編寫（middlewares.py）

user-agent中間件編寫（middlewares.py）

注意事項

參考資料

druid數據源 xml配置

筆記-Python GDAL解決TIFF去除黑邊問題

筆記-python pip國內鏡像源及配置

TensorFlow學習筆記（一）之邏輯斯地迴歸模型及Cifar-10分類的實現

PyQt5 Demo 簡單計算器

小菜升級筆記之手擼正則表達式——Python & JavaScript

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結