【學習-scrapy】下載器中間件，ip代理設置

1、process_request(self.request,spider)

  參數：

         request、spider: 發送請求對象

  返回值： 

        返回none:

       返回Response:

2、process_response(self.request,spider)

  參數：

         request、response、sider: 發送請求對象

  返回值： 

        返回none:

       返回Response:

【例】隨機請求頭中間件：

def parse(self, response):
    print(response.text)

獲取ip: http://httpbin.org/ip

》setting.py=修改

 

DOWNLOADER_MIDDLEWARES = {
   'baom.middlewares.UserAgen': 543,
}

》middlewares.py=中間件

import random

class UserAgen(object):
    User_Agens=[
        'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.6776.400 QQBrowser/10.3.2577.400'
        'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.6776.400 QQBrowser/10.3.2577.400'
        'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.6776.400 QQBrowser/10.3.2577.400'

    ]
    #隨機請求一個
    def process_request(self,request,spider):
        user_agent=random.choice(self.User_Agens)
        request.headers['User-Agent']=user_agent