1、使用代理ip和普通requests.get請求抓取頁面流程一樣,只是多了個參數proxies.
- http://www.goubanjia.com/ 找代理IP,注意http,https,選與目標網址一樣的協議。
- proxies字典格式的
import requests
url='https://www.baidu.com/s?wd=ip&ie=utf-8'
proxies={
"https":"218.60.8.99:3129"
}
headers={
"User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
}
response=requests.get(url=url,proxies=proxies,headers=headers)
with open('baiduip.html','w',encoding='utf-8') as f:
f.write(response.text)
2、使用session抓取需要登陸之後才能看到的頁面數據
- 獲取session對象:
session=requests.session()
- session.post請求登陸url存儲session信息
- session.get請求獲取登陸之後的個人頁面
以豆瓣爲例:
import requests
# 先用session登陸獲取存儲session
session=requests.session()
login_url='https://accounts.douban.com/j/mobile/login/basic'
data={
'ck':'',
'name':'',
'password':'',
'remember':'false',
'ticket':''
}
headers={
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'}
login_response=session.post(url=login_url,data=data,headers=headers)
# 再用session獲取想要的頁面 https://www.douban.com/people/193627830/
url='https://www.douban.com/people/1230/'
response=session.get(url=url,headers=headers)
with open('doubanlogin.html','w',encoding='utf-8')as f:
f.write(response.text)