今天來爬取拉鉤網站來做實戰練習
如果按照之前的直接使用的urlopen直接來爬取數據，可能會被網站的反爬蟲機制給識別，如下代碼
from urllib import request
resp=request.urlopen('https://www.lagou.com/jobs/positionAjax.json?city=%E6%9D%AD%E5%B7%9E&needAddtionalResult=false')
print(resp.read())
結果如下：
這是因爲被檢測到時爬蟲程序，會反饋出的一些的錯誤信息。
這就需要加上請求頭以及請求數據等內容進行爬蟲
# from urllib import request
#
# resp=request.urlopen('https://www.lagou.com/jobs/positionAjax.json?city=%E6%9D%AD%E5%B7%9E&needAddtionalResult=false')
# print(resp.read())
from urllib import request,parse
url='https://www.lagou.com/jobs/positionAjax.json?city=%E6%9D%AD%E5%B7%9E&needAddtionalResult=false'
headers = {
'Accept':'application/json, text/javascript, */*; q=0.01',
'Referer':'https://www.lagou.com/jobs/list_python?labelWords=&fromSearch=true&suginput=',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 UBrowser/6.2.4098.3 Safari/537.36',
'Cookie':'user_trace_token=20190823091901-fd5287bd-c543-11e9-a504-5254005c3644; LGUID=20190823091901-fd528c54-c543-11e9-a504-5254005c3644; LG_LOGIN_USER_ID=e045585e162dc778c44ed0a28d10681ab56e6e164005627e; LG_HAS_LOGIN=1; index_location_city=%E6%9D%AD%E5%B7%9E; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%2216dbf0f0c381-064661d79b3e71-46531b29-1440000-16dbf0f0c39142%22%2C%22%24device_id%22%3A%2216dbf0f0c381-064661d79b3e71-46531b29-1440000-16dbf0f0c39142%22%7D; PRE_UTM=; PRE_HOST=; PRE_SITE=https%3A%2F%2Fwww.lagou.com%2Fhangzhou; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist%5Fpython%3FlabelWords%3D%26fromSearch%3Dtrue%26suginput%3D; _gid=GA1.2.2135446708.1585361539; _gat=1; _ga=GA1.2.1509779664.1566523142; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1585380500; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1585380500; X_HTTP_TOKEN=3277d2c74afe2e2c70508358516e9efc593003644e; LGSID=20200328152815-0eb6fadf-626e-43c1-8476-5e666830effe; LGRID=20200328152828-fc6dc1b3-201a-40ff-823a-8d235e266219'
}
data={
'first':"true",
'pn':1,
'kd':"python"
}
req = request.Request(url,headers=headers,data=parse.urlencode(data).encode('utf-8'),method='POST')
resp = request.urlopen(req)
print(resp.read().decode('utf-8'))
結果放在json在線解析看一下：
在這裏補充一下：如果訪問錯誤之後可以嘗試加一下請求頭中的數據等，然後進行爬蟲，每個網站的反爬蟲的機制可能不一樣。