爬蟲學習_07小實例

1. 查看網頁源代碼定位要爬取的信息（谷歌瀏覽器）

1. 目標網頁：https://blog.csdn.net/nanhuaibeian/article/details/85521071

2. 審查網頁元素，確定爬取內容
   目標想要爬取這一個列表信息
   

3. 確定網頁在div標籤下，class屬性爲‘table-box’
   

4. Console控制檯下得到這個table
   

2.爬取下網頁並保存

1. 代碼實現

import requests
url = 'https://blog.csdn.net/nanhuaibeian/article/details/85521071'
response = requests.get(url)
response.encoding('utf-8')
with open('real_case.html','w') as f:
    f.write(response.text)

2. 由於已經確定獲取目標內容的標籤爲div(class=‘table-box’)，往下進行，會發現，爬取下來的網頁內容不含有這一標籤，審查元素髮現這一標籤是通過js設置的
   修改方法從谷歌瀏覽器將源碼下載保存到real_case.html中。

3. 獲取目標內容

import requests
import re
from lxml import etree
import html

with open('real_case.html','r',encoding='utf-8') as f:
    c = f.read()
#將換行符全部替換掉
s = re.sub(r'\n',' ',c)
tree = etree.HTML(c)
table_element = tree.xpath("//div[@class='table-box'][1]/table/tbody/tr")
# 正則表達式替換掉'<>'
pattern1_attrib = re.compile(r"<.*?>")
#讀取內容
for row in table_element:
    try:
        td1 = row.xpath('td')[0]
        s1 = etree.tostring(td1).decode('utf-8')
        #將'<>'用空替換掉
        s1 = pattern1_attrib.sub('',s1)
        #按照HTML5的定義進行轉碼
        s1 = html.unescape(s1)

        td2 = row.xpath('td')[1]
        s2 = etree.tostring(td2).decode('utf-8')
        s2 = pattern1_attrib.sub('', s2)
        s2 = html.unescape(s2)
        print(s1,':',s2)

    except Exception as err:
        print('error:',err)
        pass