[B10]爬蟲課程02

數據解析

1.Xpath語法和lxml模塊

#使用方式：使用//獲取整個頁面當中的元素，然後寫標籤名，然後再寫謂詞進行提取。

//div[@clas='abc']

需要注意的知識點：

1./和//的區別：/只獲取直接子節點，//可以獲取子孫節點
2.contains:有時候某個屬性包含多個值，可以使用cntains

//div[contains(@class,'job_detail')]

3.謂詞的下標是從1開始

使用lxml解析HTML代碼：

1.解析html字符串：使用’lxml.etree.HTML’進行

htmlElement = etree.HTML(text)
print(etree.tostring(htmlElement,encoding='utf-8').decode('utf-8'))

2.解析html文件：使用’lxml.etree.parse’進行，如果這個函數默認使用xml解析器，需要自己創建html解析器。

htmlElement = etree.parse('qingyunian.html')
print(etree.tostring(htmlElement,encoding='utf-8').decode('utf-8'))

實例

from lxml import etree

#解析慶餘年短評
def parse_qyn_file():
    parser = etree.HTMLParser(encoding='utf-8')
    htmlElement = etree.parse('qingyunian.html',parser=parser)
    print(etree.tostring(htmlElement,encoding='utf-8').decode('utf-8'))
 
#解析拉鉤網頁
def parse_lagou_file():
    parser = etree.HTMLParser(encoding='utf-8')
    htmlElement = etree.parse('lagou.html',parser=parser)
    print(etree.tostring(htmlElement,encoding='utf-8').decode('utf-8'))

if __name__ == '__main__':
    #parse_lagou_file()
    #parse_text()
    parse_qyn_file()

實例：解析騰訊招聘的網頁信息

from lxml import etree

parser = etree.HTMLParser(encoding='utf-8')
html = etree.parse("tencent.html",parser=parser)
#1.獲取所有的a標籤
#//a
#xpath返回的是一個列表
alis = html.xpath("//a[@class='recruit-list-link']")
for a in alis:
    print(etree.tostring(a,encoding='utf-8').decode("utf-8"))

#2.獲取所有崗位名稱
alis = html.xpath("//h4")
for a in alis:
    print(etree.tostring(a,encoding='utf-8').decode("utf-8"))

#3.獲取所有職位信息
alis = html.xpath("//p[@class='recruit-text']")
for a in alis:
    print(etree.tostring(a,encoding='utf-8').decode("utf-8"))

#4.獲取所有的純文本信息
alis = html.xpath("//a[@class='recruit-list-link']")
positions = []
for a in alis:
    #在某個標籤下，再執行xpath函數，獲取子孫元素，那麼應該在//前加一個點，代表在當前元素下獲取
    title = a.xpath(".//h4[@class='recruit-title']/text()")
    daihao = a.xpath("./p[1]//span[1]/text()")
    address = a.xpath("./p[1]//span[2]/text()")
    category = a.xpath("./p[1]//span[3]/text()")
    time = a.xpath("./p[1]//span[5]/text()")
    needs = a.xpath("./p[2]/text()")  
    
    position = {
        'title':title,
        'daihao':daihao,
        'address':address,
        'category':category,
        'time':time,
        'needs':needs
    }
    positions.append(position)

    print(positions)

#寫入Excel表格

#寫入Excel表格
import pandas as pd

datadf = pd.DataFrame(positions)
 
datadf.to_excel('result.xlsx',sheet_name='pachong_cc')
# 導出excel