python爬取豆瓣圖書top250榜單並存爲CSV文件

1、頁面分析

豆瓣圖書TOP250

以下是前四頁的網址：

https://book.douban.com/top250
https://book.douban.com/top250?start=25
https://book.douban.com/top250?start=50
https://book.douban.com/top250?start=75

發現把第一頁網址改爲 https://book.douban.com/top250?start=0 也能訪問

只需修改 start= 後面的數字即可構造出10頁的網址

需要爬取的信息有：書名，書本的URL地址，作者，出版社和出版日期，書本價格，評分和一句話評價。

2、分析網頁源代碼，獲取網頁結構

3、需要用到的庫

request用於請求網頁獲取網頁數據，lxml解析提取數據，csv存儲數據

import requests
from lxml import etree
import csv

4、源代碼

# 導入相應庫文件
import requests
from lxml import etree
import csv

# 創建CSV文件，並寫入表頭信息
fp = open('D:\Code\doubanbook2.csv','wt',newline='',encoding='utf-8')
writer = csv.writer(fp)
writer.writerow(('書名','地址','作者','出版社','出版日期','價格','評分','評價'))

# 構造所有的URL鏈接
urls = ['https://book.douban.com/top250?start={}'.format(str(i)) for i in range(0,251,25)]

# 添加請求頭
headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36'
}


# 循環URL
for url in urls:
    html = requests.get(url,headers=headers)
    selector = etree.HTML(html.text)
    # 取大標籤，以此循環
    infos = selector.xpath('//tr[@class="item"]')

    for info in infos:
        name = info.xpath('td/div/a/@title')[0]
        url = info.xpath('td/div/a/@href')[0]
        book_infos = info.xpath('td/p/text()')[0]
        author = book_infos.split('/')[0]
        publisher = book_infos.split('/')[-3]
        date = book_infos.split('/')[-2]
        price = book_infos.split('/')[-1]
        rate = info.xpath('td/div/span[2]/text()')[0]
        comments = info.xpath('td/p/span/text()')
        comment = comments[0] if len(comments) != 0 else "空"

        # 寫入數據
        writer.writerow((name,url,author,publisher,date,price,rate,comment))

# 關閉文件
fp.close()

5、結果