新手學習Python筆記（一）爬取網易雲音樂歌單熱評_爬蟲的簡單應用

新手學習Python筆記（一）爬蟲的簡單應用

免責聲明：本程序僅供個人學習使用

功能目標：方便摘抄網易雲音樂的熱評，寫一個爬蟲程序，通過輸入歌單號，爬取並保存歌單內歌曲的熱評。

記錄第一個自己完成的簡單爬蟲程序

展示截圖：（命令臺界面和保存文件界面）

順便安利一下這個日語歌單

下面是代碼部分：

庫：requests、bs4 、json、csv、time（可選）

import requests
from bs4 import BeautifulSoup
import json
from csv import writer
import time

t1=time.clock()
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36',
}

print("===========================\n獲取網易雲音樂歌單中所有歌曲熱評\n===========================")
id= input("請輸入歌單id：")
print("正在搜索。。。\n")

playlist_url = f'http://music.163.com/playlist?id={id}'

rs = requests.session()
r = rs.get(playlist_url, headers=headers)

soup = BeautifulSoup(r.content, 'lxml')

main = soup.find('ul', {'class': 'f-hide'})

name_list= []
id_list= []

for music in main.find_all('a'):

    music_id= music['href'][9:]
    music_name= music.text

    name_list.append(music_name)
    id_list.append(music_id)

total= len(name_list)
cout=1
allcout=0

file= open("網易雲音樂歌單熱評.csv","w",encoding="gb18030",newline="")
writer= writer(file)

for i in id_list:

    list=[cout,name_list[cout-1]]
    writer.writerow(list)
    get_url= "http://music.163.com/api/v1/resource/comments/R_SO_4_"+i+"?limit=0&offset=0"

    r= requests.get(get_url,headers=headers)

    json_dict=json.loads(r.content.decode("utf-8"))

    hotcomments= json_dict["hotComments"]

    allcout+=len(hotcomments)
    no=1

    for j in hotcomments:
        nickname= j["user"]["nickname"]
        content= j["content"].replace("\n"," ")
        liked= str(j["likedCount"])+"贊"
        list=["","",no,nickname,liked,content]
        writer.writerow(list)
        no+=1
    cout+=1
    writer.writerow('')
file.close()

t2= time.clock()-t1

print(f"共找到{total}首歌 {allcout}條熱評 用時{t2:.2f}s",end="\n===========================\n")
input("按任意鍵退出。。。")

代碼分析：

常量的定義、界面設計

#計時開始
t1=time.clock()
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36',
}

print("===========================\n獲取網易雲音樂歌單中所有歌曲熱評\n===========================")
id= input("請輸入歌單id：")
print("正在搜索。。。\n")

playlist_url = f'http://music.163.com/playlist?id={id}'#得到歌單的url

爬取歌曲信息

#得到歌單內的歌曲名稱和歌曲id
rs = requests.session()
r = rs.get(playlist_url, headers=headers)

soup = BeautifulSoup(r.content, 'lxml')

#可通過歌曲網頁的源碼得到有關信息
main = soup.find('ul', {'class': 'f-hide'})

name_list= []
id_list= []

for music in main.find_all('a'):

    music_id= music['href'][9:]
    music_name= music.text

    name_list.append(music_name)
    id_list.append(music_id)

#計數君
total= len(name_list)
cout=1
allcout=0

創建表格

#易錯點：encoding，newline應設爲""
file= open("網易雲音樂歌單熱評.csv","w",encoding="gb18030",newline="")
writer= writer(file)

爬取評論信息並保存
核心:此api會返回歌曲評論
“http://music.163.com/api/v1/resource/comments/R_SO_4_”+ id +"?limit=0&offset=0"

#用id查找評論，按照評論順序記錄
for i in id_list:

    list=[cout,name_list[cout-1]]
    writer.writerow(list)
    
    
    get_url= "http://music.163.com/api/v1/resource/comments/R_SO_4_"+i+"?limit=0&offset=0"

    r= requests.get(get_url,headers=headers)

    json_dict=json.loads(r.content.decode("utf-8"))

    hotcomments= json_dict["hotComments"]

    allcout+=len(hotcomments)
    no=1

	#從返回的json獲取有關信息
    for j in hotcomments:
        nickname= j["user"]["nickname"]
        content= j["content"].replace("\n"," ")
        liked= str(j["likedCount"])+"贊"
        list=["","",no,nickname,liked,content]
        writer.writerow(list)
        no+=1
        
    cout+=1
    
    writer.writerow('')
file.close()

結束顯示

#計時結束
t2= time.clock()-t1

#計數君
print(f"共找到{total}首歌 {allcout}條熱評 用時{t2:.2f}s",end="\n===========================\n")
input("按任意鍵退出。。。")

缺點分析：當歌曲數量多會導致用時特別長
拓展：多線程爬蟲

完

生活的最好狀態是冷冷清清的風風火火——木心