最近看韓劇想重溫一下以前看的韓劇 但是就記得劇情 到網站上找了 太多點的太麻煩,網上問了也回答不了 找的幾個片名都不對,所以就想寫個爬蟲爬去某站上所有的韓劇信息,可以方便查找想看的韓劇
爬取具體如下:
# 爬去網站韓國電視劇的信息 # 爬去片面、時間、地區、主演、簡介
1、導入模塊
import requests
import time
from lxml import etree
2、網站頁面爬取
創建類HanJuInfo
class HanJuInfo():
def __init__(self, url):
self.headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.25 Safari/537.36 Core/1.70.3641.400 QQBrowser/10.4.3284.400'}
self.url = url
# 首頁獲取頁面鏈接
def Get_Html(self):
response = requests.get(url=self.url, headers=self.headers)
if response.status_code == 200:
html = response.text
return html
# 頁面解析出詳情頁的url
def Paser_Html(self):
content = self.Get_Html()
selector = etree.HTML(content)
items = selector.xpath('//div[@class="s-tab-main"]/ul[@class="list g-clear"]/li[@class="item"]')
for item in items:
self.info_url = item.xpath('./a/@href')[0]
self.info_urls = 'https://www.360kan.com'+self.info_url
# 解析詳情頁信息
response = requests.get(url=self.info_urls, headers=self.headers)
if response.status_code == 200:
selector = etree.HTML(response.text)
name = selector.xpath('//div[@class="title-left g-clear"]/h1/text()')[0]
time = selector.xpath('//*[@id="js-desc-switch"]/div[1]/p[2]/text()')[0]
place = selector.xpath('//*[@id="js-desc-switch"]/div[1]/p[3]/text()')[0]
actors = ''.join(selector.xpath('//*[@id="js-desc-switch"]/div[1]/p[6]//a//text()'))
detials = selector.xpath('//*[@id="js-desc-switch"]/div[3]/p/text()')[0]
yield {
'片面': name,
'時間': time,
'地區': place,
'主演': actors,
'簡介': detials
}
info = {
'片面': name,
'時間': time,
'地區': place,
'主演': actors,
'簡介': detials
}
self.save_info(str(info))
def save_info(self, content):
with open('info.txt', 'a', encoding='utf-8')as f:
f.write(content+'\n')
3、主函數調用
開啓翻頁25爬去近七百部韓國電視劇信息
if __name__ == '__main__':
for x in range(1, 26):
url = 'https://www.360kan.com/dianshi/list.php?area=12&pageno={}'.format(str(x))
han = HanJuInfo(url)
time.sleep(1)
print('第%s頁' % x)
for i, x in enumerate(han.Paser_Html()):
print(i, x)
4、爬取結果如下:
圖片太大簡介如下: