數據挖掘之爬蟲獲取網頁信息

抓取的是豆瓣的API信息，並保存
豆瓣的API已經打不開了

import json
import urllib.request as urlrequest
id_list=[26387939,11803087,20451290]
with open('movie.txt',"w") as file:
    for i in id_list:
        url_visit='https://api.douban.com/v2/movie/{}'.format(i)
        crawl_content=urlrequest.urlopen(url_visit).read()
        json_content=json.loads(crawl_content.decode('utf-8'))   #API是json文件
        rank=json_content['rating']['average']
        file.write("{}{}".format(i,rank))

抓取的是豆瓣的網頁信息
加載 urllib，抓取豆瓣的電影，id_list是豆瓣電影的ID號

import urllib.request as urlrequest
id_list=[26387939,11803087,20451290]
for i in id_list:
    url_visit='https://movie.douban.com/subject/{}'.format(i)    #網址
    crawl_content=urlrequest.urlopen(url_visit).read()            
    aa=crawl_content.decode('utf-8')
    print(aa)

用beautifulsoup分析HTML網頁信息,抓取網站上某個城市最近幾天的溫度信息
網址https://forecast.weather.gov/MapClick.php?lat=37.777120000000025&lon=-122.41963999999996#.XY6v5W5uIuU
pip install beautifulsoup4
pip Install requests

介紹兩種獲取網頁信息的方式：
一種是直接requests，一種是從urllib.request
直接從request中獲取

import requests
from bs4 import BeautifulSoup
# 通過requests獲取
html = requests.get('https://forecast.weather.gov/MapClick.php?lat=37.777120000000025&lon=-122.41963999999996')
soup = BeautifulSoup(html.text)
# 處理本地文件
soup = BeautifulSoup(open('*******.html'))

從urllib.request中獲取：

from bs4 import BeautifulSoup
import urllib.request as urlrequest
url='https://forecast.weather.gov/MapClick.php?lat=37.777120000000025&lon=-122.41963999999996'
url_crawl=urlrequest.urlopen(url).read()
print(url_crawl.decode('utf-8'))
url_content=BeautifulSoup(url_crawl)

想要獲取id=‘seven-day-forecast-body’裏的信息

print(url_content.find(id='seven-day-forecast-container'))  #獲取id裏面的全部內容
print(url_content.find(id='seven-day-forecast-container').get_text())  #只關注裏面的文本信息

將日期，溫度截取下來

from bs4 import BeautifulSoup
import urllib.request as urlrequest
url='https://forecast.weather.gov/MapClick.php?lat=37.777120000000025&lon=-122.41963999999996'
url_crawl=urlrequest.urlopen(url).read()
url_content=BeautifulSoup(url_crawl,'html.parser')

text=url_content.find(id='seven-day-forecast-container')   #定位到目標行的id
#找到目標內所有的日期，天氣，溫度
date_list=text.find_all(class_='period-name')  
desc_list=text.find_all(class_='short-desc')
temp_list=text.find_all(class_='temp')
#裏面總共有最新九天的天氣預報
for i in range(9):
    data=date_list[i].get_text()   #提取文本信息
    desc=desc_list[i].get_text()
    temp=temp_list[i].get_text()
    print("{} {} {}".format(data,desc,temp))