4.1Beautiful Soup庫的小測
import requests
r=requests.get('https://python123.io/ws/demo.html')
demo=r.text
print(r.text)
# <html><head><title>This is a python demo page</title></head>
# <body>
# <p class="title"><b>The demo python introduces several python courses.</b></p>
# <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
# <a href="http://www.icourse163.org/course/BIT-268001" class="py1" id="link1">Basic Python</a> and <a href="http://www.icourse163.org/course/BIT-1001870001" class="py2" id="link2">Advanced Python</a>.</p>
# </body></html>
from bs4 import BeautifulSoup
soup=BeautifulSoup(demo,"html.parser")#給出的解釋器是html.parser
print(soup.prettify())
# <html>
# <head>
# <title>
# This is a python demo page
# </title>
# </head>
# <body>
# <p class="title">
# <b>
# The demo python introduces several python courses.
# </b>
# </p>
# <p class="course">
# Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
# <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">
# Basic Python
# </a>
# and
# <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">
# Advanced Python
# </a>
# .
# </p>
# </body>
# </html>
#解析成功4.2 Beautiful Soup庫的基本元素
4.2.1Beautiful Soup庫的理解
Beautiful Soup庫是解析、遍歷、維護“標籤樹”的功能庫
只要提供的文件是標籤類型，Beautiful Soup庫都可以對其有很好的解析
每個標籤的具體格式
p標籤
- 屬性定義標籤的特點，屬性由鍵值對構成
4.2.2Beautiful Soup庫的引用
from bs4 import BeautifulSoup4.2.3 Beautiful Soup類
Beautiful Soup對應一個HTML/XML文檔的全部內容
4.2.4Beautiful Soup類的基本元素
| 基本元素 | 說明 |
|---|---|
| Tag | 標籤，最基本的信息組織單元，分別用<>和</>標明開頭和結尾 |
| Name | 標籤的名字，< p >…</>的名字是‘p’，格式：< tag >.name |
| Attributes | 標籤的屬性，字典形式組織，格式爲：< tag>.attrs |
| NavigableString | 標籤內的非屬性字符串，<>…< />中字符串，格式爲：< tag>.string |
| Comment | 標籤內字符串的註釋部分，一種特殊的Comment類型 |
from bs4 import BeautifulSoup
import requests
r=requests.get('https://python123.io/ws/demo.html')
demo=r.text
soup=BeautifulSoup(demo,"html.parser")
print(soup)
# <html><head><title>This is a python demo page</title></head>
# <body>
# <p class="title"><b>The demo python introduces several python courses.</b></p>
# <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
# <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p>
# </body></html>
print(soup.title)
#<title>This is a python demo page</title>
tag=soup.a
print(tag)
# <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>
print(soup.a.name)#a
print(soup.a.parent.name)#p
print(tag.attrs)#{'href': 'http://www.icourse163.org/course/BIT-268001', 'class': ['py1'], 'id': 'link1'}
print(tag.attrs['class'])#['py1']
#a標籤的鏈接屬性
print(tag.attrs['href'])
#http://www.icourse163.org/course/BIT-268001
#標籤屬性的類型
print(type(tag.attrs))#<class 'dict'>
#標籤類型
print(type(tag))#<class 'bs4.element.Tag'>
#查看a標籤
print(soup.a)
# <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>
#查看a標籤的字符串
print(soup.a.string)#Basic Python
print(soup.p.string)#The demo python introduces several python courses.
print(type(soup.p.string))#<class 'bs4.element.NavigableString'>
newsoup=BeautifulSoup("<b><!--This is a comment--></b><p> This is not a comment</p>","html.parser")
print(newsoup.b.string)#This is a comment
print(type(newsoup.b.string))
#<class 'bs4.element.Comment'>
print(newsoup.p.string)
#This is not a comment
print(type(newsoup.p.string))
#<class 'bs4.element.NavigableString'>
4.3 基於bs4庫的HTML內容遍歷方法
換一種表示：
4.3.1標籤樹的下行遍歷
| 屬性 | 說明 |
|---|---|
| .contents | 子節點的列表，將< tag >所有兒子節點存入列表中（只能獲取下一級兒子節點） |
| .children | 子節點的迭代類型，與.contents類似，用於循環遍歷兒子節點 |
| .descendants | 子孫節點的迭代類型，包含所有子孫結點，用於循環遍歷（可以獲取所有子節點） |
print(soup.head)#打印head標籤
print(soup.body.contents)#打印body標籤的下一級標籤
print(len(soup.body.contents))#打印body標籤的下一級標籤的個數import requests
r=requests.get('https://python123.io/ws/demo.html')
demo=r.text
from bs4 import BeautifulSoup
soup=BeautifulSoup(demo,"html.parser")#給出的解釋器是html.parser
print(soup.prettify())
# <html>
# <head>
# <title>
# This is a python demo page
# </title>
# </head>
# <body>
# <p class="title">
# <b>
# The demo python introduces several python courses.
# </b>
# </p>
# <p class="course">
# Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
# <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">
# Basic Python
# </a>
# and
# <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">
# Advanced Python
# </a>
# .
# </p>
# </body>
# </html>
print(soup.head)
#<head><title>This is a python demo page</title></head>
print(soup.head.contents)
#[<title>This is a python demo page</title>]
print(soup.body.contents)
# ['\n', <p class="title"><b>The demo python introduces several python courses.</b></p>, '\n', <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
# <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p>, '\n']
print(len(soup.body.contents))#5
print(soup.body.contents[1])
#<p class="title"><b>The demo python introduces several python courses.</b></p>
- 遍歷兒子子孫節點
#遍歷兒子節點
for child in soup.body.children:
print(child)
#遍歷子孫節點
for child in soup.body.children:
print(child)4.3.2 標籤樹的上行遍歷
| 屬性 | 說明 |
|---|---|
| .parent | 節點的父親標籤 |
| .parents | 節點先輩標籤的迭代類型，用於循環遍歷先輩節點 |
print(soup.title.parent)
#<head><title>This is a python demo page</title></head>
print(soup.html.parent)
# <html><head><title>This is a python demo page</title></head>
# <body>
# <p class="title"><b>The demo python introduces several python courses.</b></p>
# <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
# <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p>
# </body></html>
for parent in soup.a.parents:
if parent is None:
print(parent)
else:
print(parent.name)
# p
# body
# html
# [document]4.3.3標籤樹的平行遍歷
| 屬性 | 說明 |
|---|---|
| .next_sibling | 返回按照HTML文本順序的下一個平行節點標籤 |
| .previos_sibling | 返回按照HTML文本順序的上一個平行節點標籤 |
| .next_siblings | 迭代類型，返回按照HTML文本順序的後續所有平行節點標籤 |
| .previous_siblings | 迭代類型，返回按照HTML文本順序的前續所有平行節點標籤 |
平行遍歷發生在同一個父節點下的各節點間
print(soup.a.next_sibling)#平行節點的後一個兄弟標籤
#and
print(soup.a.previous_sibling)#平行節點的前一個兄弟標籤
#Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
for sibling in soup.a.next_siblings:
print(sibling)#循環遍歷後續節點
# <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>
for sibling in soup.a.previous_siblings:
print(sibling)#循環遍歷前續節點
#Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
4.4 基於bs4庫的HTML格式化和編碼
如何讓< HTML>內容更加友好點？？
bs4庫的prettify()方法
import requests
r=requests.get('https://python123.io/ws/demo.html')
demo=r.text
from bs4 import BeautifulSoup
soup=BeautifulSoup(demo,"html.parser")#給出的解釋器是html.parser
print(soup.prettify())
# <html>
# <head>
# <title>
# This is a python demo page
# </title>
# </head>
# <body>
# <p class="title">
# <b>
# The demo python introduces several python courses.
# </b>
# </p>
# <p class="course">
# Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
# <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">
# Basic Python
# </a>
# and
# <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">
# Advanced Python
# </a>
# .
# </p>
# </body>
# </html>bs4庫將任何讀入HTML文件和字符串都轉化成“utf-8”編碼，“utf-8”編碼使用國際通用的標準編碼格式，能很好支持中文等第三國語言