首先我們來爬取 http://html-color-codes.info/color-names/ 的一些數據。
按 F12 或 ctrl+u 審查元素，結果如下:
結構很清晰簡單，我們就是要爬 tr 標籤裏面的 style 和 tr 下幾個並列的 td 標籤，下面是爬取的代碼:
#!/usr/bin/env python
# coding=utf-8
import requests
from bs4 import BeautifulSoup
import MySQLdb
print('連接到mysql服務器...')
db = MySQLdb.connect("localhost","hp","Hp12345.","TESTDB")
print('連接上了!')
cursor = db.cursor()
cursor.execute("DROP TABLE IF EXISTS COLOR")
sql = """CREATE TABLE COLOR (
Color CHAR(20) NOT NULL,
Value CHAR(10),
Style CHAR(50) )"""
cursor.execute(sql)
hdrs = {'User-Agent':'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)'}
url = "http://html-color-codes.info/color-names/"
r = requests.get(url, headers = hdrs)
soup = BeautifulSoup(r.content.decode('gbk', 'ignore'), 'lxml')
trs = soup.find_all('tr') # 獲取全部tr標籤成爲一個列表
for tr in trs: # 遍歷列表裏所有的tr標籤單項
style = tr.get('style') # 獲取每個tr標籤裏的屬性style
tds = tr.find_all('td') # 將每個tr標籤下的td標籤獲取爲列表
td = [x for x in tds] # 獲取的列表
name = td[1].text.strip() # 直接從列表裏取值
hex = td[2].text.strip()
# print u'顏色: ' + name + u'顏色值: '+ hex + u'背景色樣式: ' + style
# print 'color: ' + name + '\tvalue: '+ hex + '\tstyle: ' + style
insert_color = ("INSERT INTO COLOR(Color,Value,Style)" "VALUES(%s,%s,%s)")
data_color = (name, hex, style)
cursor.execute(insert_color, data_color)
db.commit()
# print '******完成此條插入!'
print '爬取數據並插入mysql數據庫完成...'運行結果:
$ mysql -u hp -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 28
Server version: 5.7.17 MySQL Community Server (GPL)
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> use TESTDB
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from COLOR;
+----------------------+--------+----------------------------------------+
| Color | Value | Style |
+----------------------+--------+----------------------------------------+
| IndianRed | CD5C5C | background-color:indianred; |
| LightCoral | F08080 | background-color:lightcoral; |
| Salmon | FA8072 | background-color:salmon; |
| DarkSalmon | E9967A | background-color:darksalmon; |
| LightSalmon | FFA07A | background-color:lightsalmon; |
| Crimson | DC143C | background-color:crimson; |
| Red | FF0000 | background-color:red; |
| FireBrick | B22222 | background-color:fireBrick; |
| DarkRed | 8B0000 | background-color:darkred; |
| Pink | FFC0CB | background-color:pink; |
| LightPink | FFB6C1 | background-color:lightpink; |
| HotPink | FF69B4 | background-color:hotpink; |
| DeepPink | FF1493 | background-color:deeppink; |
...
| AntiqueWhite | FAEBD7 | background-color:antiquewhite; |
| Linen | FAF0E6 | background-color:linen; |
| LavenderBlush | FFF0F5 | background-color:lavenderblush; |
| MistyRose | FFE4E1 | background-color:mistyrose; |
| Gainsboro | DCDCDC | background-color:gainsboro; |
| LightGrey | D3D3D3 | background-color:lightgrey; |
| Silver | C0C0C0 | background-color:silver; |
| DarkGray | A9A9A9 | background-color:darkgray; |
| Gray | 808080 | background-color:gray; |
| DimGray | 696969 | background-color:dimgray; |
| LightSlateGray | 778899 | background-color:lightslategray; |
| SlateGray | 708090 | background-color:slategray; |
| DarkSlateGray | 2F4F4F | background-color:darkslategray; |
| Black | 000000 | background-color:black; |
+----------------------+--------+----------------------------------------+
143 rows in set (0.00 sec)
