一、Django虛擬環境搭建
1. pip install virtualenv
新建文件夾,CMD進入此文件夾,創建虛擬環境:virtualenv .
文件夾下會生成3個文件:
2. 進入虛擬環境 .\Scripts\activate
命令行前會有文件夾名稱顯示,即當前在該虛擬環境中:
3. 安裝Django
進入虛擬環境,pip install django
pip freeze 查看當前環境的安裝
4. 後續爬蟲需要,安裝lxml,requests
pip install lxml
pip install requests
二、創建項目
1. 創建Django項目
Pycharm打開此項目,運行,報下錯誤:
安裝migrate,去除報警
2. 在同級目錄下,新建app
在mysite的settings.py中的INSTALLED_APPS添加‘kilter'
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'kilter',
]
三、配置urls
1. mysite\urls.py
from django.contrib import admin
from django.urls import path, include
urlpatterns = [
path('admin/', admin.site.urls),
path('', include('kilter.urls')), # 請求跳轉到kilter中的urls
]
2. kilter\urls.py
from django.urls import path
from . import views
urlpatterns = [
path('', views.homepage, name="homepage"),
path('allNovels/', views.allNovels, name="allNovels")
]
四、創建Html模板
1. 新建模板文件夾templates,添加顯示的html文件
2. 使用bootstrap框架,爲html添加元素
base.html(基頁)
使用bootstrap框架中最基礎的模板,添加了導航條(navbar)、巨幕(jumbotron)、主頁內容由homepage.html顯示
<!doctype html>
<html lang="en">
<head>
<!-- Required meta tags -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<!-- Bootstrap CSS -->
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/css/bootstrap.min.css" integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh" crossorigin="anonymous">
<title>Kilter小說網</title>
</head>
<body>
<nav class="navbar navbar-expand-lg navbar-dark bg-dark">
<a class="navbar-brand" href="">主頁</a>
<button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarSupportedContent"
aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
<div class="collapse navbar-collapse" id="navbarSupportedContent">
<ul class="navbar-nav mr-auto">
<li class="nav-item">
<a class="nav-link" href="allNovels/">全部小說</a>
</li>
</ul>
<form name="form" class="form-inline my-2 my-lg-0" action="http://www.xbiquge.la/modules/article/waps.php" method="post">
<input type="text" class="form-control mr-sm-2" value="請填入書名和作者"
name="searchkey" class="search" id="wd" baidusug="2" autocomplete="off">
<button id="sss" class="btn btn-outline-success my-2 my-sm-0" type="submit"> 搜 索</button>
</form>
</div>
</nav>
<br/>
<div class="jumbotron jumbotron-fluid">
<div class="container">
<h1 class="display-4">Kilter小說網</h1>
<p class="lead">Django搭建簡單的小說網,僅學習使用</p>
</div>
</div>
<div class="container">
{% block content %}
{% endblock %}
</div>
<!-- Optional JavaScript -->
<!-- jQuery first, then Popper.js, then Bootstrap JS -->
<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/jquery.slim.min.js"
integrity="sha384-J6qa4849blE2+poT4WnyKhv5vZF5SrPo0iEjwBvKU7imGFAV0wwj1yYfoRSJoZ+n"
crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/umd/popper.min.js"
integrity="sha384-Q6E9RHvbIyZFJoft+2mJbHaEWldlvI9IOYy5n3zV9zzTtmI3UksdQRVvoxMfooAo"
crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/js/bootstrap.min.js"
integrity="sha384-wfSDF2E50Y2D1uUdj0O3uMBJnjuUD4Ih7YwaYd1iqfktj0Uod8GCExl3Og8ifwB6"
crossorigin="anonymous"></script>
</body>
</html>
homepage.html 使用卡片組件(Card)展示信息,同時實現數據交互,使用js處理文本內容過長部分顯示“......”
{% extends 'base.html' %}
{% block content %}
<div class="container">
<div class="row">
{% for img,name,describe,url in content %}
<div class="col-sm">
<div class="card" style="width:230px; height:500px;">
<img src="{{img}}" style="width:230px; height:300px;" class="card-img-top" alt="Card image cap">
<div class="card-body">
<h5 class="card-title">{{name}}</h5>
<script>
var text="{{describe}}";
if (text.length>30)
{
text=text.substring(0,30)+"......";
}
document.write(text);
document.getElementById("p1").innerHTML=text;
</script>
<p id="p1" class="card-text"></p>
<a href="{{url}}" class="btn btn-primary">訪問</a>
</div>
</div>
<br>
</div>
{% endfor %}
</div>
</div>
<p>{{imgs}}</p>
{% endblock %}
allNovels.html 使用畫面輪播(Carousel)、表格(tables)
<!DOCTYPE html>
<html lang="en">
<head>
<!-- Required meta tags -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<!-- Bootstrap CSS -->
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/css/bootstrap.min.css" integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh" crossorigin="anonymous">
<title>全部小說</title>
</head>
<body>
<div class="container">
<div id="carouselExampleIndicators" class="carousel slide" data-ride="carousel">
<ol class="carousel-indicators">
<li data-target="#carouselExampleIndicators" data-slide-to="0" class="active"></li>
<li data-target="#carouselExampleIndicators" data-slide-to="1"></li>
<li data-target="#carouselExampleIndicators" data-slide-to="2"></li>
<li data-target="#carouselExampleIndicators" data-slide-to="3"></li>
</ol>
<div class="carousel-inner">
<div class="carousel-item active">
<img src="http://www.xbiquge.la/files/article/image/10/10489/10489s.jpg"
style="height: 300px;width: 1200px">
<div class="carousel-caption">
<h3>三寸人間</h3>
<p>描述文字!</p>
</div>
</div>
<div class="carousel-item">
<img src="http://www.xbiquge.la/files/article/image/0/951/951s.jpg"
style="height: 300px;width: 1200px">
<div class="carousel-caption">
<h3>伏天氏</h3>
<p>描述文字!</p>
</div>
</div>
<div class="carousel-item">
<img src="http://www.xbiquge.la/files/article/image/5/5541/5541s.jpg"
style="height: 300px;width: 1200px">
<div class="carousel-caption">
<h3>異世邪君</h3>
<p>描述文字!</p>
</div>
</div>
<div class="carousel-item">
<img src="http://www.xbiquge.la/files/article/image/5/5395/5395s.jpg"
style="height: 300px;width: 1200px">
<div class="carousel-caption">
<h3>凡人修仙傳</h3>
<p>描述文字!</p>
</div>
</div>
</div>
<a class="carousel-control-prev" href="#carouselExampleIndicators" role="button" data-slide="prev">
<span class="carousel-control-prev-icon" aria-hidden="true"></span>
<span class="sr-only">Previous</span>
</a>
<a class="carousel-control-next" href="#carouselExampleIndicators" role="button" data-slide="next">
<span class="carousel-control-next-icon" aria-hidden="true"></span>
<span class="sr-only">Next</span>
</a>
</div>
<br/>
<h>
全部小說
</h>
<table class="table">
<thead class="thead-dark">
<tr>
<th scope="col">#</th>
<th scope="col">
<a href="http://www.xbiquge.la/xuanhuanxiaoshuo/">玄幻小說</a>
</th>
<th scope="col">
<a href="http://www.xbiquge.la/xiuzhenxiaoshuo/">修真小說</a>
</th>
<th scope="col">
<a href="http://www.xbiquge.la/dushixiaoshuo/">都市小說</a>
</th>
</tr>
</thead>
<tbody>
<tr>
<th scope="row">1</th>
<td>
<a href="http://www.xbiquge.la/15/15409/">牧神記</a>
</td>
<td>
<a href="http://www.xbiquge.la/2/2210/">全職法師</a>
</td>
<td>
<a href="http://www.xbiquge.la/14/14930/">元尊</a>
</td>
</tr>
<tr>
<th scope="row">2</th>
<td>
<a href="http://www.xbiquge.la/0/656/">莽荒紀</a>
</td>
<td>
<a href="http://www.xbiquge.la/0/205/">執魔</a>
</td>
<td>
<a href="http://www.xbiquge.la/0/215/">魔天記</a>
</td>
</tr>
<tr>
<th scope="row">3</th>
<td>
<a href="http://www.xbiquge.la/20/20948/">最佳女婿</a>
</td>
<td>
<a href="http://www.xbiquge.la/36/36060/">豪婿</a>
</td>
<td>
<a href="http://www.xbiquge.la/3/3120/">極品全能學生</a>
</td>
</tr>
</tbody>
</table>
<br/>
<nav aria-label="Page navigation example">
<ul class="pagination">
<li class="page-item">
<a class="page-link" href="#" aria-label="Previous">
<span aria-hidden="true">«</span>
</a>
</li>
<li class="page-item"><a class="page-link" href="http://www.xbiquge.la/fenlei/3_1.html">1</a></li>
<li class="page-item"><a class="page-link" href="http://www.xbiquge.la/fenlei/3_2.html">2</a></li>
<li class="page-item"><a class="page-link" href="http://www.xbiquge.la/fenlei/3_3.html">3</a></li>
<li class="page-item">
<a class="page-link" href="#" aria-label="Next">
<span aria-hidden="true">»</span>
</a>
</li>
</ul>
</nav>
</div>
<!-- Optional JavaScript -->
<!-- jQuery first, then Popper.js, then Bootstrap JS -->
<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/jquery.slim.min.js"
integrity="sha384-J6qa4849blE2+poT4WnyKhv5vZF5SrPo0iEjwBvKU7imGFAV0wwj1yYfoRSJoZ+n"
crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/umd/popper.min.js"
integrity="sha384-Q6E9RHvbIyZFJoft+2mJbHaEWldlvI9IOYy5n3zV9zzTtmI3UksdQRVvoxMfooAo"
crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/js/bootstrap.min.js"
integrity="sha384-wfSDF2E50Y2D1uUdj0O3uMBJnjuUD4Ih7YwaYd1iqfktj0Uod8GCExl3Og8ifwB6"
crossorigin="anonymous"></script>
</body>
</html>
五、爬取數據
views.py
from django.shortcuts import render
import requests
import json
from lxml import html
# Create your views here.
def homepage(request):
headers = {
'Accept': 'text / html, application / xhtml + xml, application / xml;q = 0.9, image / webp, image / apng, * / *;q = 0.8',
'Cookie': '_abcde_qweasd=0; Hm_lvt_169609146ffe5972484b0957bd1b46d6=1587264547; BAIDU_SSP_lcr=https://www.baidu.com/link?url=uLcdKPYEycqTY_ZNILg7FpWtP29NV3xcIQWRAsDtcrO&wd=&eqid=d1b646a70000536f000000055e9bbc22; _abcde_qweasd=0; bdshare_firstime=1587264547833; Hm_lpvt_169609146ffe5972484b0957bd1b46d6=1587268535',
'Host': 'www.xbiquge.la',
'Referer': 'http: // www.xbiquge.la /',
'User - Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'
}
# 獲取小說url
response = requests.get("http://www.xbiquge.la/paihangbang/", headers=headers, timeout=(3, 7))
etree = html.etree
xml = etree.HTML(response.text)
urls = xml.xpath('//*[@id="main"]/div[2]/ul[1]//li/a/@href')
# 獲取小說對應圖像
imgs_url = []
# 名字
names = []
# 簡介
describes=[]
for url in urls:
response_img = requests.get(url, headers=headers, timeout=(3, 7))
response_img.encoding = "utf-8"
xml_img = etree.HTML(response_img.text)
imgs_url.append(xml_img.xpath('//div[@id="fmimg"]/img/@src')[0])
names.append(xml_img.xpath('//*[@id="info"]/h1/text()')[0])
describes.append(xml_img.xpath('//div[@id="intro"]/p[2]/text()'))
content = zip(imgs_url, names, describes, urls)
context = {}
context["content"] = content
return render(request, 'homepage.html', context)
def allNovels(request):
return render(request,'allNovels.html',{})
六、網頁展示
小說鏈接到新筆趣閣網站小說,僅做學習使用。
參考:https://www.bilibili.com/video/BV1KJ41117HL
問題:
1. cmd運行ok python manage.py runserve ,
pycharm運行報錯“ModuleNotFoundError: No module named 'django',
ImportError: Couldn't import Django. Are you sure it's installed and available on your PYTHONPATH environment variable? Did you forget to activate a virtual environment?”
解決:
a. File->settings->project interpreter ->show all
添加虛擬環境中的python路徑
apply
b. run->Edit configurations->parameters中添加“runserver”
參考:https://blog.csdn.net/xty00/article/details/95354518
2. 報錯:TimeoutError: [WinError 10060] 由於連接方在一段時間後沒有正確答覆或連接的主機沒有反應,連接嘗試失敗
解決:
requests.get(url, headers=headers, proxies=proxies, timeout=5)
如上,在request.get()函數中加個timeout參數就行了。
timeout 是用作設置響應時間(單位:秒)的,可以設置一個float或者一個tuple參數,響應時間分爲連接時間和讀取時間,timeout(3,7)表示的連接時間是3秒,響應時間是7秒。
3. from lxml import etree 報錯,無法導入etree
解決:
from lxml import html
etree = html.etree
xml = etree.HTML(response.text)
4. Django變量在前端js中不起作用
解決:
js是具備作用範圍的,由一對{}來決定。而django的內置標籤和變量也都是由大括號括起來的,這樣就會起衝突。
使用時需要將django的內置標籤和變量用引號括起來
eg: 變量:var a = "{{name}}"
靜態資源:src = "{% static path %}"