簡易的csdn博客信息爬取腳本

前言

---

最近在想辦法批量導出博客爲pdf文檔，臨時寫了半個腳本。

下方是一個簡易的用於csdn博客統計shell腳本，網頁分析就不介紹了，使用時修改部分參數即可使用，也可基於此進行優化。

---

紅框處爲你的博客id

---

#!/bin/bash


source /etc/profile

#SUM爲你的博客的總頁數
#ID填入你的博客ID

SUM=9
ID=GX_1_11_real
URL_list=`cat /script/test2.txt |grep href |grep https://blog.csdn.net/$ID/article/details |grep _blank |awk -F '"' '{print $2}' |uniq |wc -l`
DATE=`date +%F-%T`


> /script/test.txt
> /script/test2.txt

#爬取博客大展示頁的每一頁的內容，記錄入文件
for a in `seq $SUM`
do
	echo "https://blog.csdn.net/$ID/article/list/$a\?" >> /script/test.txt
	curl https://blog.csdn.net/$ID/article/list/$a\? >> /script/test2.txt
done


#篩選需要的參數到文件，也可添加新的參數（不知道什麼原因csdn的每個用戶的博客上都有這個帝都的凜冬的博客，篩選時進行剔除）
cat /script/test2.txt |grep href |grep https://blog.csdn.net/$ID/article/details |grep _blank |awk -F '"' '{print $2}' |uniq |awk '{printf("%d %s\n",NR,$0)}' > /var/log/pdf1.txt
cat /script/test2.txt |grep 'class="read-num"' |grep 閱讀數 |grep -v 閱讀數： |awk -F '>' '{print $3}' |awk -F '<' '{print $1}' |awk '{printf("%d %s\n",NR,$0)}' > /var/log/pdf2.txt
cat /script/test2.txt |grep 'class="read-num"' |grep 評論 |grep -v 評論數： |awk -F '>' '{print $3}' |awk -F '<' '{print $1}' |awk '{printf("%d %s\n",NR,$0)}' > /var/log/pdf3.txt
cat /script/test2.txt |grep article-type |grep -v 帝都的凜冬 |awk '{print $5,$6,$7,$8,$9}' |awk -F '<' '{print $1}' |awk '{printf("%d %s\n",NR,$0)}' > /var/log/pdf4.txt


#將得到的信息匹配並顯示
   
echo "$DATE 博客總數：$URL_list" >> /var/log/url_pa.log	
for i in `seq $URL_list`
	do
	    url=`grep "^$i" /var/log/pdf1.txt |head -n1 |awk '{print $2}'`  
		Read=`grep "^$i" /var/log/pdf2.txt |head -n1 |awk '{print $2}'` 
		Comment=`grep "^$i" /var/log/pdf3.txt |head -n1 |awk '{print $2}'` 
		Title=`grep "^$i" /var/log/pdf4.txt |head -n1 |awk '{print $2}'`

		echo "++++++++++++++++++++++++++++++++"
        echo "博客標題：$Title"
		echo "博客url: $url"
		echo "博客閱讀數：$Read"
		echo "博客評論數：$Comment"
        echo "++++++++++++++++++++++++++++++++"
	done >> /var/log/url_pa.log

---

---

效果如下：