使用spyder编写爬虫_CodingPark编程公园

文章介绍

本文主要讲述了利用Anaconda spyder进行爬虫编写

使用spyder编写爬虫

准备工作

这次我们使用 heartbeat -> cid

需要注意的坑

1
每行脚本按 command + 回车 —> 执行
⚠️每行都需执行一次

2
可以不写print语句
而选取所要print的部分进行 ** command + 回车 —> 执行** 输出

3
终端输出信息不完全

pd.set_option(‘display.max_rows’,n)将看不到的行显示完整

import numpy as np
import pandas as pd
pd.set_option('display.max_columns',10)
pd.set_option('display.max_rows',100)			#设置最大可见100行
df=pd.DataFrame(np.random.rand(100,10))
df.head(100)
	

pd.set_option(‘display.max_columns’,n)将看不到的列显示完整

import numpy as np
import pandas as pd
pd.set_option('display.max_columns',10)			 #给最大列设置为10列
df=pd.DataFrame(np.random.rand(2,10))
df.head()

完整代码(基础功能)

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Sat May  9 17:34:24 2020

@author: atom-g
"""

import requests 
from bs4 import BeautifulSoup
import pandas as pd      # 数据处理(美化)

url = 'https://comment.bilibili.com/83089367.xml'
request = requests.get(url)
request.status_code     # 200
request.encoding = 'utf-8'
request.text

soup = BeautifulSoup(request.text,'lxml')

results = soup.find_all('d')
type(results)       # s4.element.ResultSet

# 我们要转换成list 才能.text 取出文字
comments = [comment.text for comment in results]
# 接下来 我一条一条刷 我这么写为了读者看的清每一步操作
comments = [comment.upper() for comment in comments]  # 大写
comments = [comment.replace(' ','') for comment in comments ]   # 去除空格

# 引入stop_words
stop_words = ['▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅',
              '▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄']

comments_fin = [comment for comment in comments if comment not in stop_words]

pd.set_option('display.max_rows',100000)    #设置最大可见行
catalog = pd.DataFrame({'DanMu':comments_fin})
cipin = catalog['DanMu'].value_counts()

输出信息

完整代码(Plus)

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Sat May  9 17:34:24 2020

@author: atom-g
"""

import requests 
from bs4 import BeautifulSoup
import pandas as pd      # 数据处理(美化)
import numpy as np

url = 'https://comment.bilibili.com/83089367.xml'
request = requests.get(url)
request.status_code     # 200
request.encoding = 'unicode'
request.text

soup = BeautifulSoup(request.text,'lxml')

results = soup.find_all('d')
type(results)       # s4.element.ResultSet

# 我们要转换成list 才能.text 取出文字
comments = [comment.text for comment in results]
# 接下来 我一条一条刷 我这么写为了读者看的清每一步操作
comments = [comment.upper() for comment in comments]  # 大写
comments = [comment.replace(' ','') for comment in comments ]   # 去除空格

# 引入stop_words
stop_words = ['▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅',
              '▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▄▄▅▆▇█▇▆▅▄▄▅▆▇█▇▆▅▄▄',
              '，','！',']','。']

comments_fin = [comment for comment in comments if comment not in stop_words]

pd.set_option('display.max_rows',100000)    # 设置最大可见100行
catalog = pd.DataFrame({'DanMu':comments_fin})
cipin = catalog['DanMu'].value_counts()

import jieba

DanMustr = ''.join(i for i in comments_fin if i not in stop_words)     # 拼成串

words = list(jieba.cut(DanMustr))
words_fin_DanMustr = [word for word in words if word not in stop_words]      # 这里我学到了stop_word需完全对应才可
words_fin_DanMustr_str = ','.join(words_fin_DanMustr)
# py 生成本地txt
file_handle=open('/Users/atom-g/spyder/Cai.txt',mode='w')
file_handle.write(words_fin_DanMustr_str)
file_handle.close()

words_fin = [i for i in words if len(i)>1]

# np.set_printoptions(threshold=1e6)   #利用np全部输出
# cc = np.array(words_fin)
# cc.tofile('/Users/atom-g/spyder/Cai.txt')       # txt至本地


import  wordcloud   # 生成词云
wc = wordcloud.WordCloud(height = 1000, width = 1000, font_path = 'simsun.ttc')
wc.generate(' '.join(words_fin))

from matplotlib import pyplot as plt

plt.imshow(wc)
wc.to_file('/Users/atom-g/spyder/Cai.png')      # 图片至本地

输出信息

✏️Python-list转字符串

命令：''.join(list)
其中，引号中是字符之间的分割符，如“,”，“;”，“\t”等等
如：
list = [1, 2, 3, 4, 5]
''.join(list) 结果即为：12345
','.join(list) 结果即为：1,2,3,4,5

✏️Python-字符串转list

print list('12345')
输出： ['1', '2', '3', '4', '5']
print list(map(int, '12345'))
输出： [1, 2, 3, 4, 5]

str2 = "123 sjhid dhi" 
list2 = str2.split() #or list2 = str2.split(" ") 
print list2 
['123', 'sjhid', 'dhi']

str3 = "www.google.com" 
list3 = str3.split(".") 
print list3 
['www', 'google', 'com']

✏️Python-生成本地txt模式

特别鸣谢

📍Python spyder显示不全df列和行
https://blog.csdn.net/Arwen_H/article/details/83510364?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.nonecase
📍python（如何将数据写入本地txt文本文件）
https://blog.csdn.net/huo_1214/article/details/79153847
📍Python list 和 str 互转
https://blog.csdn.net/qq_35531549/article/details/88209377