Instgram爬蟲及其斷點續傳_一個AJAX異步加載爬蟲

原創 磐石若水 2020-06-17 05:16

任務描述

對於給定的Ins賬戶列表，需要爬下所有posts，對於每條post要有

  • 時間
  • 配文
  • 配圖
  • 點贊數
  • 評論數

如果是小視頻，需要有

  • 視頻
  • 觀看數

Ins網站結構分析

Ins的post數據在json文件裏存儲，一個json文件存儲12條post信息，並會給出查詢下一組post的cursor。

由於使用了hash加密，所以典型的查詢鏈接如下：

‘https://www.instagram.com/graphql/query/?query_hash=a5164aed103f24b03e7b7747a2d94e3c&variables=%7B%22id%22%3A%22{user_id}%22%2C%22first%22%3A12%2C%22after%22%3A%22{cursor}%22%7D’

也就是說對於每一組post，需要知道兩個參數：user_id和cursor。將這兩個參數填入url中並requests就能得到想要的json文件。事實上填入參數後的url直接在瀏覽器中打開也能得到對應的json：

顯然，以上的查詢鏈接中的user_id和end_cursor都是從上一頁中繼承的。可是對於每個賬戶的第一頁，顯然沒有辦法繼承。因此第一頁要單獨爬，好在每個賬戶第一頁的url都是固定的：

https://www.instagram.com/{account}/

從第一頁中分析得到user_id和第二頁的cursor即可。

數據存儲和可視化策略

爲了提高存儲效率，用兩層數據格式存儲。外層是一個列表，內存是一個詞典。詞典中按照需要爬取的變量設置不同的屬性，每爬取一個新的post更新一次詞典。得到的結果如下：

[{'img_url': 'https://scontent-frt3-1.cdninstagram.com/vp/2de4fe0ca443d27ead0601306a4d2d9f/5E65FD4A/t51.2885-15/e35/73497417_178613769947279_3483644294279361168_n.jpg?_nc_ht=scontent-frt3-1.cdninstagram.com&_nc_cat=1', 'comment_count': 9042, 'like_count': 565657, 'text': 'Meet today’s #WeeklyFluff, Albert (@pompous.albert), a Selkirk Rex cat who might look faux... but is keeping it real. 😻\\u2063\\n\\u2063\\nPhoto by @pompous.albert'}, 
{'img_url': 'https://scontent-frt3-1.cdninstagram.com/vp/ff83ef12404713e3584ba07441a23913/5E856EC0/t51.2885-15/e35/p1080x1080/72783038_1207153232810009_5652648210556063310_n.jpg?_nc_ht=scontent-frt3-1.cdninstagram.com&_nc_cat=1', 'comment_count': 5506, 'like_count': 637442, 'text': 'For Colombian singer-songwriter Camilo (@camilomusica), the #LatinGRAMMY Awards are a big party of close friends, who just happen to be some of the biggest artists in the world right now. 🔥🌎\\u2063\\n\\u2063\\nSee who Camilo runs into and guess who he’s going to collab with next. It’s #GameOn at the @latingrammys, right now on our story.'}]

爲了使爬取結果更直觀，寫了一個函數將以上數據轉成表格形式:

def nestedlist2csv(list, out_file):
    with open(out_file, 'w') as f:
        w = csv.writer(f)
        fieldnames=list[0].keys() 
        w.writerow(fieldnames)
        for row in list:
            w.writerow(row.values())

最終的結果如下：

主程序

import re
import json
import time
import random
import requests
from pyquery import PyQuery as pq
import pandas as pd
import csv
from datetime import datetime
import math

def baseurl(acc):
    url_base = 'https://www.instagram.com/%s/'%acc
    return(url_base)

uri = 'https://www.instagram.com/graphql/query/?query_hash=a5164aed103f24b03e7b7747a2d94e3c&variables=%7B%22id%22%3A%22{user_id}%22%2C%22first%22%3A12%2C%22after%22%3A%22{cursor}%22%7D'

idlist = pd.read_table('accidlist.txt',header=0,encoding='gb18030',delim_whitespace=True)
idlist.columns=['acc','id','postno']

headers = {
    "Origin": "https://www.instagram.com/",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/58.0.3029.110 Safari/537.36",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "accept-encoding": "gzip, deflate, sdch, br",
    "accept-language": "zh-CN,zh;q=0.8",
    "X-Instragram-AJAX": "1",
    "X-Requested-With": "XMLHttpRequest",
    "Upgrade-Insecure-Requests": "1",
}

def get_html(url):
    try:
        response = requests.get(url, headers=headers)
        if response.status_code == 200:
            return response.text
        else:
            print('請求網頁源代碼錯誤, 錯誤狀態碼：', response.status_code)
    except Exception as e:
        print(e)
        return None


def get_json(headers, url):
    try:
        response = requests.get(url, headers=headers, timeout=10)
        if response.status_code == 200:
            return response.json()
        else:
            print('請求網頁json錯誤, 錯誤狀態碼：', response.status_code)
    except Exception as e:
        print(e)
        time.sleep(60 + float(random.randint(1, 4000)) / 100)
        return get_json(headers, url)


def get_pics(picurl,picname):
    picresp = requests.get(picurl, headers=headers, timeout=10)
    with open('%s.png'%picname, 'wb') as f:
        f.write(picresp.content)

def nestedlist2csv(list, out_file):
    with open(out_file, 'w') as f:
        w = csv.writer(f)
        fieldnames=list[0].keys() 
        w.writerow(fieldnames)
        for row in list:
            w.writerow(row.values())

def get_date(timestamp):
    local_str_time = datetime.fromtimestamp(timestamp).strftime('%Y-%m-%d')
    return local_str_time

def get_samples(html,acc):
    samples = []
    page = 0
    user_id = re.findall('"profilePage_([0-9]+)"', html, re.S)[0]
    print("The user id is %s"%user_id)

    doc = pq(html)
    items = doc('script[type="text/javascript"]').items()
    for item in items:
        if item.text().strip().startswith('window._sharedData'):
            js_data = json.loads(item.text()[21:-1], encoding='utf-8')

            edges = js_data["entry_data"]["ProfilePage"][0]["graphql"]["user"]["edge_owner_to_timeline_media"]["edges"]
            totalpost = js_data["entry_data"]["ProfilePage"][0]["graphql"]["user"]["edge_owner_to_timeline_media"]["count"]
            totalpage = math.ceil(totalpost/12)
            page_info = js_data["entry_data"]["ProfilePage"][0]["graphql"]["user"]["edge_owner_to_timeline_media"][
                'page_info']
            cursor = page_info['end_cursor']
            flag = page_info['has_next_page']

            for edge in edges:
                sample = {}

                if edge['node']['display_url']:
                    sample["Influencer"] = acc
                    timestamp = edge['node']['taken_at_timestamp']
                    sample["date"] = get_date(timestamp)
                    sample["comment_count"] = edge['node']['edge_media_to_comment']["count"]
                    sample["like_count"] = edge['node']['edge_liked_by']["count"]

                if edge['node']['shortcode']:
                    shortcode = edge['node']['shortcode']
                    sample['postlink'] = 'https://www.instagram.com/p/%s/'%(shortcode)
                    textUrl = 'https://www.instagram.com/p/' + shortcode + '/?__a=1'
                    textRespose = get_json(headers, textUrl)
                    try:
                        textDict = textRespose['graphql']['shortcode_media']['edge_media_to_caption']['edges'][0][
                            'node']
                        sample["caption"] = str(textDict)[10:-2]
                    except:
                        sample["caption"] = ""
                    children = textRespose["graphql"]["shortcode_media"].get('edge_sidecar_to_children')
                    if children:
                        sample['multipic'] = 'True'
                        picurls = ""
                        for child in children['edges']:
                            picurls = picurls + child['node']['display_url'] + ','
                        sample['img_urls'] = picurls
                    else:
                        sample['multipic'] = 'False'
                        sample['img_urls'] = textRespose['graphql']['shortcode_media']['display_url']
                    isvideo = textRespose["graphql"]["shortcode_media"].get('is_video')
                    if isvideo:
                        sample['video_url'] = textRespose["graphql"]["shortcode_media"].get('video_url')
                        sample['video_view_count'] = textRespose["graphql"]["shortcode_media"].get('video_view_count')
                    else:
                        sample['video_url'] = ""
                        sample['video_view_count'] = ""

                samples.append(sample)
                time.sleep(float(random.randint(1, 3)))
            nestedlist2csv(samples,'%s_postlist.csv'%acc)
            page += 1
            print("Finish the %s page of %s, the total page number is %s"%(page,acc,totalpage))

    while flag:
        url = uri.format(user_id=user_id, cursor=cursor)
        print([user_id, cursor])
        js_data = get_json(headers, url)
        infos = js_data['data']['user']['edge_owner_to_timeline_media']['edges']
        cursor = js_data['data']['user']['edge_owner_to_timeline_media']['page_info']['end_cursor']
        flag = js_data['data']['user']['edge_owner_to_timeline_media']['page_info']['has_next_page']
        for info in infos:
            sample = {}
            sample["Influencer"] = acc
            timestamp = info['node']['taken_at_timestamp']
            sample["date"] = get_date(timestamp)
            sample["comment_count"] = info['node']['edge_media_to_comment']["count"]
            sample["like_count"] = info['node']['edge_media_preview_like']["count"]

            if info['node']['shortcode']:
                time.sleep(1)
                shortcode = info['node']['shortcode']
                sample['postlink'] = 'https://www.instagram.com/p/%s/' % (shortcode)
                textUrl = 'https://www.instagram.com/p/' + shortcode + '/?__a=1'
                textRespose = get_json(headers, textUrl)

                try:
                    textDict = textRespose['graphql']['shortcode_media']['edge_media_to_caption']['edges'][0][
                        'node']
                    sample["caption"] = str(textDict)[10:-2]
                except:
                    sample["caption"] = ""

                children = textRespose["graphql"]["shortcode_media"].get('edge_sidecar_to_children')
                if children:
                    sample['multipic'] = 'True'
                    picurls = ""
                    for child in children['edges']:
                        picurls = picurls + child['node']['display_url'] + ','
                    sample['img_urls'] = picurls
                else:
                    sample['multipic'] = 'False'
                    sample['img_urls'] = textRespose['graphql']['shortcode_media']['display_url']
                isvideo = textRespose["graphql"]["shortcode_media"].get('is_video')
                if isvideo:
                    sample['video_url'] = textRespose["graphql"]["shortcode_media"].get('video_url')
                    sample['video_view_count'] = textRespose["graphql"]["shortcode_media"].get('video_view_count')
                else:
                    sample['video_url'] = ""
                    sample['video_view_count'] = ""
            samples.append(sample)
            time.sleep(float(random.randint(1, 3)))
        nestedlist2csv(samples,'%s_postlist.csv'%acc)
        page += 1
        print("Finish the %s page of %s, the total page number is %s" % (page, acc, totalpage))


def main():
    for i in range(len(idlist.loc[:, 'acc'])):
        acc = idlist.loc[i, 'acc']
        url = baseurl(acc)
        print(url)
        html = get_html(url)
        ticks = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
        print("開始處理賬戶%s,當前時間爲:%s"%(acc,ticks))
        try:
            get_samples(html,acc)
        except:
            print("程序中斷，中斷時間爲%s" % (ticks))
            break
        print("結束處理賬戶%s,當前時間爲:%s" % (acc, ticks))
        time.sleep(float(random.randint(1, 4000)/10))

if __name__ == '__main__':
    start = time.time()
    main()

斷點續傳

import time
import random
import requests
import pandas as pd
import csv
from datetime import datetime

uri = 'https://www.instagram.com/graphql/query/?query_hash=a5164aed103f24b03e7b7747a2d94e3c&variables=%7B%22id%22%3A%22{user_id}%22%2C%22first%22%3A12%2C%22after%22%3A%22{cursor}%22%7D'

idlist = pd.read_table('accidlist.txt',header=0,encoding='gb18030',delim_whitespace=True)
idlist.columns=['acc','id','postno']

headers = {
    "Origin": "https://www.instagram.com/",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/58.0.3029.110 Safari/537.36",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "accept-encoding": "gzip, deflate, sdch, br",
    "accept-language": "zh-CN,zh;q=0.8",
    "X-Instragram-AJAX": "1",
    "X-Requested-With": "XMLHttpRequest",
    "Upgrade-Insecure-Requests": "1",
}

def get_html(url):
    try:
        response = requests.get(url, headers=headers)
        if response.status_code == 200:
            return response.text
        else:
            print('請求網頁源代碼錯誤, 錯誤狀態碼：', response.status_code)
    except Exception as e:
        print(e)
        return None


def get_json(headers, url):
    try:
        response = requests.get(url, headers=headers, timeout=10)
        if response.status_code == 200:
            return response.json()
        else:
            print('請求網頁json錯誤, 錯誤狀態碼：', response.status_code)
    except Exception as e:
        print(e)
        time.sleep(60 + float(random.randint(1, 4000)) / 100)
        return get_json(headers, url)


def get_pics(picurl,picname):
    picresp = requests.get(picurl, headers=headers, timeout=10)
    with open('%s.png'%picname, 'wb') as f:
        f.write(picresp.content)

def nestedlist2csv(list, out_file):
    with open(out_file, 'a') as f:
        w = csv.writer(f)
        fieldnames=list[0].keys()  
        for row in list:
            w.writerow(row.values())

def get_date(timestamp):
    local_str_time = datetime.fromtimestamp(timestamp).strftime('%Y-%m-%d')
    return local_str_time

def get_breakpoint(breakpage, id, cursor, acc, flag):
    page = breakpage
    while flag:
        samples = []
        url = uri.format(user_id=id, cursor=cursor)
        print([id, cursor])
        js_data = get_json(headers, url)
        infos = js_data['data']['user']['edge_owner_to_timeline_media']['edges']
        cursor = js_data['data']['user']['edge_owner_to_timeline_media']['page_info']['end_cursor']
        flag = js_data['data']['user']['edge_owner_to_timeline_media']['page_info']['has_next_page']
        for info in infos:
            sample = {}
            sample["Influencer"] = acc
            timestamp = info['node']['taken_at_timestamp']
            sample["date"] = get_date(timestamp)
            sample["comment_count"] = info['node']['edge_media_to_comment']["count"]
            sample["like_count"] = info['node']['edge_media_preview_like']["count"]

            if info['node']['shortcode']:
                time.sleep(1)
                shortcode = info['node']['shortcode']
                sample['postlink'] = 'https://www.instagram.com/p/%s/' % (shortcode)
                textUrl = 'https://www.instagram.com/p/' + shortcode + '/?__a=1'
                textRespose = get_json(headers, textUrl)

                try:
                    textDict = textRespose['graphql']['shortcode_media']['edge_media_to_caption']['edges'][0]['node']
                    sample["caption"] = str(textDict)[10:-2]
                except:
                    sample["caption"] = ""

                children = textRespose["graphql"]["shortcode_media"].get('edge_sidecar_to_children')
                if children:
                    sample['multipic'] = 'True'
                    picurls = ""
                    for child in children['edges']:
                        picurls = picurls + child['node']['display_url'] + ','
                    sample['img_urls'] = picurls
                else:
                    sample['multipic'] = 'False'
                    sample['img_urls'] = textRespose['graphql']['shortcode_media']['display_url']
                isvideo = textRespose["graphql"]["shortcode_media"].get('is_video')
                if isvideo:
                    sample['video_url'] = textRespose["graphql"]["shortcode_media"].get('video_url')
                    sample['video_view_count'] = textRespose["graphql"]["shortcode_media"].get('video_view_count')
                else:
                    sample['video_url'] = ""
                    sample['video_view_count'] = ""
            samples.append(sample)
            time.sleep(float(random.randint(1, 3)))
        nestedlist2csv(samples, '%s_postlist.csv' % acc)
        page += 1
        print("Finish the %s page of %s" % (page, acc))

if __name__ == '__main__':
    breakpage = 58
    id = "89899"
    cursor = "QVFBajF5bVdqV0otYUhfSGJHTFZOdDhULTQ3X19kU0J3ZXd5cXJ2UnNkblNkQW5sU3A0UHFNeU1YbjU1Sm5UZ3pkaUphTC1xZVVyeTRaLXFFdDRyc0lXNw=="
    acc = "oliviermorisse"
    flag = "true"
    ticks = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
    print("Restart time is %s"%ticks)
    get_breakpoint(breakpage, id, cursor, acc, flag)

爬所有圖片

import requests
import pandas as pd
import random
import os
import time
from datetime import datetime


picpath = '/Users/mengjiexu/Googledrive/Influencers_pic/'
postpath = '/Users/mengjiexu/Googledrive/Influencers_post/'

headers = {
    "Origin": "https://www.instagram.com/",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/58.0.3029.110 Safari/537.36",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "accept-encoding": "gzip, deflate, sdch, br",
    "accept-language": "zh-CN,zh;q=0.8",
    "X-Instragram-AJAX": "1",
    "X-Requested-With": "XMLHttpRequest",
    "Upgrade-Insecure-Requests": "1",
}

def parsepics(filename):
    data = pd.read_excel(postpath + filename)
    influencer = filename.split('_postlist')[0]
    print(influencer)
    inpicpath = '%s%s_pic/'%(picpath,influencer)
    print(inpicpath)
    #os.makedirs(inpicpath)
    os.chdir(inpicpath)
    postlink = data['postlink']
    piclink = data['img_urls']
    multipic = data['multipic']
    for i in range(len(postlink)):
        postid = postlink[i].split('p/')[-1].split('/')[0]
        postindex = len(postlink) - i
        print('This is the %s post of %s' % (i, influencer))
        while multipic[i]:
            pics = piclink[i].split(',')[:-1]
            for j in range(len(pics)):
                try:
                    picresp = requests.get(pics[j], headers=headers, timeout=10)
                    with open('%s%s_%s_%s_%s.jpeg' % (inpicpath, influencer, postindex, j, postid), 'wb') as f:
                        f.write(picresp.content)
                    time.sleep(float(random.randint(0, 2)))
                except:
                    pass
            break
        else:
            try:
                picresp = requests.get(piclink[i], headers=headers, timeout=10)
                with open('%s%s_%s_%s.jpeg' % (inpicpath, influencer, postindex, postid), 'wb') as f:
                    f.write(picresp.content)
                time.sleep(float(random.randint(0, 1)))
            except:
                pass


for filename in os.listdir(postpath):
    ticks = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
    with open(picpath + 'Processinghistory.txt','a') as f:
        f.write('Start to process %s, starting time is %s'%(filename, ticks)+'\r')
        parsepics(filename)
        f.write('End processing %s, ending time is %s'%(filename, ticks)+'\r')
        time.sleep(float(random.randint(0, 1)))

後續

  • 將程序放到服務器或colab上跑
  • 爬所有的followers

主要參考鏈接

https://blog.csdn.net/qq_27297393/article/details/82915102

發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章

pyppeteer初次運行需要下載Chrome無法下載或者下載慢的解決方案

目前網上最多的其實都是自己手動下載，這裏說一下最簡單的、最方便的解決方案。 這裏進行替換就好了，這裏是不需要修改源碼的。 只要在我們引用之前設置下環境變量就好了。 import os DEFAULT_DOWNLOAD_HOST

成小新 2020-07-08 05:42:09

爬蟲入門--糗百

文章目錄1.爬蟲基本概念2.Requests 庫基本介紹3.Beautiful Soup 庫介紹 本文目標： 掌握爬蟲的基本概念 Requests 及 Beautiful Soup 兩個 Python 庫的基本使用 通過以上知識完

shelley__huang 2020-07-08 05:17:04

爬蟲知識梳理

爬蟲知識簡單梳理 文章目錄爬蟲知識簡單梳理1.分類2.模擬瀏覽器請求數據（常用）3.數據提取XML和HTML區別a. 正則（re模塊）b. Beautiful Soup4模塊[API鏈接](http://beautifulsoup

AggressionStorm 2020-07-08 04:15:54

爬取高清站長之家美圖（想爬多少爬多少）

方方面面總是會需要使用一些圖片做封面或背景。圖片有兩種來源：一是通過創可貼自己動手修改下就可以用了，還一種就是在網上下載圖片。 那如何下載高清並且可以供使用的圖片了？我是使用的站長之家網站下載來的圖片（http://sc.chin

iplaypy(蟒蛇师) 2020-07-08 10:43:01

爬蟲-獲取指定城市所有企業信息

之前寫過一個博客：python爬蟲從企查查獲取企業信息-手工繞開企查查的登錄驗證 https://blog.csdn.net/rock4you/article/details/88254612 但只能獲取指定公司的信息，這篇博客的目的是獲

rock4you 2020-07-08 08:24:13

學習爬蟲基礎6-HTTP響應狀態碼參考：

HTTP響應狀態碼參考： 1xx:信息 100 Continue 服務器僅接收到部分請求，但是一旦服務器並沒有拒絕該請求，客戶端應該繼續發送其餘的請求。 101 Switching Protocols 服務器轉換協議：服務器

疯狂打码中 2020-07-08 07:56:24

【Python成長之路】Boss直聘爬蟲第2彈：selenium找不到元素的常見問題

哈嘍大家好，我是鵬哥。   今天繼續上週的主題是 —— boss直聘網站的爬蟲。   ~~~上課鈴~~~   盜墓筆記·十年人間李常超（Lao乾媽） - 盜墓筆記·十年人間 1 寫在前面 上一篇文章講的如何破解boss直聘網站的滑塊

鹏哥贼优秀 2020-07-08 06:35:40

【Python成長之路】破解Boss直聘網站滑塊驗證

  哈嘍大家好，我是鵬哥。   今天要記錄的內容是 —— 破解Boss直聘網站的滑塊驗證。     …… 上 課 鈴…… TuesdayBurak Yeter;Danelle Sandoval - Tuesday 1 寫在前面 最近

鹏哥贼优秀 2020-07-08 06:35:38

4.8實驗記錄（爬取so問答）

只爬取指定的類的div from bs4 import BeautifulSoup soup=BeautifulSoup(h,'html.parser') a=soup.find_all('div',class_="post-te

Rbdash 2020-07-08 06:12:59

爬蟲MOOC 第二週 入門

首先下載Beautiful Soup 4，然後解壓安裝，記得安裝代碼是 python setup.py install 這裏要說明一點！！！很重要，我吃了一個多小時的虧。 我把文件命名爲 bs4.py 這時候如果要from bs4 im

Dr_David_S 2020-07-08 03:09:02

Wireshark抓包工具介紹

      在編寫爬蟲抓取app數據的時候，往往需要通過抓包工具對app進行抓包分析，例如：Fiddler、Charles等，但是在偶爾的情況下，你會發現通過這些工具抓包抓取失敗，一啓用抓包工具，就會顯示網絡錯誤。爲什麼會這樣了？ 我個人

刘延林 | 梦陆 2020-07-08 02:38:37

Python根據地名獲取經緯度

以下是通過訪問高德地圖接口獲取指定地名的經緯度信息代碼： import requests import re import json '''獲取地址座標''' def get_get_location_m(name): url

刘延林 | 梦陆 2020-07-08 02:38:27

【python】解決網頁gb2312編碼中文亂碼問題

在寫爬蟲的時候，有時候會遇到非utf-8的網頁，可能會造成中文亂碼問題，比如說遇到的是gb2312的編碼。直接打印中文就會變成亂碼 1.查看網頁編碼 document.characterSet 2.正確解碼 網頁使用的是gb23

冰冷的希望 2020-07-08 01:22:12

Scrapy 架構分析(一)

Scrapy（/ˈskreɪpi/) 是一個Python編寫的開源網絡爬蟲框架。它是一個被設計用於爬取網絡數據、提取結構性數據的程序框架。 使用Scrapy框架可以很方便地爬取網站。 一、Scrapy入門 菜鳥教程Scrapy入門

Tawn0000 2020-07-08 00:37:36

菜鳥爬蟲——獲取安居客二手房信息

以安居客二手房爲例前言瞭解爬蟲爬蟲目錄結構爬蟲主體代碼items.py反反爬蟲策略運行爬蟲 前言 因爲需要一些二手房數據，菜鳥開啓了爬蟲之路！不過需要注意的是，在爬取數據時，要遵守《中華人民共和國網絡安全法》以及《規範互聯網信息服

温柔小静 2020-07-08 00:33:51