使用scrapy抓取Youtube播放頁數據

可參看Knowsmore

抓取Youtube播放頁數據的前提是scrapy部署的機器可以正常訪問Youtube網站

示例網址

抓取的原理是讀取Youtube播放頁桌面版網頁源代碼中的全局變量： ytInitialData

存取到Mongo中的數據如下：

{
    "title" : "20130410 鏘鏘三人行 陳平原談中國教育問題",
    "view_count" : "12,407 views"
}

代碼如下：

# -*- coding: utf-8 -*-
import scrapy
import re
import json
from scrapy import Selector
from knowsmore.items import YoutubeItem
from ..common import *

class YoutubeSpider(scrapy.Spider):
    custom_settings = {
        'DOWNLOADER_MIDDLEWARES' : {
        }
    }

    name = 'youtube'
    allowed_domains = ['www.youtube.com']
    start_urls = ['https://www.youtube.com/watch?v=3vkqOdMBP48']

    def parse(self, response):
        ytInitialData = r1(r'ytInitialData"\] = (.*?)}};', response.body)

        if ytInitialData:
            ytInitialData = '%s}}' % ytInitialData
            ytInitialDataObj = json.loads(ytInitialData)
            
            videoInfo = ytInitialDataObj['contents']['twoColumnWatchNextResults']['results']['results']['contents'][0]['videoPrimaryInfoRenderer']
            Item = YoutubeItem(
                title = videoInfo['title']['simpleText'].encode('utf-8'),
                view_count = videoInfo['viewCount']['videoViewCountRenderer']['viewCount']['simpleText']
            )
            yield Item