Crawl AJAX dynamic web page using Python 2.x and 3.x

The term AJAX is short for Asynchronous Javascript and XML. It uses the Javascript XMLHttpRequest function to create a tunnel between the client's browser and the server to transmit information back and forth without having to refresh the page. 

To crawl the contents created by AJAX, sometimes it's easy to identify the URL requested by the AJAX directly. Take the IE 11 as an example. First, press F12 and enter the developer tools mode. Select the "Network" tab, click the button to trigger the XMLHttpRequest, notice the URL tab and find out the URL links caused by the AJAX.

However, sometimes we cannot identify the URL caused by XMLHttpRequest directly. In this case, we have to build up the URL Request manually. 

1. identify the URL with the POST protocol.

2. double click the above URL and copy the value of "User-Agent"

3. select the Request body tab and copy the values.

4. the python code:

Python 2.x

import urllib2
import urllib
import json

url = 'http://www.huxiu.com/v2_action/article_list'
user_agent = 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0)'
data = {'huxiu_hash_code' : '63b69ec3342ee8c7e6ec4cab561482c9', 'page':2, 'last_dateline':1466664240}
data = urllib.urlencode(data)

request = urllib2.Request(url=url,data=data)
response = urllib2.urlopen(request)

result = json.loads(response.read())
print result

Python 3.x

import urllib
import json

url = 'http://www.huxiu.com/v2_action/article_list'
user_agent = 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0)'
data = {'huxiu_hash_code' : '63b69ec3342ee8c7e6ec4cab561482c9', 'page':2, 'last_dateline':1466664240}
data = (urllib.parse.urlencode(data)).encode('utf-8')
response = urllib.request.urlopen(url, data)

#parse json
result = json.loads(response.read().decode('utf-8'))
print (response)
print (result)