爬蟲---基礎1

最基本的爬蟲抓取
requests:
import requests   # python3
response = requests.get("http://wwww.baidu.com")
content = response.content
print("headers:", response.headers)
print("content:", content)


urllib2:
import urllib2  # python2
url = "http://wwww.baidu.com"
req = urllib2.urlopen(url)
content = req.read()
print "headers:", req.headers
print "content:", content


httplib2:
import httplib2   # python3
url = "http://wwww.baidu.com"
http = httplib2.Http()
response_headers, content = http.request(url, 'GET')
print("headers:", response_headers)

print("content:", content)

此外，對於帶有查詢字段的url,get請求一般會將請求的數據附在url之後，以？分割url和傳輸數據，多個參數用&連接。

requests中data爲dict，json：

    data = {‘data1’：‘xxxx’，‘data2’：‘xxxxx’}

    response = requests.get(url = url,params = data)

urllib2中data爲string：

    data = urllib.urlencode(data)

    full_url = url +'?' + data

    req = urllib2.urlopen(full_url)