在java中獲取一個網站的HTML內容可以通過HttpURLConnection來獲取.我們在HttpURLConnection中可以設置referer來僞造referer,輕鬆繞過這類防採集的網站
HttpURLConnection conn = (HttpURLConnection) newURL(path).openConnection();
conn.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible;MSIE 7.0; Windows NT 5.1; Maxthon;)");
conn.setRequestProperty("Accept-Encoding", "gzip");
conn.setRequestProperty("referer","http://control.blog.sina.com.cn/selienceblog");
conn.setRequestProperty("cookie","http://control.blog.sina.com.cn");
InputStream inputStream = conn.getInputStream();
//保存inputstream中的東西就OK了其實原理非常簡單就是設置下請求頭,通過設置user-agent讓服務器識別我們的身份,上面的例子中,我們就告訴瀏覽器我是用的maxthon遨遊瀏覽器.通過設置useragent大部分的防採集的網站可以通過了,第二中比較嚴格的防採集的網站是通過請求頭的referer和cookie來判斷的.使用jetty的HttpClient的可以通過ContentExchange的addRequestHeader來設置請求頭
contentExchange.addRequestHeader("User-Agent",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;Maxthon;)");
contentExchange.addRequestHeader("Accept-Encoding", "gzip");
contentExchange.addRequestHeader("referer","http://control.blog.sina.com.cn/selienceblog");
contentExchange.addRequestHeader("cookie", "data");
contentExchange.setMethod("GET");