HTTP報文基本結構，自己寫程序發送http包登陸某些網站。

          對http協議一直沒有深入的研究，前幾天寢室一哥們需要 登陸 某網站，批量查詢一些客戶的信息（大約有幾萬個客戶），因爲他嫌自己手動查詢太慢（當然他也比較懶），所以找我幫他寫一個程序，進行批量查詢。剛開始他給我說的時候，我只是簡單的說 叫他去用一用 按鍵精靈 這個軟件，能夠自動做重複的操作，於是 他聽我的 去下載了一個按鍵精靈 玩了一玩，結果發現滿足不了他的需求，並且他嫌按鍵精靈太慢了，太傻了。沒辦法 ，我只能給他寫代碼解決這個問題啦 ，思路很簡單就是自己寫程序與 某網站的服務器通信，然後寫一個循環重複提交自己的客戶信息，並解析返回的結果就ok了。由於對http不是非常熟悉，之前也沒有類似的開發經歷，於是我決定先看看http協議的基本結構，在網上看到一個圖，覺得不錯（具體鏈接地址：點擊打開鏈接），要是對http不是很瞭解的可參考點擊打開鏈接：

看完之後 ， 心裏有點感覺 ，於是開始在網上摸索，於是找到了這樣幾個包：commons-codec-1.6.jar，commons-httpclient-3.0.1.jar，commons-logging-1.1.3.jar，jsoup-1.8.1.jar（所有的包下載地址：點擊打開鏈接）

，前三個包是對http協議的封裝，jsoup包是解析網頁的包（jsoup中文參考資料：點擊打開鏈接），在網上找了一個demo，然後自己就開始動手了，首先 你需要 去訪你想要登陸的 網站去查看 其 登陸 過程，通過查看其 使用的是 get 還是 post 方法  ，當然一般的網站登陸都是使用（https+post），如果是使用https協議，你可能需要把 該網站上使用的證書下載下來（一會我會給出證書下載的代碼），不然你和其進行通信是會提示證書錯誤的。初次訪問某網站你能夠 得到一個session 值，當然這個session值需要你解析網頁獲取（一般在返回的頭部信息（set-Cookie屬性中會有）），有些網站可能還會給你返回一個token（令牌），不過這個值是不會變的，這個值也可以通過解析網頁獲取到。總之，你需要在你的瀏覽器中登陸一遍某網站，然後你根據你的瀏覽器中登陸的過程，你自己寫程序模擬瀏覽器的操作就ok啦，在登陸完成之後，你可能要保存一寫你自己的私有信息（對瀏覽器來說就是cookie），比如，用戶名，sessionid，token等，在之後的操作中，你只有給服務器傳入這些你的私有信息，瀏覽器才知道你已經登陸了（一般在你的請求報頭中的cookie信息中給出）。總結出來就幾點：

1. 如果網站使用https協議，把該網站的證書下載下來 ，並在你的程序中設置一下證書。

2. 在瀏覽器中模擬登陸過程，並查看每次點擊 之後 ，瀏覽器發送的報文信息。

3. 編寫程序模擬瀏覽器的操作。

下載證書代碼：在你的項目工程目錄下會出現文件

jssecacerts，即爲證書，下一個程序使用了此證書。

package com;

/*
 * Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *
 *   - Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *
 *   - Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *
 *   - Neither the name of Sun Microsystems nor the names of its
 *     contributors may be used to endorse or promote products derived
 *     from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
 * IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.security.KeyStore;
import java.security.MessageDigest;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;

import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLException;
import javax.net.ssl.SSLSocket;
import javax.net.ssl.SSLSocketFactory;
import javax.net.ssl.TrustManager;
import javax.net.ssl.TrustManagerFactory;
import javax.net.ssl.X509TrustManager;

public class InstallCert {

	public static void main(String[] args) throws Exception {
		String str[] = {"url"};//網址
		String host;
		int port;
		char[] passphrase;
		if ((str.length == 1) || (str.length == 2)) {
			String[] c = str[0].split(":");
			host = c[0];
			port = (c.length == 1) ? 443 : Integer.parseInt(c[1]);
			String p = (str.length == 1) ? "changeit" : str[1];
			passphrase = p.toCharArray();
		} else {
			System.out
					.println("Usage: java InstallCert <host>[:port] [passphrase]");
			return;
		}

		File file = new File("jssecacerts");
		if (file.isFile() == false) {
			char SEP = File.separatorChar;
			File dir = new File(System.getProperty("java.home") + SEP + "lib"
					+ SEP + "security");
			file = new File(dir, "jssecacerts");
			if (file.isFile() == false) {
				file = new File(dir, "cacerts");
			}
		}
		System.out.println("Loading KeyStore " + file + "...");
		InputStream in = new FileInputStream(file);
		KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
		ks.load(in, passphrase);
		in.close();

		SSLContext context = SSLContext.getInstance("TLS");
		TrustManagerFactory tmf = TrustManagerFactory
				.getInstance(TrustManagerFactory.getDefaultAlgorithm());
		tmf.init(ks);
		X509TrustManager defaultTrustManager = (X509TrustManager) tmf
				.getTrustManagers()[0];
		SavingTrustManager tm = new SavingTrustManager(defaultTrustManager);
		context.init(null, new TrustManager[] { tm }, null);
		SSLSocketFactory factory = context.getSocketFactory();

		System.out
				.println("Opening connection to " + host + ":" + port + "...");
		SSLSocket socket = (SSLSocket) factory.createSocket(host, port);
		socket.setSoTimeout(10000);
		try {
			System.out.println("Starting SSL handshake...");
			socket.startHandshake();
			socket.close();
			System.out.println();
			System.out.println("No errors, certificate is already trusted");
		} catch (SSLException e) {
			System.out.println();
			e.printStackTrace(System.out);
		}

		X509Certificate[] chain = tm.chain;
		if (chain == null) {
			System.out.println("Could not obtain server certificate chain");
			return;
		}

		BufferedReader reader = new BufferedReader(new InputStreamReader(
				System.in));

		System.out.println();
		System.out.println("Server sent " + chain.length + " certificate(s):");
		System.out.println();
		MessageDigest sha1 = MessageDigest.getInstance("SHA1");
		MessageDigest md5 = MessageDigest.getInstance("MD5");
		for (int i = 0; i < chain.length; i++) {
			X509Certificate cert = chain[i];
			System.out.println(" " + (i + 1) + " Subject "
					+ cert.getSubjectDN());
			System.out.println("   Issuer  " + cert.getIssuerDN());
			sha1.update(cert.getEncoded());
			System.out.println("   sha1    " + toHexString(sha1.digest()));
			md5.update(cert.getEncoded());
			System.out.println("   md5     " + toHexString(md5.digest()));
			System.out.println();
		}

		System.out
				.println("Enter certificate to add to trusted keystore or 'q' to quit: [1]");
		String line = reader.readLine().trim();
		int k;
		try {
			k = (line.length() == 0) ? 0 : Integer.parseInt(line) - 1;
		} catch (NumberFormatException e) {
			System.out.println("KeyStore not changed");
			return;
		}

		X509Certificate cert = chain[k];
		String alias = host + "-" + (k + 1);
		ks.setCertificateEntry(alias, cert);

		OutputStream out = new FileOutputStream("jssecacerts");
		ks.store(out, passphrase);
		out.close();

		System.out.println();
		System.out.println(cert);
		System.out.println();
		System.out
				.println("Added certificate to keystore 'jssecacerts' using alias '"
						+ alias + "'");
	}

	private static final char[] HEXDIGITS = "0123456789abcdef".toCharArray();

	private static String toHexString(byte[] bytes) {
		StringBuilder sb = new StringBuilder(bytes.length * 3);
		for (int b : bytes) {
			b &= 0xff;
			sb.append(HEXDIGITS[b >> 4]);
			sb.append(HEXDIGITS[b & 15]);
			sb.append(' ');
		}
		return sb.toString();
	}

	private static class SavingTrustManager implements X509TrustManager {

		private final X509TrustManager tm;
		private X509Certificate[] chain;

		SavingTrustManager(X509TrustManager tm) {
			this.tm = tm;
		}

		public X509Certificate[] getAcceptedIssuers() {
			throw new UnsupportedOperationException();
		}

		public void checkClientTrusted(X509Certificate[] chain, String authType)
				throws CertificateException {
			throw new UnsupportedOperationException();
		}

		public void checkServerTrusted(X509Certificate[] chain, String authType)
				throws CertificateException {
			this.chain = chain;
			tm.checkServerTrusted(chain, authType);
		}
	}

}

post協議實現代碼：

public void init(String url,int count)
	{
	System.setProperty("javax.net.ssl.trustStore", "jssecacerts"); //設置證書
        PostMethod postMethod = new PostMethod(url); 

<span style="white-space:pre">	</span>//這些信息根據你在瀏覽器中，模擬登陸是看到的報頭來設置
        postMethod.setRequestHeader("Accept", "text/html, application/xhtml+xml, */*");
        postMethod.setRequestHeader("Accept-Language", "zh-CN");
       // postMethod.setRequestHeader("Accept-Encoding", "gzip, deflate");

<span style="white-space:pre">	</span>//不設置<span style="font-family: Arial, Helvetica, sans-serif;">Accept-Encoding</span>的話，不需要對網頁進行gzip解壓，但是流量會大一些
       //postMethod.setRequestHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko");
        postMethod.setRequestHeader("Host", "192.168.1.1");
        postMethod.setRequestHeader("DNT", "1");
        postMethod.setRequestHeader("Connection", "Keep-Alive");
        postMethod.setRequestHeader("Cache-Control", "no-cache");
        int statusCode = 0;  
        try {  
            statusCode = httpClient.executeMethod(postMethod);  
        } catch (HttpException e) {  
            e.printStackTrace();  
        } catch (IOException e) {  
            e.printStackTrace();  
        } 
        if (statusCode == HttpStatus.SC_MOVED_PERMANENTLY  
                || statusCode == HttpStatus.SC_MOVED_TEMPORARILY) {  
            // 從頭中取出轉向的地址  
            Header locationHeader = postMethod.getResponseHeader("location");  
            String location = null;  
            if (locationHeader != null) {  
                location = locationHeader.getValue();  
                System.out.println("The page was redirected to:" + location);
                if ( count <= 10 )
                {
                	init(location,count+1);//用跳轉後的頁面重新請求。
                }
                else
                {
                	return;
                }
            } else {  
                System.err.println("Location field value is null.");  
            }  
        } else {  
            try {  
            	token = ParsePage.getToken(postMethod.getResponseBodyAsString());
            	if ( token == null || token.length() == 0 )
            	{
            		if ( count <= 10 )
            			init(url,count+1);//重新初始化，獲取token
            		return;
            	}
            	System.out.println("token : " + token);
            	if ( postMethod.getResponseHeader("Set-Cookie") != null && 
            			postMethod.getResponseHeader("Set-Cookie").getValue().length() > 0)
            	{
            		String temp = postMethod.getResponseHeader("Set-Cookie").getValue();
            		jsessionid = temp.substring(temp.indexOf('=') + 1,temp.indexOf(';'));//獲取session id
                	System.out.println("jsessionid : " + jsessionid);
            		//System.out.println(temp + "   " + jssesionid);
            	}
            	else
            	{
            		if ( count <= 10 )
            			init(url,count+1);//重新初始化，獲取session id
            		return;
            	}
            } catch (IOException e) {  
                e.printStackTrace();  
            }  
            //postMethod.releaseConnection();  
        }  
	}

因爲我寫的程序涉及到一些隱私，就不全部給出了，關於其他的使用方式，讀者自己摸索吧，如果有什麼問題我希望能與各位讀者一起探討。