httpClient、jsoup、okhttp、junitHtml抓取頁面方法介紹(未完成)

1.本文宗旨

記錄日常所用並與網友分享

2.httpClient、jsoup、okhttp、junitHtml能幹什麼

用來發送或者接受http請求，或者說能夠抓取網頁爬去信息，就是java版的爬蟲.

3.httpClient、okhttp、restTemplate收發請求

3.1httpClient的get請求

  /**
  * httpClient的get請求介紹
  */
  @Test
    public void getHtmlPost(){
        // 創建默認的httpClient實例.
        String url = "https://search.17k.com/search.xhtml";
        Map<String,Object> map = new HashMap<>();
        map.put("c.st",0);
        map.put("c.q","近戰狂兵");
        url = setDoGetUrl(url,map);
        RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(120000).setSocketTimeout(60000)
                .setConnectionRequestTimeout(60000).build();

        CloseableHttpClient httpclient;
        HttpClientContext httpClientContext = HttpClientContext.create();
        CookieStore cookieStore = null;
        cookieStore = new BasicCookieStore();
        httpclient = HttpClientBuilder.create().setKeepAliveStrategy(new DefaultConnectionKeepAliveStrategy())
                .setRedirectStrategy(new DefaultRedirectStrategy()).setDefaultRequestConfig(requestConfig)
                .setDefaultCookieStore(cookieStore).build();
        Function<Object,Object> function=null;
        String resGet = doGet(url,map,httpclient,httpClientContext);

    }
	
    private String doGet(String url) {
        return doGet(url,null,null,null);
    }
    private String doGet(String url,CloseableHttpClient httpclient, HttpClientContext httpClientContext) {
        return doGet(url,null,httpclient,httpClientContext);
    }
	/*
	* get請求方法封裝
	*/
    private String doGet(String url, Map map, CloseableHttpClient httpclient, HttpClientContext httpClientContext) {
        if(httpclient==null) httpclient = HttpClients.createDefault();
        if(httpClientContext==null)httpClientContext = HttpClientContext.create();
        setDoGetUrl(url,map);
        CookieStore cookieStore223 = httpClientContext.getCookieStore();
        if(cookieStore223!=null){
            System.out.println("===============================什麼這裏有cookie====================");
            List<Cookie> listCookie = cookieStore223.getCookies();
            System.out.println(listCookie);
        }
        String web="";
        HttpGet httpget = new HttpGet(url);
        CloseableHttpResponse response = null;
        httpget.setHeader("Connection", "close");//請求頭設置
        httpget.setHeader("Content-Encoding", "gzip");
        httpget.setHeader("Server", "openresty");
        httpget.setHeader("Transfer-Encoding", "chunked");
        httpget.setHeader("Vary", "Accept-Encoding");
        httpget.setHeader("Content-Type","text/html;charset=UTF-8");
        try {
        //這一步很重要，如果只傳一個參數就是普通的get請求，如果是帶了httpClientContext上下文沒那麼可以將多次請求維持在同一session裏面，前提是別關閉httpClient
            response = httpclient.execute(httpget,httpClientContext);
            HttpEntity entity = response.getEntity();
            if (entity != null) {
                //獲取請求頭，根據請求頭進行不同的處理
                String contentType = entity.getContentType().getValue();
                web = EntityUtils.toString(entity,"utf-8");

                httpClientContext.getCookieStore().getCookies().forEach(System.out::println);
            }
            response.close();
            httpclient.close();
        }catch (Exception e) {
            e.printStackTrace();
        } finally {
            //關閉連接,釋放資源
            try {
                response.close();
                httpclient.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        return web;
    }
    
	/*
	*	get請求參數封裝，手寫
	*/
    private static String setDoGetUrl(String url, Map<String,Object> map) {
        if(StringUtils.isBlank(url))return url;
        if(MapUtils.isEmpty(map))return url;
        if(!url.endsWith("?")) url += "?";
        for (Map.Entry<String,Object> entry:map.entrySet()) {
            if(StringUtils.isBlank(entry.getKey()))continue;
            if(!url.endsWith("?")){
                url += "&" + entry.getKey() + "=" + entry.getValue().toString();
            }else{
                url += entry.getKey() + "=" + entry.getValue().toString();
            }
        }
        return url;
    }

3.2
restTemplate的get請求

restTemplate並沒有重寫底層的HTTP請求技術，而是提供配置，可選用OkHttp/HttpClient等，說白了就這對這兩個封裝。

這個就很簡單了，