selemium環境準備

查找對應版本的chrome driver: http://chromedriver.chromium.org/downloads
解決DevToolsActivePort file doesn't exist: https://cloud.tencent.com/developer/article/1404558

測試代碼:
    public static void main(String[] args) {
        // System.setProperty("webdriver.chrome.driver", "/Users/junhao.zhang/Downloads/chromedriver");
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--no-sandbox");
        options.addArguments("--disable-dev-shm-usage");
        options.setHeadless(true);
        Proxy proxy = new Proxy();
        proxy.setHttpProxy("proxy.com:8080");
        options.setProxy(proxy);
        ChromeDriver driver = new ChromeDriver(options);
        // driver.get("https://selenium.dev");
        driver.get("http://www.baidu.com");
        // driver.get("http://sports.sina.com.cn");
        System.out.println("Title: " + driver.getTitle());
        driver.quit();  
    }

對應的Dockerfile:
FROM registry-hulk.xxx.com/xxx/centos:offline_java_191206_centos7
MAINTAINER xxx.yyy "[email protected]"
RUN yum install -y google-chrome-stable
RUN /bin/wget https://chromedriver.storage.googleapis.com/73.0.3683.68/chromedriver_linux64.zip -O /usr/local/bin/chromedriver_linux64.zip
RUN cd /usr/local/bin/ && /bin/unzip chromedriver_linux64.zip && rm chromedriver_linux64.zip
ENV PATH /usr/local/bin:$PATH
 

selemium不適合做全網爬蟲,只適合做app內h5/native以及特定web站點的抓取,因爲它無法優雅地獲取瀏覽器內部狀態,比如http response code,比如網絡連接是否有問題.