selemium环境准备

查找对应版本的chrome driver: http://chromedriver.chromium.org/downloads
解决DevToolsActivePort file doesn't exist: https://cloud.tencent.com/developer/article/1404558

测试代码:
    public static void main(String[] args) {
        // System.setProperty("webdriver.chrome.driver", "/Users/junhao.zhang/Downloads/chromedriver");
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--no-sandbox");
        options.addArguments("--disable-dev-shm-usage");
        options.setHeadless(true);
        Proxy proxy = new Proxy();
        proxy.setHttpProxy("proxy.com:8080");
        options.setProxy(proxy);
        ChromeDriver driver = new ChromeDriver(options);
        // driver.get("https://selenium.dev");
        driver.get("http://www.baidu.com");
        // driver.get("http://sports.sina.com.cn");
        System.out.println("Title: " + driver.getTitle());
        driver.quit();  
    }

对应的Dockerfile:
FROM registry-hulk.xxx.com/xxx/centos:offline_java_191206_centos7
MAINTAINER xxx.yyy "[email protected]"
RUN yum install -y google-chrome-stable
RUN /bin/wget https://chromedriver.storage.googleapis.com/73.0.3683.68/chromedriver_linux64.zip -O /usr/local/bin/chromedriver_linux64.zip
RUN cd /usr/local/bin/ && /bin/unzip chromedriver_linux64.zip && rm chromedriver_linux64.zip
ENV PATH /usr/local/bin:$PATH
 

selemium不适合做全网爬虫,只适合做app内h5/native以及特定web站点的抓取,因为它无法优雅地获取浏览器内部状态,比如http response code,比如网络连接是否有问题.