Docker開發環境

開發環境圖示：
Centos防火牆
CentOS7查看端口開放情況：firewall-cmd --list-ports
CentOS7開放端口：firewall-cmd --zone=public --add-port=80/tcp --permanent
CentOS7關閉端口：firewall-cmd --remove-port=80/tcp --permanent
配置Pycharm連接Docker：
在/etc/docker/daemon.json中寫入hosts配置
```
{
  "registry-mirrors": ["https://fo7bca6n.mirror.aliyuncs.com"],
  "insecure-registries": ["192.168.33.6:8000", "192.168.33.6:9000"],
  "hosts": [
    "tcp://0.0.0.0:2375",
    "unix:///var/run/docker.sock"
  ]
}
```
▲ 啓動異常查看詳細日誌：journalctl -amu docker
報錯：

unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: hosts

解決：查看docker啓動腳本/usr/lib/systemd/system/docker.service中的啓動語句

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

這裏有指定-H參數，即–host參數，將-H fd://刪除重新啓動即可：
systemctl daemon-reload
systemctl restart docker
Pycharm中配置：View - Tool Windows - Docker配置中選擇TCP socket -> URL填tcp://192.168.33.6:2375
注意：配置完以後要修改傳到服務器的路徑需要添加mappings
Dockerfile配置：創建目錄 -> 新建Dockerfile -> 寫入內容右擊文件執行 -> 生成無命名的鏡像 -> 編輯Image tag再執行一次可對鏡像進行命名
指定運行程序的鏡像：File -> Settings -> Project:xxx - Interpreter- Show All -> 選中對應docker環境進行編輯Image name
docker-compose配置：Pycharm可以編輯docker-compose，但是只能在服務器中執行
啓動容器的參數配置： docker-library - 搜索對應鏡像進入查看
在pycharm中容器啓動配置頁中Environment variables添加

飯粒1：Docker中selenium與chrome爬蟲開發環境搭建與使用

拉取selenium鏡像並啓動容器：docker-selenium
docker pull selenium/standalone-chrome:3.141.59-titanium
docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome:3.141.59-titanium

編寫Dockerfile並執行構建

FROM python:latest
# 安裝支持http/https協議客戶端請求的庫
RUN pip install requests && \
    pip install aiohttp && \
    pip install tornado && \
    pip install selenium && \
    pip install appium-python-client

# 安裝gcc編譯環境
RUN yum install -y gcc gcc-c++

# 下列模塊的安裝需要gcc編譯環境
RUN pip install twisted && \
    pip install gevent

# 安裝scrapy
RUN yum install -y libffi-devel openssl-devel libxml2 libxslt-devel libxml2-devel python-devel && \
RUN pip install scrapy

# 安裝pyspider
RUN yum install -y curl libcurl4-openssl-dev libxml2-dev libxslt-dev zlib && \
RUN pip install pyspider

測試安裝的環境
command_executor中IP使用第1步啓動的容器IP

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

driver = webdriver.Remote(
    command_executor="http://172.17.0.2:4444/wd/hub",
    desired_capabilities=DesiredCapabilities.CHROME
)

driver.get("http://www.baidu.com")
print(driver.title)
print(driver.page_source)
driver.close()

進階：
baidu_spider.py

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import requests


def test_chrome_server():
    while True:
        i = 0
        try:
            res = requests.get("http://chrome:4444/wd/hub/status", timeout=0.3)
        except Exception as e:
            i += 1
            if i > 10:
                raise
        else:
            print(res.status_code)
            break


test_chrome_server()

driver = webdriver.Remote(
    command_executor="http://chrome:4444/wd/hub",
    desired_capabilities=DesiredCapabilities.CHROME
)

driver.get("http://www.baidu.com")
print(driver.title)
with open('/data/baidu.html', 'w') as f:
    f.write(driver.page_source)
driver.close()

docker-compose.yml

version: "3"
services:
  spider:
    image: spider-dev
    volumes:
      - ./baidu_spider.py:/code/baidu_spider.py
      - ./data:/data
    command: python /code/baidu_spider.py
    depends_on:
      - chrome
  chrome:
    image: selenium/standalone-chrome:3.141.59-titanium
    ports:
      - "4444:4444"
    shm_size: 2g

配置docker連接手機 appium

拉取鏡像並啓動容器

docker run --privileged -d -p 4723:4723 -v ~/.android:/root/.android -v /dev/bus/usb:/dev/bus/usb --name container-appium appium/appium

加上-v ~/.android:/root/.android可以避免重複授權

數據線連接手機，進行授權

docker exec -it container-appium adb devices

嘗試操作app
查找appPackage和appActivity：執行命令docker exec -it container-appium adb shell進入shell環境，執行logcat | grep ActivityManager後點擊APP查看cmp內容 \ 之前即爲appPackage，之後爲appActivity

import time
from appium import webdriver

desired_caps = {}
desired_caps["platformName"] = "Android"
desired_caps["platformVersion"] = "8.0.0"
desired_caps["deviceName"] = "Honor V10"

desired_caps["appPackage"] = "com.ted.android"
desired_caps["appActivity"] = ".view.splash.SplashActivity"

print("嘗試連接")
driver = webdriver.Remote(
    command_executor="http://172.17.0.2:4723/wd/hub",
    desired_capabilities=desired_caps
)
print("連接成功")
time.sleep(3)
driver.find_element_by_android_uiautomator('text(\"熱門\")').click()
driver.quit()

進階：
appium_android.py

import time
import requests
from appium import webdriver

desired_caps = {}
desired_caps["platformName"] = "Android"
desired_caps["platformVersion"] = "8.0.0"
desired_caps["deviceName"] = "Honor V10"

desired_caps["appPackage"] = "com.ted.android"
desired_caps["appActivity"] = ".view.splash.SplashActivity"


def test_chrome_server():
    while True:
        i = 0
        try:
            res = requests.get("http://appium:4723/wd/hub/status", timeout=0.3)
        except Exception as e:
            i += 1
            if i > 10:
                raise
        else:
            print(res.status_code)
            break


test_chrome_server()

print("嘗試連接")
driver = webdriver.Remote(
    command_executor="http://appium:4723/wd/hub",
    desired_capabilities=desired_caps
)
print("連接成功")
time.sleep(3)
driver.find_element_by_android_uiautomator('text(\"熱門\")').click()
driver.quit()

docker-compose.yml

version: "3"
services:
  spider:
    image: spider-dev
    volumes:
      - ./appium_android.py:/code/appium_android.py
    command: python /code/appium_android.py
    depends_on:
      - appium
  appium:
    image: appium/appium
    ports:
      - "4723:4723"
    privileged: true
    volumes:
      - ~/.android:/root/.android
      - /dev/bus/usb:/dev/bus/usb

Docker開發環境

Prometheus+Grafana監控系統

爬蟲初體驗

Kubernetes初識

scrapy初體驗

celery初識

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結