標註環境CVAT在Ubuntu18.04下搭建

CVAT Ubuntu 18.04 標註環境搭建

記錄一次在Ubuntu 18.04下搭建CVAT 環境的過程，將會在本月內再次搭建一次

環境準備

1. Ubuntu環境搭建
2. Anaconda 下載
3. CVAT 下載
4. 換源路徑

Ubuntu環境搭建

傳送門鏈接: link.
U盤啓動製作工具: link.

換源路徑

推薦Ubuntu清華源 link.

# 默認註釋了源碼鏡像以提高 apt update 速度，如有需要可自行取消註釋
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse

# 預發佈軟件源，不建議啓用
# deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse

推薦Anaconda清華源 link.

conda config --set show_channel_urls yes
nano ~/.condarc

# 在.condarc中貼入如下源信息

channels:
  - defaults
show_channel_urls: true
default_channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
custom_channels:
  conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud

# 在.condarc追加如下超時時間

remote_connect_timeout_secs: 40.0
remote_read_timeout_secs: 100.0

Anaconda 下載安裝

下載對應的版本 ，路徑來源（清華源）
Anaconda3-2019.10-Linux-x86_64.sh .
Anaconda3-2019.10-Windows-x86_64.exe .

CVAT 下載

git clone https://github.com/opencv/cvat

基礎環境配置

Ubuntu Docker 環境配置

# 摘自 https://github.com/opencv/cvat/blob/develop/cvat/apps/documentation/installation.md

sudo apt-get update
sudo apt-get install -y \
  apt-transport-https \
  ca-certificates \
  curl \
  gnupg-agent \
  software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io


# docker進程使用Unix Socket而不是TCP端口。
# 而默認情況下，Unix socket屬於root用戶，需要root權限才能訪問。
# add user in docker group
sudo groupadd docker     #添加docker用戶組
sudo gpasswd -a $USER docker     #將登陸用戶加入到docker用戶組中
newgrp docker     #更新用戶組
docker ps    #測試docker命令是否可以使用sudo正常使用

# install docker-compose
pip install docker-compose -i https://pypi.tuna.tsinghua.edu.cn/simple

★ CVAT Dockerfile修改

原生的Dockerfile會自動去官方源去做更新，因爲國內網絡問題，所以要換源
步驟如下

1. 在CVAT目錄下，放入source.list
   
   source內容是清華源，對應Ubuntu版本

2. 修改如下位置的Dockerfile
   
   

3. 變動後結果如下

FROM ubuntu:16.04

ARG http_proxy
ARG https_proxy
ARG no_proxy
ARG socks_proxy

ENV TERM=xterm \
    http_proxy=${http_proxy}   \
    https_proxy=${https_proxy} \
    no_proxy=${no_proxy} \
    socks_proxy=${socks_proxy}

ENV LANG='C.UTF-8'  \
    LC_ALL='C.UTF-8'

ARG USER
ARG DJANGO_CONFIGURATION
ENV DJANGO_CONFIGURATION=${DJANGO_CONFIGURATION}

# Install necessary apt packages
RUN apt-get update && \
    apt-get install -yq \
        python-software-properties \
        software-properties-common \
        wget && \
    add-apt-repository ppa:mc3man/xerus-media -y && \
    add-apt-repository ppa:mc3man/gstffmpeg-keep -y && \
    apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get install -yq \
        apache2 \
        apache2-dev \
        libapache2-mod-xsendfile \
        supervisor \
        ffmpeg \
        gstreamer0.10-ffmpeg \
        libldap2-dev \
        libsasl2-dev \
        python3-dev \
        python3-pip \
        unzip \
        unrar \
        p7zip-full \
        vim && \
    add-apt-repository --remove ppa:mc3man/gstffmpeg-keep -y && \
    add-apt-repository --remove ppa:mc3man/xerus-media -y && \
    rm -rf /var/lib/apt/lists/*

# Add a non-root user
ENV USER=${USER}
ENV HOME /home/${USER}
WORKDIR ${HOME}
RUN adduser --shell /bin/bash --disabled-password --gecos "" ${USER}

COPY components /tmp/components

# OpenVINO toolkit support
ARG OPENVINO_TOOLKIT
ENV OPENVINO_TOOLKIT=${OPENVINO_TOOLKIT}
RUN if [ "$OPENVINO_TOOLKIT" = "yes" ]; then \
        /tmp/components/openvino/install.sh; \
    fi

# CUDA support
ARG CUDA_SUPPORT
ENV CUDA_SUPPORT=${CUDA_SUPPORT}
RUN if [ "$CUDA_SUPPORT" = "yes" ]; then \
        /tmp/components/cuda/install.sh; \
    fi

# Tensorflow annotation support
ARG TF_ANNOTATION
ENV TF_ANNOTATION=${TF_ANNOTATION}
ENV TF_ANNOTATION_MODEL_PATH=${HOME}/rcnn/inference_graph
RUN if [ "$TF_ANNOTATION" = "yes" ]; then \
        bash -i /tmp/components/tf_annotation/install.sh; \
    fi

ARG WITH_TESTS
RUN if [ "$WITH_TESTS" = "yes" ]; then \
        wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \
        echo 'deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main' | tee /etc/apt/sources.list.d/google-chrome.list && \
        wget -qO- https://deb.nodesource.com/setup_9.x | bash - && \
        apt-get update && \
        DEBIAN_FRONTEND=noninteractive apt-get install -yq \
            google-chrome-stable \
            nodejs && \
        rm -rf /var/lib/apt/lists/*; \
        mkdir tests && cd tests && npm install \
            eslint \
            eslint-detailed-reporter \
            karma \
            karma-chrome-launcher \
            karma-coveralls \
            karma-coverage \
            karma-junit-reporter \
            karma-qunit \
            qunit; \
        echo "export PATH=~/tests/node_modules/.bin:${PATH}" >> ~/.bashrc; \
    fi

# Install and initialize CVAT, copy all necessary files
COPY cvat/requirements/ /tmp/requirements/
COPY supervisord.conf mod_wsgi.conf wait-for-it.sh manage.py ${HOME}/
RUN pip3 install --no-cache-dir -r /tmp/requirements/${DJANGO_CONFIGURATION}.txt  -i https://pypi.tuna.tsinghua.edu.cn/simple

# Install git application dependencies
RUN apt-get update
RUN apt-get install apt-transport-https
ADD sources.list /etc/apt/
RUN apt-get update && \
    apt-get install -y ssh netcat-openbsd git curl zip  && \
#    wget -qO /dev/stdout https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash && \
    apt-get install -y git-lfs && \
    git lfs install && \
    rm -rf /var/lib/apt/lists/* && \
    if [ -z ${socks_proxy} ]; then \
        echo export "GIT_SSH_COMMAND=\"ssh -o StrictHostKeyChecking=no -o ConnectTimeout=30\"" >> ${HOME}/.bashrc; \
    else \
        echo export "GIT_SSH_COMMAND=\"ssh -o StrictHostKeyChecking=no -o ConnectTimeout=30 -o ProxyCommand='nc -X 5 -x ${socks_proxy} %h %p'\"" >> ${HOME}/.bashrc; \
    fi

# Download model for re-identification app
ENV REID_MODEL_DIR=${HOME}/reid
RUN if [ "$OPENVINO_TOOLKIT" = "yes" ]; then \
        mkdir ${HOME}/reid && \
        wget https://download.01.org/openvinotoolkit/2018_R5/open_model_zoo/person-reidentification-retail-0079/FP32/person-reidentification-retail-0079.xml -O reid/reid.xml && \
        wget https://download.01.org/openvinotoolkit/2018_R5/open_model_zoo/person-reidentification-retail-0079/FP32/person-reidentification-retail-0079.bin -O reid/reid.bin; \
    fi

# TODO: CHANGE URL
ARG WITH_DEXTR
ENV WITH_DEXTR=${WITH_DEXTR}
ENV DEXTR_MODEL_DIR=${HOME}/dextr
RUN if [ "$WITH_DEXTR" = "yes" ]; then \
        mkdir ${DEXTR_MODEL_DIR} -p && \
        wget https://download.01.org/openvinotoolkit/models_contrib/cvat/dextr_model_v1.zip -O ${DEXTR_MODEL_DIR}/dextr.zip && \
        unzip ${DEXTR_MODEL_DIR}/dextr.zip -d ${DEXTR_MODEL_DIR} && rm ${DEXTR_MODEL_DIR}/dextr.zip; \
    fi

COPY ssh ${HOME}/.ssh
COPY cvat/ ${HOME}/cvat
COPY cvatjs/ ${HOME}/cvatjs
COPY tests ${HOME}/tests
# Binary option is necessary to correctly apply the patch on Windows platform.
# https://unix.stackexchange.com/questions/239364/how-to-fix-hunk-1-failed-at-1-different-line-endings-message
RUN patch --binary -p1 < ${HOME}/cvat/apps/engine/static/engine/js/3rdparty.patch
RUN chown -R ${USER}:${USER} .

# RUN all commands below as 'django' user
USER ${USER}

RUN mkdir data share media keys logs /tmp/supervisord
RUN python3 manage.py collectstatic

EXPOSE 8080 8443
ENTRYPOINT ["/usr/bin/supervisord"]

★ CVAT docker-compose.yml修改

1. 變動如下

# cvat ports 下增加
    ports:
      - "0.0.0.0:9000:8080"
# 使外網可以訪問 例如192.168.1.100:9000

# cvat environment 下增加
    environment:
      DJANGO_MODWSGI_EXTRA_ARGS: ""
      ALLOWED_HOSTS: "*"
# 使外網可以訪問

2. 變動後結果如下

#
# Copyright (C) 2018 Intel Corporation
#
# SPDX-License-Identifier: MIT
#
version: "2.3"

services:
  cvat_db:
    container_name: cvat_db
    image: postgres:10.3-alpine
    networks:
      default:
        aliases:
          - db
    restart: always
    environment:
      POSTGRES_USER: root
      POSTGRES_DB: cvat
    volumes:
      - cvat_db:/var/lib/postgresql/data

  cvat_redis:
    container_name: cvat_redis
    image: redis:4.0.5-alpine
    networks:
      default:
        aliases:
          - redis
    restart: always

  cvat:
    container_name: cvat
    image: cvat
    restart: always
    depends_on:
      - cvat_redis
      - cvat_db
    ports:
      - "0.0.0.0:9000:8080"
    build:
      context: .
      args:
        http_proxy:
        https_proxy:
        no_proxy:
        socks_proxy:
        TF_ANNOTATION: "no"
        USER: "django"
        DJANGO_CONFIGURATION: "production"
        WITH_TESTS: "no"
    environment:
      DJANGO_MODWSGI_EXTRA_ARGS: ""
      ALLOWED_HOSTS: "*"
    volumes:
      - /cvat/data:/home/ai/cvatdata/data
      - /cvat/keys:/home/ai/cvatdata/keys
      - /cvat/logs:/home/ai/cvatdata/logs
      - /cvat/Users:/home/ai/cvatdata/models

volumes:
  cvat_db:
  cvat_data:
  cvat_keys:
  cvat_logs:
  cvat_models:

耐心等待與使用

docker-compose build

安裝過程中，不可避免有些遺漏的地方也是用官方源去下載的，不過數量較少，所以也沒有完全去把所有的源都換了。
在網絡限制在400kb/s時，docker-compose build一次大概45min

可能遇到的問題

1. 創建超級用戶失敗

docker exec -it cvat bash -ic 'python3 ~/manage.py createsuperuser'

錯誤信息是：LINE 1: ...user"."is_active", "auth_user"."date_joined" FROM "auth_user...

處理辦法

# 1. 進入到docker鏡像中
docker exec -it cvat /bin/bash
# 2. python3 ~/manage.py createsuperuser
# 再運行一次，看看具體的錯誤信息，一般情況下，是要進行一次migrate
python3 ~/manage.py migrate
# 3. 然後再次運行創建超級用戶的命令
python3 ~/manage.py createsuperuser

2. 上述操作之後，依舊沒法訪問設定的9000端口

處理辦法1

# 1. 還是進入到docker鏡像中
docker exec -it cvat /bin/bash
# 2. 啓動django
python3 manage.py runserver 0.0.0.0:8080
然後按住ctrl + p + q 退出當前docker exec 環境

3. 20191210 增加因PC突然斷電，然後無法啓動的處理辦法

# 不要用 -d
docker-compose up
# 此時注意看打印的日誌，可能會多如下日誌
cvat          | 2019-12-10 10:23:33,918 DEBG 'runserver' stderr output:
cvat          | [Tue Dec 10 10:23:33.918640 2019] [core:warn] [pid 18:tid 140294961167296] AH00098: pid file /tmp/mod_wsgi-localhost:8080:1000/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
# 怎麼辦？
# 0 照常啓動 不後臺
docker-compose up
# 1 還是進入到docker裏
docker exec -it cvat /bin/bash
# 2 刪掉或備份 mod_wsgi_localhost:8080:1000
cd /tmp
rm -rf mod_wsgi-localhost\:8080\:1000/
# 3 停止docker-compose
ctrl c
# 4 重啓
docker-compose up -d

4. 啓動後頁面展示不全，日誌展示

[2020-05-27 12:53:59,190] WARNING django.request: Not Found: /static/CACHE/js/239c24a082e8.js
[2020-05-27 12:53:59,195] WARNING django.request: Not Found: /static/CACHE/js/d91026595cdb.js
[2020-05-27 12:53:59,197] WARNING django.server: "GET /static/CACHE/js/239c24a082e8.js HTTP/1.1" 404 77
[2020-05-27 12:53:59,197] WARNING django.server: "GET /static/CACHE/js/d91026595cdb.js HTTP/1.1" 404 77

原因未知。。。
解決辦法：
回到docker-compose.yml所在的目錄，docker-compose restart

創建超級用戶

方法有兩種

1. 直接用bash

docker exec -it cvat bash -ic 'python3 ~/manage.py createsuperuser'

2. 進入到鏡像內運行

# 1. 進入到docker鏡像中
docker exec -it cvat /bin/bash
# 2. 運行命令
python3 ~/manage.py createsuperuser

創建普通用戶

1. 打開admin網頁

http://192.168.1.100:9000/admin/

可以探索的使用方法

1. 與git集成
2. 自動創建標註任務
3. 自動標註

！對了！在chrome下使用這個CVAT