docker，consul，mesos集羣日常維護

docker啓動失敗
root@jjh35:/home/qboxserver/boots-docker/log# cat docker.log-0102144556
time="2019-01-02T14:45:56.182146778+08:00" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING
[!]"
time="2019-01-02T14:45:56.185443427+08:00" level=info msg="libcontainerd: new containerd process, pid: 24887"
time="2019-01-02T14:45:57.228799012+08:00" level=error msg="[graphdriver] prior storage driver \"aufs\" failed: driver not supported"
time="2019-01-02T14:45:57.228909333+08:00" level=fatal msg="Error starting daemon: error initializing graphdriver: driver not supported"

解決辦法：刪除原來docker目錄，啓動自動重建
cd /disk1
mv docker /tmp/

然後啓動即可
root@jjh630:/disk1# supervisorctl start dockerd
dockerd: started

dockerd啓動失敗，報錯日誌：
time="2019-03-01T14:52:09.091818748+08:00" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
time="2019-03-01T14:52:09.093160163+08:00" level=info msg="libcontainerd: new containerd process, pid: 52587"
time="2019-03-01T14:52:10.164273335+08:00" level=fatal msg="Error starting daemon: error initializing graphdriver: \"/disk1/docker\" contains several valid graphdrivers: aufs, overlay2; Please cleanup or explicitly choose storage driver (-s <DRIVER>)"
解決辦法：
root@cd-dora-jp-6:/disk1# cd /disk1
root@cd-dora-jp-6:/disk1# rm -rf docker

刪除docker目錄失敗，報錯：
rm: cannot remove 'docker/aufs': Device or resource busy
文件佔用，卸載掛載點：
root@cd-dora-jp-6:/disk1# cat /proc/mounts|grep docker
/dev/sdc1 /disk1/docker/overlay2 ext4 rw,noatime,data=ordered 0 0
/dev/sdc1 /disk1/docker/aufs ext4 rw,noatime,data=ordered 0 0

root@cd-dora-jp-6:/disk1# cat /proc/mounts|grep docker |awk '{print $2}'|while read line;do umount -l $line;done
root@cd-dora-jp-6:/disk1# df -h | egrep 'shm|overlay' | grep -v tmpfs | awk '{print $6}' | xargs umount

然後刪除docker目錄：
root@cd-dora-jp-6:/disk1# rm -rf docker
再啓動dockerd：
root@cd-dora-jp-6:/disk1# supervisorctl start dockerd
dockerd: started

consul啓動失敗：
root@cd-dora-jp-6:/home/qboxserver/consul/log# cat consul.log-0301154443
==> Starting Consul agent...
==> Error starting agent: Failed to start Consul client: Failed to start lan serf: Failed to create memberlist: Failed to start TCP listener. Err: listen tcp 10.24.34.241:8301: bind: cannot assign requested address
解決辦法：bind_addr 改爲本機內網 ip地址
vim /home/qboxserver/consul/current/conf/consul.conf
"bind_addr": "192.168.1.68",

mesos啓動失敗：ip改爲本機ip地址
cd /home/qboxserver/mesos-agent/current/conf/mesos-agent
vim ip
192.168.1.68
保存退出啓動

mesos清理髒數據方法：
root@xsgpu92:/disk1/mesos# curl -vX PUT "localhost:8500/v1/agent/service/deregister/boots:image.0c9dc975-f7a9-11e8-b21d-00e0ed50e991.1543919320179542264:HTTP"* Trying 127.0.0.1...

• Connected to localhost (127.0.0.1) port 8500 (#0)

  PUT /v1/agent/service/deregister/boots:image.0c9dc975-f7a9-11e8-b21d-00e0ed50e991.1543919320179542264:HTTP HTTP/1.1
  Host: localhost:8500
  User-Agent: curl/7.47.0
  Accept: /

  < HTTP/1.1 200 OK
  < Date: Wed, 02 Jan 2019 08:53:16 GMT
  < Content-Length: 0
  < Content-Type: text/plain; charset=utf-8