工作中疑難問題彙總

常見錯誤解決辦法：

EFK：

    1.   無法啓動kibana，解決“Elasticsearch is still initializing the kibana index... ”
          解決辦法：
            1. /_plugin/gui/ 使用gui刪除.kibana的索引
            2. curl -XDELETE http://localhost:9200/.kibana

        注：http://www.4wei.cn/archives/1002492

    2.  FORBIDDEN/12/index read-only / allow delete (api)]
         解決方法：
PUT _settings
  {
    "index": {
      "blocks": {
        "read_only_allow_delete": “false”
      }
    }
  }
原因： 磁盤空間不足

graylog:

    1. How to fix “Deflector exists as an index and is not an alias” warning?
        解決方法：
            https://community.graylog.org/t/how-to-fix-deflector-exists-as-an-index-and-is-not-an-alias-warning/2826

K8S:

    1. fs.go:418] Stat fs failed. Error: no such file or directory

        bug: https://github.com/kubernetes/kubernetes/issues/66335

    2.  dns.go:131] Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 

        修改宿主機上的 /etc/resolv.conf 留三個dns nameserver即可

        注：linux服務器默認最大支持3個Nameserver(https://www.systutorials.com/docs/linux/man/5-resolv.conf/)

    3. dial tcp: lookup k8s7 on 183.60.83.19:53: no such host

        修改 docker-dns.conf 中的dns 並重啓docker 和 kubelet 服務

Fabric:

    1. signal SIGSEGV: segmentation violation code=0x1 addr=0x47 pc=0x7fd6d15ec259]

        加入環境變量GODEBUG=netdns=go (https://yq.aliyun.com/articles/238940)

    2. Error: got unexpected status: FORBIDDEN — Failed to reach implicit threshold of 1 sub-policies, required 1 remaining: permission denied

        修改：AdminPrincipal: Role.MEMBER  to AdminPrincipal: Role.ADMIN ——沒解決

        fabric: 1.2  

            configtxgen -profile ComposerOrdererGenesis -outputBlock genesis.block -channelID ${channelName} 

            configtxgen -profile ComposerChannel -outputCreateChannelTx ${TXNAME}.tx -channelID ${channelName} 

            其中 channelName 兩條命令不能移一樣 
            https://stackoverflow.com/questions/52155272/hyperledger-fabric-creating-channel-error-on-the-cluster

    3.  Error: proposal failed (err: bad proposal response 500)

        peer channel fetch 0 composer-channel.block -c composer-channel --orderer orderer-umark-com:7050

        peer channel join -b composer-channel.block -o orderer-umark-com:7050

    4. 
    kubelet log: 
        W1119 10:26:39.751105   11623 prober.go:103] No ref for container "docker://2fb05afa9519e31e8a359f0dd106202702bad6451db79c66aedba7c8af913c11" (peers-hospital-com-0_long1117085926(092c5c12-ea04-11e8-9948-5254007939c0):mongodb)



    docker log: 
        Nov 19 12:04:34 k8s3 dockerd[21715]: time="2018-11-19T12:04:34.649425081+08:00" level=info msg="Container 8b00756e8159f0795df00fcad97d6f0d20e68aaef648a051c04a3f622ee52903 failed to exit within 10 seconds of signal 15 - using the force"
Nov 19 12:04:34 k8s3 dockerd[21715]: time="2018-11-19T12:04:34.649855145+08:00" level=warning msg="container kill failed because of 'container not found' or 'no such process': Cannot kill container 8b00756e8159f0795df00fcad97d6f0d20e68aaef648a051c04a3f622ee52903: rpc error: code = 2 desc = containerd: container not found"
Nov 19 12:04:36 k8s3 dockerd[21715]: time="2018-11-19T12:04:36.647369073+08:00" level=info msg="Container d19ff25ecafc failed to exit within 10 seconds of kill - trying direct SIGKILL"
Nov 19 12:04:39 k8s3 dockerd[21715]: time="2018-11-19T12:04:39.745941825+08:00" level=error msg="Error running exec in container: rpc error: code = 2 desc = containerd: container not found"

    原因：
        docker 本身的bug引起的，
    解決方法：

        https://github.com/moby/moby/issues/35091
        https://github.com/docker/for-linux/issues/1

Ceph:

    錯誤：CephFS Provisioner Input/Output Error 或 

    解決方法：
        1.   升級內核後：
            報：write error: File name too long

        2.  添加參數 到 cephfs-provisioner deployment.yaml 文件中：
        https://github.com/kubernetes-incubator/external-storage/issues/345

    ok, i just add -disable-ceph-namespace-isolation=true in the deployment, Input/output error just gone
    - args:
    - '-id=cephfs-provisioner-1'
    - '-disable-ceph-namespace-isolation=true'
    command:
    - /usr/local/bin/cephfs-provisioner