kubernetes實戰-基於redis和docker的留言簿案例

一、介紹

本案例基於Kubernetes和Docker，其中包括

1、web前端

2、redis master

3、redis slave

其中web前端通過javascript redis api和redis master交互

kubernetes體系架構

二、配置

0、先決條件

Kubernetes 集羣

1、啓動redis master

使用replication controller確保只有一個pod在運行（當某個節點down了，rc會在另一個健康的node啓動redis master），但可能會有數據丟失。

[root@centos1 example]# kubectl create -f redis-master-controller.json replicationcontrollers/redis-master[root@centos1 example]# kubectl get rcCONTROLLER     CONTAINER(S)   IMAGE(S)   SELECTOR            REPLICASredis-master   master         redis      name=redis-master   1

驗證master運行成功，如下展示了pod運行在centos2/192.168.1.112這臺機器上    

[root@centos1 example]# kubectl get podsPOD                  IP           CONTAINER(S)   IMAGE(S)   HOST                    LABELS              STATUS    CREATED      MESSAGEredis-master-svar7   172.17.0.9                             centos2/192.168.1.112   name=redis-master   Running   55 seconds                                     master         redis                                                  Running   55 seconds

SSH到centos2/192.168.1.112查看docker狀態

[root@centos2 yum.repos.d]# docker psCONTAINER ID        IMAGE                                  COMMAND                CREATED             STATUS              PORTS               NAMES91689ce56668        redis:latest                           "/entrypoint.sh redi   3 minutes ago       Up 3 minutes                            k8s_master.52732b08_redis-master-svar7_default_5b6d5485-1894-11e5-b3ad-000c293c8c19_97e79b7b   38c3180813c3        gcr.io/google_containers/pause:0.8.0   "/pause"               3 minutes ago       Up 3 minutes                            k8s_POD.49eee8c2_redis-master-svar7_default_5b6d5485-1894-11e5-b3ad-000c293c8c19_298e038f

注意：kubectl create執行後，如果鏡像不存在，會執行docker pull，根據網絡情況，下載中的pods 在kubertnetes UI上會顯示pending狀態

2、啓動master service

一個kubernetes service會對一個或多個container進行負載均衡，這是通過我們上面redis-master中定義的labels元數據實現的，值得注意的是，在redis中只有一個master，但是我們依然爲它創建一個service，這是因爲這樣我們就能使用一個elastic IP來路由到具體某一個master。

kubernetes集羣中的service是通過container中的環境變量實現服務發現的，service基於pod label實現container的負載均衡。

在第一步中創建的pod包含了一個label“name=redis-master”，service的selector字段決定了service將流量轉發給哪個pod，port和targetPort信息定義了service proxy運行在什麼端口。

[root@centos1 example]# kubectl create -f redis-master-service.json services/redis-master[root@centos1 example]# kubectl get servicesNAME            LABELS                                    SELECTOR            IP(S)           PORT(S)redis-master    name=redis-master                         name=redis-master   10.254.154.90   6379/TCP

上面的運行成功後，所有pods都能發現redis master運行在6379端口，從salve到master流量走向會有以下兩步：

1、一個redis slave會連接到redis master service的port上

2、流量會從service節點上的port到targetPort

如果targetPort未指定，默認和port一致

3、啓動replicated slave pod

雖然redis master是一個單獨的pod，redis slaves是一個replicated pod，在Kubernetes中，一個Replication Controller負責管理一個replicated pod的多個實例，RC會自動拉起down掉的replica（可以通過殺死docker 進程方式簡單測試）

[root@centos1 example]# kubectl create -f redis-slave-controller.json replicationcontrollers/redis-slave[root@centos1 example]# kubectl get rcCONTROLLER     CONTAINER(S)   IMAGE(S)                    SELECTOR            REPLICASredis-master   master         redis                       name=redis-master   1redis-slave    slave          kubernetes/redis-slave:v2   name=redis-slave    2[root@centos1 example]# kubectl get podsPOD                  IP            CONTAINER(S)   IMAGE(S)                    HOST                    LABELS              STATUS    CREATED      MESSAGEredis-master-svar7   172.17.0.9                                               centos2/192.168.1.112   name=redis-master   Running   41 minutes                                      master         redis                                                                   Running   41 minutes   redis-slave-31tkb    172.17.0.10                                              centos2/192.168.1.112   name=redis-slave    Running   29 seconds                                      slave          kubernetes/redis-slave:v2                                               Running   28 seconds   redis-slave-uk8nu    172.17.0.11                                              centos2/192.168.1.112   name=redis-slave    Running   29 seconds                                      slave          kubernetes/redis-slave:v2                                               Running   28 seconds

可以看到一個master pod和兩個slave pod

4、啓動slave service

和master一樣，我們希望有一個代理服務連接到redis slave，除了服務發現之外，slave service還爲web app client提供了透明代理。

這次service 的selector是name=redis-slave，我們可以方便的使用kubectl get services -l "label=value"命令來定位這些服務

[root@centos1 example]# kubectl create -f redis-slave-service.json services/redis-slave[root@centos1 example]# kubectl get servicesNAME            LABELS                                    SELECTOR            IP(S)            PORT(S)redis-master    name=redis-master                         name=redis-master   10.254.154.90    6379/TCPredis-slave     name=redis-slave                          name=redis-slave    10.254.159.145   6379/TCP

5、創建frontend pod

這是一個簡單的PHP 服務，用來和master service（寫請求）或slave service（讀請求）交互

[root@centos1 example]# kubectl create -f frontend-controller.json replicationcontrollers/frontend[root@centos1 example]# kubectl get rcCONTROLLER     CONTAINER(S)   IMAGE(S)                                    SELECTOR            REPLICASfrontend       php-redis      kubernetes/example-guestbook-php-redis:v2   name=frontend       3redis-master   master         redis                                       name=redis-master   1redis-slave    slave          kubernetes/redis-slave:v2                   name=redis-slave    2

運行成功後，查看當前pod運行狀態

[root@centos1 example]# kubectl get podsPOD                  IP            CONTAINER(S)   IMAGE(S)                                    HOST                    LABELS              STATUS    CREATED      MESSAGEfrontend-fr5z1       172.17.0.13                                                              centos2/192.168.1.112   name=frontend       Running   2 minutes                                       php-redis      kubernetes/example-guestbook-php-redis:v2                                               Running   2 minutes    frontend-gjx3t       172.17.0.14                                                              centos2/192.168.1.112   name=frontend       Running   2 minutes                                       php-redis      kubernetes/example-guestbook-php-redis:v2                                               Running   2 minutes    frontend-v608r       172.17.0.12                                                              centos2/192.168.1.112   name=frontend       Running   2 minutes                                       php-redis      kubernetes/example-guestbook-php-redis:v2                                               Running   2 minutes    redis-master-svar7   172.17.0.9                                                               centos2/192.168.1.112   name=redis-master   Running   53 minutes                                      master         redis                                                                                   Running   53 minutes   redis-slave-31tkb    172.17.0.10                                                              centos2/192.168.1.112   name=redis-slave    Running   12 minutes                                      slave          kubernetes/redis-slave:v2                                                               Running   12 minutes   redis-slave-uk8nu    172.17.0.11                                                              centos2/192.168.1.112   name=redis-slave    Running   12 minutes                                      slave          kubernetes/redis-slave:v2                                                               Running   12 minutes

可以看到一個redis master，兩個redis slave和三個frontend pods

6、創建guestbook service

和其他service一樣，你可以創建一個service管理frontend pods

[root@centos1 example]# kubectl create -f frontend-service.json services/frontend[root@centos1 example]# kubectl get servicesNAME            LABELS                                    SELECTOR            IP(S)            PORT(S)frontend        name=frontend                             name=frontend       10.254.154.111   80/TCPredis-master    name=redis-master                         name=redis-master   10.254.154.90    6379/TCPredis-slave     name=redis-slave                          name=redis-slave    10.254.159.145   6379/TCP

我們可以通過frontend service（10.254.154.111）訪問pods，但是這個IP明顯是無法在外部訪問的，下一節講解如何在外部網絡訪問guestbook

7、外部網絡訪問guestbook

kubernetes 支持兩種訪問暴露一個服務到外部IP地址NodePorts 和LoadBalancers，https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/services.md#external-services

另外可以查看防火牆，找到service對應的端口，如下

[root@centos1 example]# kubectl get pods,servicesPOD                  IP            CONTAINER(S)   IMAGE(S)                                    HOST                    LABELS              STATUS    CREATED         MESSAGEfrontend-fr5z1       172.17.0.13                                                              centos2/192.168.1.112   name=frontend       Running   22 minutes                                         php-redis      kubernetes/example-guestbook-php-redis:v2                                               Running   22 minutes      frontend-gjx3t       172.17.0.14                                                              centos2/192.168.1.112   name=frontend       Running   22 minutes                                         php-redis      kubernetes/example-guestbook-php-redis:v2                                               Running   22 minutes      frontend-v608r       172.17.0.12                                                              centos2/192.168.1.112   name=frontend       Running   22 minutes                                         php-redis      kubernetes/example-guestbook-php-redis:v2                                               Running   22 minutes      redis-master-svar7   172.17.0.9                                                               centos2/192.168.1.112   name=redis-master   Running   About an hour                                      master         redis                                                                                   Running   About an hour   redis-slave-31tkb    172.17.0.10                                                              centos2/192.168.1.112   name=redis-slave    Running   32 minutes                                         slave          kubernetes/redis-slave:v2                                                               Running   32 minutes      redis-slave-uk8nu    172.17.0.11                                                              centos2/192.168.1.112   name=redis-slave    Running   32 minutes                                         slave          kubernetes/redis-slave:v2                                                               Running   32 minutes      NAME            LABELS                                    SELECTOR            IP(S)            PORT(S)frontend        name=frontend                             name=frontend       10.254.154.111   80/TCPkubernetes      component=apiserver,provider=kubernetes   <none>              10.254.0.2       443/TCPkubernetes-ro   component=apiserver,provider=kubernetes   <none>              10.254.0.1       80/TCPredis-master    name=redis-master                         name=redis-master   10.254.154.90    6379/TCPredis-slave     name=redis-slave                          name=redis-slave    10.254.159.145   6379/TCP[root@centos1 example]# kubectl get servicesNAME            LABELS                                    SELECTOR            IP(S)            PORT(S)frontend        name=frontend                             name=frontend       10.254.154.111   80/TCPkubernetes      component=apiserver,provider=kubernetes   <none>              10.254.0.2       443/TCPkubernetes-ro   component=apiserver,provider=kubernetes   <none>              10.254.0.1       80/TCPredis-master    name=redis-master                         name=redis-master   10.254.154.90    6379/TCPredis-slave     name=redis-slave                          name=redis-slave    10.254.159.145   6379/TCP

發現redis-master是在10.254.154.90上，登錄到centos2上，執行iptables-save，發現其中有這樣一條規則

-A KUBE-PORTALS-HOST -d 10.254.154.90/32 -p tcp -m comment --comment "default/redis-master:" -m tcp --dport 6379 -j DNAT --to-destination 192.168.1.112:49038

說明，centos2本機的49038端口映射到master container內的6379了，當然我們就能執行在本機訪問redis了

[root@centos2 yum.repos.d]# redis-cli -p 49038127.0.0.1:49038> set a bOK127.0.0.1:49038> get a"b"

8、使用curl簡單測試

提交數據

curl "localhost:8000/index.php?cmd=set&key=messages&value=jay_sais_hi"

查詢數據

curl "localhost:8000/index.php?cmd=get&key=messages"

附本案例用到的6個.json文件

1、redis-master-controller.json

{   "kind":"ReplicationController",   "apiVersion":"v1beta3",   "metadata":{      "name":"redis-master",      "labels":{         "name":"redis-master"      }   },   "spec":{      "replicas":1,      "selector":{         "name":"redis-master"      },      "template":{         "metadata":{            "labels":{               "name":"redis-master"            }         },         "spec":{            "containers":[               {                  "name":"master",                  "p_w_picpath":"redis",                  "ports":[                     {                        "containerPort":6379                     }                  ]               }            ]         }      }   }}

2、redis-master-service.json

{   "kind":"Service",   "apiVersion":"v1beta3",   "metadata":{      "name":"redis-master",      "labels":{         "name":"redis-master"      }   },   "spec":{      "ports": [        {          "port":6379,          "targetPort":6379        }      ],      "selector":{         "name":"redis-master"      }   }}

3、redis-slave-controller.json

{   "kind":"ReplicationController",   "apiVersion":"v1beta3",   "metadata":{      "name":"redis-slave",      "labels":{         "name":"redis-slave"      }   },   "spec":{      "replicas":2,      "selector":{         "name":"redis-slave"      },      "template":{         "metadata":{            "labels":{               "name":"redis-slave"            }         },         "spec":{            "containers":[               {                  "name":"slave",                  "p_w_picpath":"kubernetes/redis-slave:v2",                  "ports":[                     {                        "containerPort":6379                     }                  ]               }            ]         }      }   }}

4、redis-slave-service.json

{   "kind":"Service",   "apiVersion":"v1beta3",   "metadata":{      "name":"redis-slave",      "labels":{         "name":"redis-slave"      }   },   "spec":{      "ports": [        {          "port":6379        }      ],      "selector":{         "name":"redis-slave"      }   }}

5、frontend-controller.json

{   "kind":"ReplicationController",   "apiVersion":"v1beta3",   "metadata":{      "name":"frontend",      "labels":{         "name":"frontend"      }   },   "spec":{      "replicas":3,      "selector":{         "name":"frontend"      },      "template":{         "metadata":{            "labels":{               "name":"frontend"            }         },         "spec":{            "containers":[               {                  "name":"php-redis",                  "p_w_picpath":"kubernetes/example-guestbook-php-redis:v2",                  "ports":[                     {                        "containerPort":80                     }                  ]               }            ]         }      }   }}

6、frontend-service.json

{   "kind":"Service",   "apiVersion":"v1beta3",   "metadata":{      "name":"frontend",      "labels":{         "name":"frontend"      }   },   "spec":{      "ports": [        {          "port":80        }      ],      "selector":{         "name":"frontend"      }   }}