K8s資源第四篇（Ingress-nginx）

Ingress：

 Kubernetes 暴露服務的有三種方式，分別爲 LoadBlancer Service、NodePort Service、Ingress。

 官網對 Ingress 的定義爲管理對外服務到集羣內服務之間規則的集合，通俗點講就是它定義規則來允許進入集羣的請求被轉發到集羣中對應服務上，從來實現服務暴漏。 Ingress 能把集羣內 Service 配置成外網能夠訪問的 URL，流量負載均衡，終止SSL，提供基於域名訪問的虛擬主機等等。

 可以把Ingress理解爲一個調度器。

原因：

Pod漂移問題：

 Kubernetes 具有強大的副本控制能力，能保證在任意副本（Pod）掛掉時自動從其他機器啓動一個新的，還可以動態擴容等，通俗地說，這個 Pod 可能在任何時刻出現在任何節點上，也可能在任何時刻死在任何節點上；那麼自然隨着 Pod 的創建和銷燬，Pod IP 肯定會動態變化；那麼如何把這個動態的 Pod IP 暴露出去？這裏藉助於 Kubernetes 的 Service 機制，Service 可以以標籤的形式選定一組帶有指定標籤的 Pod，並監控和自動負載他們的 Pod IP，那麼我們向外暴露只暴露 Service IP 就行了；這就是 NodePort 模式：即在每個節點上開起一個端口，然後轉發到內部 Pod IP 上，如下圖所示：

 此時的訪問方式：http://nodeip:nodeport/

端口管理問題：

 採用 NodePort 方式暴露服務面臨問題是，服務一旦多起來，NodePort 在每個節點上開啓的端口會及其龐大，而且難以維護；這時，我們可以能否使用一個Nginx直接對內進行轉發呢？衆所周知的是，Pod與Pod之間是可以互相通信的，而Pod是可以共享宿主機的網絡名稱空間的，也就是說當在共享網絡名稱空間時，Pod上所監聽的就是Node上的端口。那麼這又該如何實現呢？簡單的實現就是使用 DaemonSet 在每個 Node 上監聽 80，然後寫好規則，因爲 Nginx 外面綁定了宿主機 80 端口（就像 NodePort），本身又在集羣內，那麼向後直接轉發到相應 Service IP 就行了，如下圖所示：

域名分配及動態更新問題：

 從上面的方法，採用 Nginx-Pod 似乎已經解決了問題，但是其實這裏面有一個很大缺陷：當每次有新服務加入又該如何修改 Nginx 配置呢？？我們知道使用Nginx可以通過虛擬主機域名進行區分不同的服務，而每個服務通過upstream進行定義不同的負載均衡池，再加上location進行負載均衡的反向代理，在日常使用中只需要修改nginx.conf即可實現，那在K8S中又該如何實現這種方式的調度呢？？？

 假設後端的服務初始服務只有ecshop，後面增加了bbs和member服務，那麼又該如何將這2個服務加入到Nginx-Pod進行調度呢？總不能每次手動改或者Rolling Update 前端 Nginx Pod 吧！！此時 Ingress 出現了，如果不算上面的Nginx，Ingress 包含兩大組件：Ingress Controller 和 Ingress。

Ingress Controller：

• Nginx：通過nginx代理到後端的資源。
• Traefick：通過treafik代理到後端的資源，Traefik 是一款開源的反向代理與負載均衡工具，它可以監聽後端的變化並自動更新服務配置。
• Envoy：通過Envoy代理後端的資源，Envoy具有一些強大的功能，如斷路器、插件式的處理器鏈，以及可觀測性和可調試性。

Ingress-Nginx：

一個簡單的工作原理圖：

• Nginx 對後端運行的服務（Service1、Service2）提供反向代理。
• 在配置文件中配置了域名與後端服務 Endpoints 的對應關係。
• 客戶端通過使用 DNS 服務或者直接配置本地的 hosts 文件，將域名都映射到 Nginx 代理服務器。
• 當客戶端訪問 service1.com 時，瀏覽器會把包含域名的請求發送給 nginx 服務器，nginx 服務器根據傳來的域名，選擇對應的 Service，這裏就是選擇 Service 1 後端服務，然後根據一定的負載均衡策略，選擇 Service1 中的某個容器接收來自客戶端的請求並作出響應。

Github：https://github.com/kubernetes/Ingress-nginx/

官方文檔：https://kubernetes.io/zh/docs/concepts/services-networking/ingress/

部署ingress-nginx：

#使用Git把項目克隆到本地。
[root@k8smaster data]# git clone https://github.com/kubernetes/ingress-nginx.git
Cloning into 'ingress-nginx'...
remote: Enumerating objects: 669, done.
remote: Counting objects: 100% (669/669), done.
remote: Compressing objects: 100% (497/497), done.
remote: Total 84579 (delta 144), reused 646 (delta 137), pack-reused 83910
Receiving objects: 100% (84579/84579), 102.78 MiB | 118.00 KiB/s, done.
Resolving deltas: 100% (46038/46038), done.
[root@k8smaster data]# cd 

#創建一個ingress目錄，並把主要的配置文件拷貝過來。
[root@k8smaster ~]# mkdir ingress
[root@k8smaster ~]# cd ingress/
[root@k8smaster ingress]# cp /data/ingress-nginx/deploy/static/mandatory.yaml  .
[root@k8smaster ingress]# cp /data/ingress-nginx/deploy/static/provider/baremetal/service-nodeport.yaml  .

#修改配置清單。
[root@k8smaster ingress]# sed  -i '/targetPort: 80/a\ \ \ \ \ \ nodePort: 30080' service-nodeport.yaml
[root@k8smaster ingress]# sed  -i '/targetPort: 443/a\ \ \ \ \ \ nodePort: 30443' service-nodeport.yaml

#創建資源。
[root@k8smaster ingress]# kubectl apply  -f mandatory.yaml 
namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created
[root@k8smaster ingress]# kubectl apply  -f service-nodeport.yaml 
service/ingress-nginx created

#手動創建一個nginx Pod，一個service。
#nginx-svc代理nginx-dm。
[root@k8smaster ingress]# vim ingress-nginx-demo.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-dm
spec:
  replicas: 2
  template:
    metadata:
      labels:
        name: nginx
    spec:
      containers:
      - name: myapp
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
spec:
  selector:
    name: nginx
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP

#創建Ingress   
[root@k8smaster ingress]# vim ingress-nginx-http.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-nginx-http-demo
spec:
  rules:
    - host: abc1.tt.dev
      http:
        paths:
        - path: /
          backend:
            serviceName: nginx-svc
            servicePort: 80

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-nginx-https-demo
spec:
  tls:
    - hosts:
      - abc2.tt.dev
      secretName: tls-secret
  rules:
    - host: abc2.tt.dev
      http:
        paths:
        - path: /
          backend:
            serviceName: nginx-svc
            servicePort: 80

[root@k8smaster ingress]# kubectl apply -f ingress-nginx-demo.yaml 
deployment.extensions/nginx-dm created
service/nginx-svc created
[root@k8smaster ingress]# kubectl apply -f ingress-nginx-http.yaml 
ingress.extensions/ingress-nginx-http-demo created
ingress.extensions/ingress-nginx-https-demo created

#使用openssl生產證書，如果沒有openssl，則使用yum去安裝一個。
[root@k8smaster ingress]# openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/C=CN/O=DevOps/CN=abc2.tt.dev"
Generating a 2048 bit RSA private key
.............................................................................................+++
................................................................................................................................+++
writing new private key to 'tls.key'
-----
#創建secret
[root@k8smaster ingress]# kubectl create secret tls tls-secret --key tls.key --cert tls.crt

#查看Pod的詳細信息。
[root@k8smaster ingress]# kubectl get pods -o wide
NAME                       READY   STATUS    RESTARTS   AGE     IP             NODE       NOMINATED NODE   READINESS GATES
nginx-dm-7c46ff4c5-l8snl   1/1     Running   0          2m39s   10.244.1.170   k8snode1   <none>           <none>
nginx-dm-7c46ff4c5-vggl2   1/1     Running   0          2m39s   10.244.2.176   k8snode2   <none>           <none>
#查看Deployment信息。
[root@k8smaster ingress]# kubectl get deployment
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
nginx-dm   2/2     2            2           6m36s
#查看service信息。
[root@k8smaster ingress]# kubectl get svc 
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP   48d
nginx-svc    ClusterIP   10.99.53.204   <none>        80/TCP    2m45s
#發現nginx-svc的Endpoints已經綁定到了Pod。
[root@k8smaster ingress]# kubectl describe svc nginx-svc
Name:              nginx-svc
Namespace:         default
Labels:            <none>
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                    {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"nginx-svc","namespace":"default"},"spec":{"ports":[{"port":80,"pr...
Selector:          name=nginx
Type:              ClusterIP
IP:                10.99.53.204
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.1.170:80,10.244.2.176:80
Session Affinity:  None
Events:            <none>
[root@k8smaster ingress]# kubectl get svc -n ingress-nginx
NAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx   NodePort   10.103.152.44   <none>        80:30080/TCP,443:30443/TCP   4m8s

#並且ingress-nginx的Endpoints也都成功綁定到。
[root@k8smaster ingress]# kubectl describe svc ingress-nginx -n ingress-nginx
Name:                     ingress-nginx
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/par...
Selector:                 app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
Type:                     NodePort
IP:                       10.103.152.44
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  30080/TCP
Endpoints:                10.244.1.169:80
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  30443/TCP
Endpoints:                10.244.1.169:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

#查看創建的ingress
[root@k8smaster ingress]# kubectl get ingress
NAME                       HOSTS               ADDRESS         PORTS     AGE
ingress-nginx-http-demo    abc1.tt.dev         10.103.152.44   80        4m29s
ingress-nginx-https-demo   abc2.tt.dev         10.103.152.44   80, 443   4m29s
nginx-deployment           nginx.example.com   10.103.152.44   80        45d
[root@k8smaster ingress]# kubectl describe  ingress ingress-nginx-http-demo 
Name:             ingress-nginx-http-demo
Namespace:        default
Address:          10.103.152.44
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host         Path  Backends
  ----         ----  --------
  abc1.tt.dev  
                /   nginx-svc:80 (10.244.1.170:80,10.244.2.176:80)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"ingress-nginx-http-demo","namespace":"default"},"spec":{"rules":[{"host":"abc1.tt.dev","http":{"paths":[{"backend":{"serviceName":"nginx-svc","servicePort":80},"path":"/"}]}}]}}

Events:
  Type    Reason  Age    From                      Message
  ----    ------  ----   ----                      -------
  Normal  CREATE  4m53s  nginx-ingress-controller  Ingress default/ingress-nginx-http-demo
  Normal  UPDATE  4m22s  nginx-ingress-controller  Ingress default/ingress-nginx-http-demo
[root@k8smaster ingress]# kubectl describe  ingress ingress-nginx-https-demo 
Name:             ingress-nginx-https-demo
Namespace:        default
Address:          10.103.152.44
Default backend:  default-http-backend:80 (<none>)
TLS:
  tls-secret terminates abc2.tt.dev
Rules:
    Host         Path  Backends
    ----         ----  --------
abc2.tt.dev  
                /   nginx-svc:80 (10.244.1.170:80,10.244.2.176:80)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"ingress-nginx-https-demo","namespace":"default"},"spec":{"rules":[{"host":"abc2.tt.dev","http":{"paths":[{"backend":{"serviceName":"nginx-svc","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["abc2.tt.dev"],"secretName":"tls-secret"}]}}

Events:
  Type    Reason  Age    From                      Message
  ----    ------  ----   ----                      -------
  Normal  CREATE  4m57s  nginx-ingress-controller  Ingress default/ingress-nginx-https-demo
  Normal  UPDATE  4m26s  nginx-ingress-controller  Ingress default/ingress-nginx-https-demo

#查看nginx的配置文件。
[root@k8smaster ingress]# kubectl exec -it -n ingress-nginx $pod_ingress -- cat /etc/nginx/nginx.conf  | grep abc
    ## start server abc1.tt.dev
	    server_name abc1.tt.dev ;
    ## end server abc1.tt.dev
    ## start server abc2.tt.dev
	    server_name abc2.tt.dev ;
    ## end server abc2.tt.dev

#由於我們的Ingress-nginx的service爲Nodeport，所以需要修改abc1.tt.dev和abc2.tt.dev域名對應每個節點的IP。
[root@k8smaster ingress]# vim /etc/hosts
192.168.43.45 k8smaster abc1.tt.dev abc2.tt.dev
192.168.43.136 k8snode1 abc1.tt.dev abc2.tt.dev
192.168.43.176 k8snode2 abc1.tt.dev abc2.tt.dev

#訪問測試。
[root@k8smaster ingress]# curl http://abc1.tt.dev:30080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@k8smaster ingress]# curl https://abc2.tt.dev:30443 -k
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

密碼驗證：

[root@k8smaster ingress]# vim ingress-with-auth.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-with-auth
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo'
spec:
  rules:
  - host: auth.tt.dev
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-svc
          servicePort: 80
[root@k8smaster ingress]# yum -y install httpd-tools
[root@k8smaster ingress]# htpasswd -bc auth testuser pswd  #用戶名testuser，密碼pswd。
Adding password for user testuser
[root@k8smaster ingress]# docker run -it --rm -v $(pwd):/data -w /data jess/htpasswd -bc auth testuser pswd
Unable to find image 'jess/htpasswd:latest' locally
latest: Pulling from jess/htpasswd
Image docker.io/jess/htpasswd:latest uses outdated schema1 manifest format. Please upgrade to a schema2 image for better future compatibility. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
e7c96db7181b: Pull complete 
92704bed51b2: Pull complete 
Digest: sha256:4ef4db77006976d385eb487877def47c29b68cf8d784242fb9114e90463ce2d0
Status: Downloaded newer image for jess/htpasswd:latest
Adding password for user testuser
[root@k8smaster ingress]# kubectl create secret generic basic-auth --from-file=auth
secret/basic-auth created
[root@k8smaster ingress]# kubectl get secret basic-auth -o yaml
apiVersion: v1
data:
  auth: dGVzdHVzZXI6JGFwcjEkL2V1MkN5VXMkY0J1S3dhRlVMRnE0MjVkZTdIVGREMQo=
kind: Secret
metadata:
  creationTimestamp: "2019-12-16T07:49:35Z"
  name: basic-auth
  namespace: default
  resourceVersion: "383740"
  selfLink: /api/v1/namespaces/default/secrets/basic-auth
  uid: 0d972bb5-7219-4f98-87e5-7c6f55214e0d
type: Opaque
[root@k8smaster ingress]# kubectl apply -f ingress-with-auth.yaml 
ingress.extensions/ingress-with-auth created
[root@k8smaster ingress]# vim /etc/hosts
192.168.43.45 k8smaster abc1.tt.dev abc2.tt.dev auth.tt.dev
192.168.43.136 k8snode1 abc1.tt.dev abc2.tt.dev auth.tt.dev
192.168.43.176 k8snode2 abc1.tt.dev abc2.tt.dev auth.tt.dev

#測試驗證：
[root@k8smaster ingress]# curl http://auth.tt.dev:30080   #訪問被拒絕了。
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>openresty/1.15.8.2</center>
</body>
</html>
#我們再指定用戶名和密碼，發現訪問成功。
[root@k8smaster ingress]# curl http://auth.tt.dev:30080 -u 'testuser:pswd'
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

參考文檔：https://www.cnblogs.com/linuxk/p/9706720.html

參考文檔：https://blog.csdn.net/shida_csdn/article/details/84032019

參考文檔：https://blog.csdn.net/qq_37950254/article/details/89486057