Kubernetes进阶 -- ingress 负载均衡服务详解

简介

• 一种全局的、为了代理不同后端 Service 而设置的负载均衡服务,就是 Kubernetes 里的Ingress服务。
• Ingress由两部分组成:Ingress controller和Ingress服务。
• Ingress Controller 会根据你定义的 Ingress 对象,提供对应的代理能力。业界常用的各种反向代理项目,比如 Nginx、HAProxy、Envoy、Traefik 等,都已经为 Kubernetes 专门维护了对应的 Ingress Controller。
• 官网:https://kubernetes.github.io/ingress-nginx/

ingress相当于一个负载均衡器，集群中的service 做的是 pod 的暴露端口，然后做后端负载均衡，而 ingress 是在 service 之前，只连接 service。做 service 的负载均衡，反向代理。使用热更新的方式。

ingress部署

部署的时候可以针对很多的环境，我们选择部署bare-metal环境下的ingress，它默认是使用nodeport的方式进行暴露的。

/获取它的部署文件
[root@server2 manifest]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml

[root@server2 manifest]# vim deploy.yaml		
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0
          image: docker.io/jettech/kube-webhook-certgen:v1.2.0		/查看它需要的镜像

我们还是一样，先把镜像下载下来，打标签上传到harbor仓库，来提升试验速度。
然后：

[root@server2 manifest]# vim deploy.yaml
          image: nginx-ingress-controller:0.33.0
          image: kube-webhook-certgen:v1.2.0
          image: kube-webhook-certgen:v1.2.0		//镜像拉取位置从我们设置的harbor仓库拉取
[root@server2 manifest]# vim /etc/docker/daemon.json
"registry-mirrors": ["https://reg.caoaoyuan.org"],

就可以使用了：

[root@server2 manifest]# kubectl apply -f deploy.yaml 

[root@server2 manifest]# kubectl get namespaces 
NAME              STATUS   AGE
default           Active   11d
ingress-nginx     Active   15s		/首先会生成一个namespace
kube-node-lease   Active   11d
kube-public       Active   11d
kube-system       Active   11d
[root@server2 manifest]# kubectl get pod -n ingress-nginx 	/指定namespace才可以访问
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-hm2nv        0/1     Completed   0          99s
ingress-nginx-admission-patch-hmxbt         0/1     Completed   0          99s
ingress-nginx-controller-77b5fc5746-z55wh   1/1     Running     0          109s

[root@server2 manifest]# kubectl get all -n ingress-nginx 
NAME                                            READY   STATUS      RESTARTS   AGE
pod/ingress-nginx-admission-create-hm2nv        0/1     Completed   0          2m15s
pod/ingress-nginx-admission-patch-hmxbt         0/1     Completed   0          2m15s
pod/ingress-nginx-controller-77b5fc5746-z55wh   1/1     Running     0          2m25s

NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-controller             NodePort    10.97.148.61     <none>        80:32066/TCP,443:31334/TCP   2m25s
service/ingress-nginx-controller-admission   ClusterIP   10.103.172.168   <none>        443/TCP                      2m26s
/还是用服务的方式管理，使用nodeport的方式暴露端口

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   1/1     1            1           2m25s

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-controller-77b5fc5746   1         1         1       2m25s

NAME                                       COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   1/1           7s         2m25s
job.batch/ingress-nginx-admission-patch    1/1           6s         2m25s

查看pod：

[root@server2 manifest]# kubectl get pod -n ingress-nginx -o wide
NAME                                        READY   STATUS      RESTARTS   AGE     IP            NODE      NOMINATED NODE   READINESS GATES
ingress-nginx-admission-create-hm2nv        0/1     Completed   0          3m58s   10.244.22.3   server4   <none>           <none>
ingress-nginx-admission-patch-hmxbt         0/1     Completed   0          3m58s   10.244.22.4   server4   <none>           <none>
ingress-nginx-controller-77b5fc5746-z55wh   1/1     Running     0          4m8s    10.244.22.5   server4   <none>           <none>

查看svc信息;

[root@server2 manifest]# kubectl get svc -n ingress-nginx 
NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.97.148.61     <none>        80:32066/TCP,443:31334/TCP   4m20s
ingress-nginx-controller-admission   ClusterIP   10.103.172.168   <none>        443/TCP                      4m21s
[root@server2 manifest]# kubectl -n ingress-nginx describe svc ingress-nginx-controller
Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/version=0.33.0
                          helm.sh/chart=ingress-nginx-2.9.0
Annotations:              Selector:  app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     NodePort
IP:                       10.97.148.61
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  32066/TCP
Endpoints:                10.244.22.5:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  32066/TCP		//开放32066端口
Endpoints:                10.244.22.5:443		//后端对应的是 pod 的 ip 地址
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

它使用的是 nodeport 的方式我们就可以在外部进行访问了：

[root@rhel7host k8s]# curl 172.25.254.3:20666
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.19.0</center>
</body>
</html>
//说明服务没有问题

这是用户访问的顺序就是：

user -> (ingress -> svc -> pod) -> svc -> pod

用户访问 ingress，ingress 用自己的 svc 均衡 pod ，然后 ingress 的 pod 在对后面的 其它 svc 进行管理，实现负载均衡,
我们现在就去部署。

创建ingress服务：

[root@server2 manifest]# vim ingress.yml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress1
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: www1.westos.org
    http:
      paths:
      - path: /
        backend:
          serviceName: myservice
          servicePort: 80

//意思是当我们访问 www1.westos.org 时，负载到 myservice 的服务上

我们应该给 www1.westos.org 做一个解析。我们做到server4上.

[root@rhel7host k8s]#  vim /etc/hosts
172.25.254.4 server4 www1.westos.org

但其实3和4 是都可以访问的，因为ingress服务 在每个结点都开放了端口：

[root@server3 ~]# netstat -tnlp |grep :32066
tcp        0      0 0.0.0.0:32066           0.0.0.0:*               LISTEN      4844/kube-proxy
[root@server4 ~]# netstat -tnlp |grep :32066
tcp        0      0 0.0.0.0:32066           0.0.0.0:*               LISTEN      4844/kube-proxy

访问：

[root@rhel7host k8s]# curl www1.westos.org:32066
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
[root@rhel7host k8s]# curl www1.westos.org:32066/hostname.html
deployment-example-846496db9d-2jznm
[root@rhel7host k8s]# curl www1.westos.org:32066/hostname.html
deployment-example-846496db9d-rn6sx
[root@rhel7host k8s]# curl www1.westos.org:32066/hostname.html
deployment-example-846496db9d-2jznm
[root@rhel7host k8s]# curl www1.westos.org:32066/hostname.html
deployment-example-846496db9d-rn6sx

[root@server2 manifest]# kubectl get pod -n ingress-nginx 
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-hm2nv        0/1     Completed   0          90m
ingress-nginx-admission-patch-hmxbt         0/1     Completed   0          90m
ingress-nginx-controller-77b5fc5746-z55wh   1/1     Running     0          90m	

//ingress-nginx-controller 这个 pod 其实就是一个nginx，我们登录进去查看
[root@server2 manifest]# kubectl -n ingress-nginx exec -it ingress-nginx-controller-77b5fc5746-z55wh -- sh
/etc/nginx $ vi nginx.conf
        ## start server www1.westos.org                                                                    
        server {                                                                                                                               
                server_name www1.westos.org ;                                                                                    
                                                                                                                                               
                listen 80  ;                                                                                                     
                listen 443  ssl http2 ;                                                                    
                                                                                                                                               
                set $proxy_upstream_name "-";                                                                                    
                                                                                                                                               
                ssl_certificate_by_lua_block {                                                                                   
                        certificate.call()                                                                 
                }                                                                                                                              
                                                                                                                                 
                location / {                                                                                                                   
                                                                                                                                 
                        set $namespace      "default";                                                     
                        set $ingress_name   "ingress1";                                                                                        
                        set $service_name   "myservice";                                                                         
                        set $service_port   "80";                                                                                              
                        set $location_path  "/";  

我们在ingress.yml中写的内容被记录到了这里，这是因为yml文件中的内容存储到etcd中，pod会自动连接api接口把这些内容读出来，转换成nginx的配置文件，它之后才能生效，然后pod在进行reload。

多个ingress

ingress也是支持列表的，我们现在尝试做两个，但首先得为他们各分配一个后端的svc

加一个service：

[root@server2 manifest]# vim service.yml
kind: Service
apiVersion: v1
metadata:
  name: myservice
spec:
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  selector:
    app: myapp
  type: ClusterIP

---
kind: Service		/新加的service，和上面不同。
apiVersion: v1
metadata:
  name: myservice2
spec:
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  selector:
    app: myappv2
  type: ClusterIP

[root@server2 manifest]# kubectl describe svc myservice2
Name:              myservice2
Namespace:         default
Labels:            <none>
Annotations:       Selector:  app=myappv2
Type:              ClusterIP
IP:                10.104.106.74
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         <none>		//myservice2 还没有后端
Session Affinity:  None
Events:            <none>

为新加的service建立两个后端：

[root@server2 manifest]# kubectl delete -f pod2.yml 
deployment.apps "deployment-example" deleted
[root@server2 manifest]# vim deployment.yml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-v1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:v1

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-v2
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myappv2		/修改标签
  template:
    metadata:
      labels:
        app: myappv2
    spec:
      containers:
      - name: myapp
        image: myapp:v2
[root@server2 manifest]# kubectl apply -f deployment.yml

[root@server2 manifest]# kubectl describe svc myservice
Name:              myservice
Namespace:         default
Labels:            <none>
Annotations:       Selector:  app=myapp
Type:              ClusterIP
IP:                10.109.176.196
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.141.201:80,10.244.22.6:80
Session Affinity:  None
Events:            <none>
[root@server2 manifest]# kubectl describe svc myservice2
Name:              myservice2
Namespace:         default
Labels:            <none>
Annotations:       Selector:  app=myappv2
Type:              ClusterIP
IP:                10.104.106.74
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.141.202:80,10.244.22.7:80
Session Affinity:  None				//现在各有两个终端
Events:            <none>

添加ingress：

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress1
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: www1.westos.org
    http:
      paths:
      - path: /
        backend:
          serviceName: myservice
          servicePort: 80
  - host: www2.westos.org
    http:
      paths:
      - path: /		访问www2.westos.com 时，走 myservice2 这个svc
        backend:
          serviceName: myservice2
          servicePort: 80

[root@server2 manifest]# kubectl describe ingress ingress1 
Name:             ingress1
Namespace:        default
Address:          172.25.254.4
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host             Path  Backends
  ----             ----  --------
  www1.westos.org  
                   /   myservice:80 (10.244.141.201:80,10.244.22.6:80)
  www2.westos.org  
                   /   myservice2:80 (10.244.141.202:80,10.244.22.7:80)
Annotations:       kubernetes.io/ingress.class: nginx

在做解析：
[root@rhel7host k8s]# vim /etc/hosts
172.25.254.4 server4 www1.westos.org www2.westos.org

当前每个service都又2两个后端，我们进行访问：

[root@rhel7host k8s]# curl www1.westos.org:32066
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>

[root@rhel7host k8s]# curl www1.westos.org:32066/hostname.html
deployment-v1-7449b5b68f-qw4kr
[root@rhel7host k8s]# curl www1.westos.org:32066/hostname.html
deployment-v1-7449b5b68f-8pz5c
/ 访问 www1 时又两台后端在轮询

[root@rhel7host k8s]# curl www2.westos.org:32066
Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a>
[root@rhel7host k8s]# curl www2.westos.org:32066/hostname.html
deployment-v2-755458b96d-qsgt8
[root@rhel7host k8s]# curl www2.westos.org:32066/hostname.html
deployment-v2-755458b96d-5ccqt
/ 访问www2时有另外两台后端在轮询

此时pod内部同样也从etcd中抓取了信息，进行了配置：

[root@server2 manifest]# kubectl -n ingress-nginx exec -it ingress-nginx-controller-77b5fc5746-z55wh -- sh
/etc/nginx $ vi nginx.conf

我们并没有编辑这个pod的配置文件，但是它却自动生效了，自动重载。

DaemonSet方式的ingress

[root@server2 manifest]# kubectl  -n ingress-nginx get all
...
NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-controller             NodePort    10.97.148.61     <none>        80:32066/TCP,443:31334/TCP   129m
service/ingress-nginx-controller-admission   ClusterIP   10.103.172.168   <none>        443/TCP                      129m

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   1/1     1            1           129m

目前使用的是deployment+ NodePort  的方式部署ingress。

我们还可以使用 DaemonSet 的方式进行部署。

• 用DaemonSet结合nodeselector来部署ingress-controller到特定的node上,然后使用
  HostNetwork直接把该pod与宿主机node的网络打通,直接使用宿主机的80/433端口就能访问服务。
• 优点是整个请求链路最简单,性能相对NodePort模式更好。
• 缺点是由于直接利用宿主机节点的网络和端口,一个node只能部署一个ingress-controller pod。
• 比较适合大并发的生产环境使用。

---
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: DaemonSet		/改为DaemonSet控制器
...
     	/指定部署到server4

这样的话我们就不需要ingress的svc服务了，不需要暴露端口，因为我们访问的是物理结点。

[root@server2 manifest]# kubectl -n ingress-nginx  delete service/ingress-nginx-controller 
service "ingress-nginx-controller" deleted
[root@server2 manifest]# kubectl -n ingress-nginx delete svc ingress-nginx-controller-admission 
service "ingress-nginx-controller-admission" deleted

[root@server2 manifest]# kubectl -n ingress-nginx delete deployments.apps ingress-nginx-controller 
deployment.apps "ingress-nginx-controller" deleted
[root@server2 manifest]# kubectl apply -f deploy.yaml
/删除deployment控制器，在应用
[root@server2 manifest]# kubectl get all -n ingress-nginx 

NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                    AGE
daemonset.apps/ingress-nginx-controller   1         1         1       1            1           kubernetes.io/hostname=server4   49s
/daemonset控制器就生成了

在server4上查看:

[root@server4 ~]# netstat -atnlp|grep :80
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      28955/nginx: master 
tcp6       0      0 :::80                   :::*                    LISTEN      28955/nginx: master 
[root@server4 ~]# netstat -atnlp|grep :443
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      28955/nginx: master 
tcp        0      0 10.96.0.1:39272         10.96.0.1:443           ESTABLISHED 4864/calico-node    
tcp        0      0 10.96.0.1:39270         10.96.0.1:443           ESTABLISHED 4862/calico-node    
tcp        0      0 10.96.0.1:34348         10.96.0.1:443           ESTABLISHED 28917/nginx-ingress 
tcp6       0      0 :::443                  :::*                    LISTEN      28955/nginx: master
/ 80和443端口出现

server3查看:

[root@server3 ~]# netstat -tnlp |grep :80
[root@server3 ~]# netstat -tnlp |grep :443		
[root@server3 ~]# 

没有端口，因为我们绑定了只在server4上生效。

[root@rhel7host k8s]# curl www1.westos.org/hostname.html
deployment-v1-7449b5b68f-qw4kr
[root@rhel7host k8s]# curl www1.westos.org/hostname.html
deployment-v1-7449b5b68f-8pz5c
[root@rhel7host k8s]# curl www2.westos.org/hostname.html
deployment-v2-755458b96d-5ccqt
[root@rhel7host k8s]# curl www2.westos.org/hostname.html
deployment-v2-755458b96d-qsgt8

都指定的是server4，所以直接访问，不用加端口，就访问成功了。

使用cookie 实现sticky sissions 会话保持

[root@server2 manifest]# vim ingress.yml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress1
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: www1.westos.org
    http:
      paths:
      - path: /
        backend:
          serviceName: myservice
          servicePort: 80

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress2
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/affinity: cookie		//分离开来，只对ingress2 使用会话保持
spec:
  rules:
  - host: www2.westos.org
    http:
      paths:
      - path: /
        backend:
          serviceName: myservice2
          servicePort: 80

我们先把 daemonset 的模式切换回 deployment 的模式。

[root@server2 manifest]# kubectl apply -f ingress.yml 
ingress.networking.k8s.io/ingress1 created
ingress.networking.k8s.io/ingress2 created
[root@server2 manifest]# kubectl get ingress
NAME       CLASS    HOSTS             ADDRESS        PORTS   AGE
ingress1   <none>   www1.westos.org   172.25.254.4   80      94s
ingress2   <none>   www2.westos.org   172.25.254.4   80      94s

[root@server2 manifest]# kubectl describe ingress ingress1
Name:             ingress1
Rules:
  Host             Path  Backends
  ----             ----  --------
  www1.westos.org  
                   /   myservice:80 (10.244.141.201:80,10.244.22.6:80)
Annotations:       kubernetes.io/ingress.class: nginx		//1上没有会话保持

[root@server2 manifest]# kubectl describe ingress ingress2
Name:             ingress2
Rules:
  Host             Path  Backends
  ----             ----  --------
  www2.westos.org  
                   /   myservice2:80 (10.244.141.202:80,10.244.22.7:80)
Annotations:       kubernetes.io/ingress.class: nginx
                   nginx.ingress.kubernetes.io/affinity: cookie		// 2上有会话保持

访问www2.westos.org时，页面一直不变。
但是当我们访问 www1.westos.org 时，就会做负载均衡。

ingress的TLS配置

创建证书：

[root@server2 ~]# openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginxsvc/O=nginxsvc"	
[root@server2 ~]# ls
tls.crt  tls.key

tls加密：
[root@server2 ~]# kubectl create secret tls tls-secret --key tls.key --cert tls.crt	
secret/tls-secret created

[root@server2 ~]# kubectl get secrets 
NAME                  TYPE                                  DATA   AGE
default-token-j7pl7   kubernetes.io/service-account-token   3      11d
tls-secret            kubernetes.io/tls                     2      2m7s	/里面存放key

生效:

[root@server2 ~]# vim tls.yml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nginx-tls
spec:
  tls:
    - hosts:
      - www1.westos.org			/使用tls认证
      secretName: tls-secret
  rules:
    - host: www1.westos.org
      http:
        paths:
        - path: /
          backend:
            serviceName: myservice
            servicePort: 80

[root@server2 ~]# kubectl apply -f tls.yml 
ingress.networking.k8s.io/nginx-tls created
[root@server2 ~]# kubectl get ingress
NAME        CLASS    HOSTS             ADDRESS        PORTS     AGE
ingress1    <none>   www1.westos.org   172.25.254.4   80        18m
ingress2    <none>   www2.westos.org   172.25.254.4   80        18m
nginx-tls   <none>   www1.westos.org                  80, 443   4s

访问:

访问www1的时候会自动跳转到443端口，并使用自签名证书。

访问www2的时候就不会有反应。

进入pod去访问：

[root@server2 ~]# kubectl -n ingress-nginx exec -it ingress-nginx-controller-jskzg -- sh
/etc/nginx $ vi nginx.conf
        ## start server www1.westos.org                   
        server {                               
                server_name www1.westos.org ;
                                               
                listen 80  ;                   
                listen [::]:80  ;              
                listen 443  ssl http2 ;			//监听了443端口