K8s资源第四篇（Ingress-nginx）

Ingress：

 Kubernetes 暴露服务的有三种方式，分别为 LoadBlancer Service、NodePort Service、Ingress。

 官网对 Ingress 的定义为管理对外服务到集群内服务之间规则的集合，通俗点讲就是它定义规则来允许进入集群的请求被转发到集群中对应服务上，从来实现服务暴漏。 Ingress 能把集群内 Service 配置成外网能够访问的 URL，流量负载均衡，终止SSL，提供基于域名访问的虚拟主机等等。

 可以把Ingress理解为一个调度器。

原因：

Pod漂移问题：

 Kubernetes 具有强大的副本控制能力，能保证在任意副本（Pod）挂掉时自动从其他机器启动一个新的，还可以动态扩容等，通俗地说，这个 Pod 可能在任何时刻出现在任何节点上，也可能在任何时刻死在任何节点上；那么自然随着 Pod 的创建和销毁，Pod IP 肯定会动态变化；那么如何把这个动态的 Pod IP 暴露出去？这里借助于 Kubernetes 的 Service 机制，Service 可以以标签的形式选定一组带有指定标签的 Pod，并监控和自动负载他们的 Pod IP，那么我们向外暴露只暴露 Service IP 就行了；这就是 NodePort 模式：即在每个节点上开起一个端口，然后转发到内部 Pod IP 上，如下图所示：

 此时的访问方式：http://nodeip:nodeport/

端口管理问题：

 采用 NodePort 方式暴露服务面临问题是，服务一旦多起来，NodePort 在每个节点上开启的端口会及其庞大，而且难以维护；这时，我们可以能否使用一个Nginx直接对内进行转发呢？众所周知的是，Pod与Pod之间是可以互相通信的，而Pod是可以共享宿主机的网络名称空间的，也就是说当在共享网络名称空间时，Pod上所监听的就是Node上的端口。那么这又该如何实现呢？简单的实现就是使用 DaemonSet 在每个 Node 上监听 80，然后写好规则，因为 Nginx 外面绑定了宿主机 80 端口（就像 NodePort），本身又在集群内，那么向后直接转发到相应 Service IP 就行了，如下图所示：

域名分配及动态更新问题：

 从上面的方法，采用 Nginx-Pod 似乎已经解决了问题，但是其实这里面有一个很大缺陷：当每次有新服务加入又该如何修改 Nginx 配置呢？？我们知道使用Nginx可以通过虚拟主机域名进行区分不同的服务，而每个服务通过upstream进行定义不同的负载均衡池，再加上location进行负载均衡的反向代理，在日常使用中只需要修改nginx.conf即可实现，那在K8S中又该如何实现这种方式的调度呢？？？

 假设后端的服务初始服务只有ecshop，后面增加了bbs和member服务，那么又该如何将这2个服务加入到Nginx-Pod进行调度呢？总不能每次手动改或者Rolling Update 前端 Nginx Pod 吧！！此时 Ingress 出现了，如果不算上面的Nginx，Ingress 包含两大组件：Ingress Controller 和 Ingress。

Ingress Controller：

• Nginx：通过nginx代理到后端的资源。
• Traefick：通过treafik代理到后端的资源，Traefik 是一款开源的反向代理与负载均衡工具，它可以监听后端的变化并自动更新服务配置。
• Envoy：通过Envoy代理后端的资源，Envoy具有一些强大的功能，如断路器、插件式的处理器链，以及可观测性和可调试性。

Ingress-Nginx：

一个简单的工作原理图：

• Nginx 对后端运行的服务（Service1、Service2）提供反向代理。
• 在配置文件中配置了域名与后端服务 Endpoints 的对应关系。
• 客户端通过使用 DNS 服务或者直接配置本地的 hosts 文件，将域名都映射到 Nginx 代理服务器。
• 当客户端访问 service1.com 时，浏览器会把包含域名的请求发送给 nginx 服务器，nginx 服务器根据传来的域名，选择对应的 Service，这里就是选择 Service 1 后端服务，然后根据一定的负载均衡策略，选择 Service1 中的某个容器接收来自客户端的请求并作出响应。

Github：https://github.com/kubernetes/Ingress-nginx/

官方文档：https://kubernetes.io/zh/docs/concepts/services-networking/ingress/

部署ingress-nginx：

#使用Git把项目克隆到本地。
[root@k8smaster data]# git clone https://github.com/kubernetes/ingress-nginx.git
Cloning into 'ingress-nginx'...
remote: Enumerating objects: 669, done.
remote: Counting objects: 100% (669/669), done.
remote: Compressing objects: 100% (497/497), done.
remote: Total 84579 (delta 144), reused 646 (delta 137), pack-reused 83910
Receiving objects: 100% (84579/84579), 102.78 MiB | 118.00 KiB/s, done.
Resolving deltas: 100% (46038/46038), done.
[root@k8smaster data]# cd 

#创建一个ingress目录，并把主要的配置文件拷贝过来。
[root@k8smaster ~]# mkdir ingress
[root@k8smaster ~]# cd ingress/
[root@k8smaster ingress]# cp /data/ingress-nginx/deploy/static/mandatory.yaml  .
[root@k8smaster ingress]# cp /data/ingress-nginx/deploy/static/provider/baremetal/service-nodeport.yaml  .

#修改配置清单。
[root@k8smaster ingress]# sed  -i '/targetPort: 80/a\ \ \ \ \ \ nodePort: 30080' service-nodeport.yaml
[root@k8smaster ingress]# sed  -i '/targetPort: 443/a\ \ \ \ \ \ nodePort: 30443' service-nodeport.yaml

#创建资源。
[root@k8smaster ingress]# kubectl apply  -f mandatory.yaml 
namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created
[root@k8smaster ingress]# kubectl apply  -f service-nodeport.yaml 
service/ingress-nginx created

#手动创建一个nginx Pod，一个service。
#nginx-svc代理nginx-dm。
[root@k8smaster ingress]# vim ingress-nginx-demo.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-dm
spec:
  replicas: 2
  template:
    metadata:
      labels:
        name: nginx
    spec:
      containers:
      - name: myapp
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
spec:
  selector:
    name: nginx
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP

#创建Ingress   
[root@k8smaster ingress]# vim ingress-nginx-http.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-nginx-http-demo
spec:
  rules:
    - host: abc1.tt.dev
      http:
        paths:
        - path: /
          backend:
            serviceName: nginx-svc
            servicePort: 80

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-nginx-https-demo
spec:
  tls:
    - hosts:
      - abc2.tt.dev
      secretName: tls-secret
  rules:
    - host: abc2.tt.dev
      http:
        paths:
        - path: /
          backend:
            serviceName: nginx-svc
            servicePort: 80

[root@k8smaster ingress]# kubectl apply -f ingress-nginx-demo.yaml 
deployment.extensions/nginx-dm created
service/nginx-svc created
[root@k8smaster ingress]# kubectl apply -f ingress-nginx-http.yaml 
ingress.extensions/ingress-nginx-http-demo created
ingress.extensions/ingress-nginx-https-demo created

#使用openssl生产证书，如果没有openssl，则使用yum去安装一个。
[root@k8smaster ingress]# openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/C=CN/O=DevOps/CN=abc2.tt.dev"
Generating a 2048 bit RSA private key
.............................................................................................+++
................................................................................................................................+++
writing new private key to 'tls.key'
-----
#创建secret
[root@k8smaster ingress]# kubectl create secret tls tls-secret --key tls.key --cert tls.crt

#查看Pod的详细信息。
[root@k8smaster ingress]# kubectl get pods -o wide
NAME                       READY   STATUS    RESTARTS   AGE     IP             NODE       NOMINATED NODE   READINESS GATES
nginx-dm-7c46ff4c5-l8snl   1/1     Running   0          2m39s   10.244.1.170   k8snode1   <none>           <none>
nginx-dm-7c46ff4c5-vggl2   1/1     Running   0          2m39s   10.244.2.176   k8snode2   <none>           <none>
#查看Deployment信息。
[root@k8smaster ingress]# kubectl get deployment
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
nginx-dm   2/2     2            2           6m36s
#查看service信息。
[root@k8smaster ingress]# kubectl get svc 
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP   48d
nginx-svc    ClusterIP   10.99.53.204   <none>        80/TCP    2m45s
#发现nginx-svc的Endpoints已经绑定到了Pod。
[root@k8smaster ingress]# kubectl describe svc nginx-svc
Name:              nginx-svc
Namespace:         default
Labels:            <none>
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                    {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"nginx-svc","namespace":"default"},"spec":{"ports":[{"port":80,"pr...
Selector:          name=nginx
Type:              ClusterIP
IP:                10.99.53.204
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.1.170:80,10.244.2.176:80
Session Affinity:  None
Events:            <none>
[root@k8smaster ingress]# kubectl get svc -n ingress-nginx
NAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx   NodePort   10.103.152.44   <none>        80:30080/TCP,443:30443/TCP   4m8s

#并且ingress-nginx的Endpoints也都成功绑定到。
[root@k8smaster ingress]# kubectl describe svc ingress-nginx -n ingress-nginx
Name:                     ingress-nginx
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/par...
Selector:                 app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
Type:                     NodePort
IP:                       10.103.152.44
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  30080/TCP
Endpoints:                10.244.1.169:80
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  30443/TCP
Endpoints:                10.244.1.169:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

#查看创建的ingress
[root@k8smaster ingress]# kubectl get ingress
NAME                       HOSTS               ADDRESS         PORTS     AGE
ingress-nginx-http-demo    abc1.tt.dev         10.103.152.44   80        4m29s
ingress-nginx-https-demo   abc2.tt.dev         10.103.152.44   80, 443   4m29s
nginx-deployment           nginx.example.com   10.103.152.44   80        45d
[root@k8smaster ingress]# kubectl describe  ingress ingress-nginx-http-demo 
Name:             ingress-nginx-http-demo
Namespace:        default
Address:          10.103.152.44
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host         Path  Backends
  ----         ----  --------
  abc1.tt.dev  
                /   nginx-svc:80 (10.244.1.170:80,10.244.2.176:80)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"ingress-nginx-http-demo","namespace":"default"},"spec":{"rules":[{"host":"abc1.tt.dev","http":{"paths":[{"backend":{"serviceName":"nginx-svc","servicePort":80},"path":"/"}]}}]}}

Events:
  Type    Reason  Age    From                      Message
  ----    ------  ----   ----                      -------
  Normal  CREATE  4m53s  nginx-ingress-controller  Ingress default/ingress-nginx-http-demo
  Normal  UPDATE  4m22s  nginx-ingress-controller  Ingress default/ingress-nginx-http-demo
[root@k8smaster ingress]# kubectl describe  ingress ingress-nginx-https-demo 
Name:             ingress-nginx-https-demo
Namespace:        default
Address:          10.103.152.44
Default backend:  default-http-backend:80 (<none>)
TLS:
  tls-secret terminates abc2.tt.dev
Rules:
    Host         Path  Backends
    ----         ----  --------
abc2.tt.dev  
                /   nginx-svc:80 (10.244.1.170:80,10.244.2.176:80)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"ingress-nginx-https-demo","namespace":"default"},"spec":{"rules":[{"host":"abc2.tt.dev","http":{"paths":[{"backend":{"serviceName":"nginx-svc","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["abc2.tt.dev"],"secretName":"tls-secret"}]}}

Events:
  Type    Reason  Age    From                      Message
  ----    ------  ----   ----                      -------
  Normal  CREATE  4m57s  nginx-ingress-controller  Ingress default/ingress-nginx-https-demo
  Normal  UPDATE  4m26s  nginx-ingress-controller  Ingress default/ingress-nginx-https-demo

#查看nginx的配置文件。
[root@k8smaster ingress]# kubectl exec -it -n ingress-nginx $pod_ingress -- cat /etc/nginx/nginx.conf  | grep abc
    ## start server abc1.tt.dev
	    server_name abc1.tt.dev ;
    ## end server abc1.tt.dev
    ## start server abc2.tt.dev
	    server_name abc2.tt.dev ;
    ## end server abc2.tt.dev

#由于我们的Ingress-nginx的service为Nodeport，所以需要修改abc1.tt.dev和abc2.tt.dev域名对应每个节点的IP。
[root@k8smaster ingress]# vim /etc/hosts
192.168.43.45 k8smaster abc1.tt.dev abc2.tt.dev
192.168.43.136 k8snode1 abc1.tt.dev abc2.tt.dev
192.168.43.176 k8snode2 abc1.tt.dev abc2.tt.dev

#访问测试。
[root@k8smaster ingress]# curl http://abc1.tt.dev:30080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@k8smaster ingress]# curl https://abc2.tt.dev:30443 -k
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

密码验证：

[root@k8smaster ingress]# vim ingress-with-auth.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-with-auth
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo'
spec:
  rules:
  - host: auth.tt.dev
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-svc
          servicePort: 80
[root@k8smaster ingress]# yum -y install httpd-tools
[root@k8smaster ingress]# htpasswd -bc auth testuser pswd  #用户名testuser，密码pswd。
Adding password for user testuser
[root@k8smaster ingress]# docker run -it --rm -v $(pwd):/data -w /data jess/htpasswd -bc auth testuser pswd
Unable to find image 'jess/htpasswd:latest' locally
latest: Pulling from jess/htpasswd
Image docker.io/jess/htpasswd:latest uses outdated schema1 manifest format. Please upgrade to a schema2 image for better future compatibility. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
e7c96db7181b: Pull complete 
92704bed51b2: Pull complete 
Digest: sha256:4ef4db77006976d385eb487877def47c29b68cf8d784242fb9114e90463ce2d0
Status: Downloaded newer image for jess/htpasswd:latest
Adding password for user testuser
[root@k8smaster ingress]# kubectl create secret generic basic-auth --from-file=auth
secret/basic-auth created
[root@k8smaster ingress]# kubectl get secret basic-auth -o yaml
apiVersion: v1
data:
  auth: dGVzdHVzZXI6JGFwcjEkL2V1MkN5VXMkY0J1S3dhRlVMRnE0MjVkZTdIVGREMQo=
kind: Secret
metadata:
  creationTimestamp: "2019-12-16T07:49:35Z"
  name: basic-auth
  namespace: default
  resourceVersion: "383740"
  selfLink: /api/v1/namespaces/default/secrets/basic-auth
  uid: 0d972bb5-7219-4f98-87e5-7c6f55214e0d
type: Opaque
[root@k8smaster ingress]# kubectl apply -f ingress-with-auth.yaml 
ingress.extensions/ingress-with-auth created
[root@k8smaster ingress]# vim /etc/hosts
192.168.43.45 k8smaster abc1.tt.dev abc2.tt.dev auth.tt.dev
192.168.43.136 k8snode1 abc1.tt.dev abc2.tt.dev auth.tt.dev
192.168.43.176 k8snode2 abc1.tt.dev abc2.tt.dev auth.tt.dev

#测试验证：
[root@k8smaster ingress]# curl http://auth.tt.dev:30080   #访问被拒绝了。
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>openresty/1.15.8.2</center>
</body>
</html>
#我们再指定用户名和密码，发现访问成功。
[root@k8smaster ingress]# curl http://auth.tt.dev:30080 -u 'testuser:pswd'
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

参考文档：https://www.cnblogs.com/linuxk/p/9706720.html

参考文档：https://blog.csdn.net/shida_csdn/article/details/84032019

参考文档：https://blog.csdn.net/qq_37950254/article/details/89486057