在k8s上面搭建apisix網關

本地安裝apisix

使用yum安裝

yum install -y https://github.com/apache/incubator-apisix/releases/download/1.1/apisix-1.1-0.el7.noarch.rpm

本地安裝dashboard

參見：https://github.com/apache/incubator-apisix/tree/v1.1 – dashboard安裝部分
或者 https://blog.csdn.net/cyxinda/article/details/105099571–dashboard安裝部分

工作目錄結構：

如下是最終生成的工作目錄

workspace
|-- docker
|   |-- apisix
|   |   |-- apisix.dockerfile  #apisix的dockerfile
|   |   |-- config.yaml       #apisix的配置文件
|   |   |-- dashboard         #dashboard安裝目錄
|   |-- etcd
|   |   |-- etcd.conf.yml     #etcd的配置文件
|   |   |-- etcd.dockerfile   #etcd的dockerfile
|   |-- uuid                  #測試項目【uuid生成器】,普通的jar項目
|       |-- uuid-service-0.0.1-SNAPSHOT.jar
|       |-- uuid.dockerfile
|   |-- incubator-apisix-docker-master  #製作第一層docker鏡像的目錄
|-- k8s
    |-- apisix.yaml          #apisix的k8s編排文件
    |-- etcd.yaml            #etcd的k8s編排文件
    |-- uuid.yaml            #測試項目【uuid生成器】的k8s編排文件

製作apisix的docker鏡像

下載incubator-apisix-docker到workspace/docker目錄中
下載地址: https://github.com/apache/incubator-apisix-docker
build docker鏡像：
在本地目錄workspace/docker/incubator-apisix-docker-master/ 下，執行如下build命令：
```
docker build -t harbor.aipo.lenovo.com/apisix/apisix:master-alpine -f alpine/Dockerfile alpine
docker push harbor.aipo.lenovo.com/apisix/apisix:master-alpine
```
生成apisix的第一層鏡像，並保存到了本地的harbor服務器上面

製作etcd的docker鏡像

在目錄workspace/docker/etcd/ 中，複製etcd配置文件到當前目錄

cp ../incubator-apisix-docker-master/example/etcd_conf/etcd.conf.yml .

創建etcd的dockerfile：etcd.dockerfile，內容如下：

FROM bitnami/etcd:3.3.13-r80
COPY etcd.conf.yml /opt/bitnami/etcd/conf/etcd.conf.yml
EXPOSE 2379 2380

生成docker鏡像

docker build -t harbor.aipo.lenovo.com/apisix/etcd:3.3.13 -f etcd.dockerfile .
docker push harbor.aipo.lenovo.com/apisix/etcd:3.3.13

生成etcd鏡像，並將鏡像保存到harbor上面

獲取集羣dns信息

在k8s集羣的終端，獲取集羣域名
隨便找到一個部署在k8s上面的pod，進入到容器內部：
```
kubectl -n kube-system exec -it pod/metrics-server-749c9798c6-zkg2m -- /bin/sh
```
在容器內部，查看dns配置：
```
[root@k8s-1 bin]# kubectl -n kube-system exec -it pod/metrics-server-749c9798c6-zkg2m -- /bin/sh
/ # cat /etc/resolv.conf
nameserver 10.1.0.10
search kube-system.svc.cluster.local svc.cluster.local cluster.local openstacklocal
options ndots:5
/ #
```
其中kube-system.svc.cluster.local的結構是

<service_name>.<namespace>.svc.<domain>
每個部分的字段意思：

service_name: 服務名稱，就是定義 service 的時候取的名字
namespace：service 所在 namespace 的名字
domain：提供的域名後綴，比如默認的 cluster.local
可以參考：https://cizixs.com/2017/04/11/kubernetes-intro-kube-dns/

domain這個字段就是我們要找的：在本集羣中的值就是：cluster.local
nameserver 10.1.0.10也可以知道dns的域名服務的ClusterIP爲：10.1.0.10

獲取集羣dns的ClusterIP
check一下步驟一中，找到的ClusterIP是否正確
執行如下命令：

[root@k8s-1 bin]# kubectl -n kube-system get svc
NAME                               TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE
etcd                               ClusterIP   None          <none>        2379/TCP                 133d
kube-controller-manager-headless   ClusterIP   None          <none>        10252/TCP                133d
kube-dns                           ClusterIP   10.1.0.10     <none>        53/UDP,53/TCP,9153/TCP   191d
kube-scheduler-headless            ClusterIP   None          <none>        10251/TCP                133d
kubelet                            ClusterIP   None          <none>        10250/TCP                133d
kubernetes-dashboard               NodePort    10.1.213.19   <none>        443:30003/TCP            157d
metrics-server                     ClusterIP   10.1.185.71   <none>        443/TCP                  133d
tiller-deploy                      ClusterIP   10.1.55.173   <none>        44134/TCP                155d

kube-dns服務的ip就是k8s的DNS服務的ClusterIP，可以驗證步驟1找到的ip與此步驟的ip是一致的：10.1.0.10

重新制作鏡像

目的有兩個：

加入dashboard
便於修改apisix的配置文件

在workspace/docker/apisix/目錄下

拷貝incubator-apisix-docker-master中的配置文件到當前目錄

cp ../incubator-apisix-docker-master/example/apisix_conf/config.yaml .

拷貝步驟【本地安裝dashboard】中安裝好的dashboard
```
	cp -r /usr/local/apisix/dashboard .
```

修改apisix的配置文件

apisix:
  node_listen: 9080              # APISIX listening port
  node_ssl_listen: 9443
  enable_heartbeat: true
  enable_admin: true
  enable_debug: false
  enable_ipv6: true
  config_center: etcd             # etcd: use etcd to store the config value
                                  # yaml: fetch the config value from local yaml file `/your_path/conf/apisix.yaml`
  # allow_admin:                  # http://nginx.org/en/docs/http/ngx_http_access_module.html#allow
  #   - 127.0.0.0/24              # If we don't set any IP list, then any IP access is allowed by default.
  #   - "::/64"
  # port_admin: 9180              # use a separate port
  router:
    http: 'radixtree_uri'         # radixtree_uri: match route by uri(base on radixtree)
                                  # radixtree_host_uri: match route by host + uri(base on radixtree)
                                  # r3_uri: match route by uri(base on r3)
                                  # r3_host_uri: match route by host + uri(base on r3)

    ssl: 'radixtree_sni'          # r3_sni: match route by SNI(base on r3)
                                  # radixtree_sni: match route by SNI(base on radixtree)
  # stream_proxy:                 # TCP/UDP proxy
  #   tcp:                        # TCP proxy port list
  #     - 9100
  #     - 9101
  #   udp:                        # UDP proxy port list
  #     - 9200
  #     - 9211
  dns_resolver:                   # default DNS resolver, with disable IPv6 and enable local DNS
    - 10.1.0.10

  dns_resolver_valid: 30          # valid time for dns result 30 seconds

  ssl:
    enable: true
    enable_http2: true
    listen_port: 9443
    ssl_protocols: "TLSv1 TLSv1.1 TLSv1.2 TLSv1.3"
    ssl_ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA"

nginx_config:                     # config for render the template to genarate nginx.conf
  error_log: "logs/error.log"
  error_log_level: "warn"         # warn,error
  event:
    worker_connections: 10620
  http:
    access_log: "logs/access.log"
    keepalive_timeout: 60s         # timeout during which a keep-alive client connection will stay open on the server side.
    client_header_timeout: 60s     # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
    client_body_timeout: 60s       # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
    send_timeout: 10s              # timeout for transmitting a response to the client.then the connection is closed
    underscores_in_headers: "on"   # default enables the use of underscores in client request header fields
    real_ip_header: "X-Real-IP"     # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
    real_ip_from:                  # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
      - 127.0.0.1
      - 'unix:'

etcd:
  host: "http://etcd-service.saas.svc.cluster.local:12379"   # etcd address
  prefix: "/apisix"               # apisix configurations prefix
  timeout: 1                      # 1 seconds

plugins:                          # plugin list
  - example-plugin
  - limit-req
  - limit-count
  - limit-conn
  - key-auth
  - prometheus
  - node-status
  - jwt-auth
  - zipkin
  - ip-restriction
  - grpc-transcode
  - serverless-pre-function
  - serverless-post-function
  - openid-connect
  - proxy-rewrite

stream_plugins:
  - mqtt-proxy

修改兩項：

apisix.dns_resolver的值爲10.1.0.10（前面獲取到的k8s DNS的ClusterIP）
etcd.host的值爲http://etcd-service.saas.svc.cluster.local :12379
1. etcd-service是k8s啓動的etcd的service的名稱，後文中會創建
2. saas是etcd所屬的命名空間
3. cluster.local是前文中獲取的k8s集羣的域名
4. 12379是etcd的service的port端口

備註1：apisix依賴於nginx，而nginx的域名解析服務是需要獨立配置的，nginx域名解析不會自動依賴系統設置的域名解析服務器
備註2：k8s的域名解析服務，只能夠解析本域下的命名，即以cluster.local結尾的命名

創建apisix的dockerfile：apisix.dockerfile，內容如下：

FROM harbor.aipo.lenovo.com/apisix/apisix:master-alpine
COPY config.yaml /usr/local/apisix/conf/config.yaml
COPY dashboard /usr/local/apisix/dashboard
EXPOSE 9080 9443

buid鏡像

docker build -t harbor.aipo.lenovo.com/apisix/apisix:v1.1 -f apisix.dockerfile .
docker push harbor.aipo.lenovo.com/apisix/apisix:v1.1

創建k8s的編排文件

etcd編排文件

apiVersion: v1
kind: Namespace
metadata:
  name: saas
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: etcd-server
  namespace: saas #命名空間
  labels:
    app: etcd-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: etcd-server
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: etcd-server
    spec:
      containers:
      - name: etcd-server
        image: harbor.aipo.lenovo.com/apisix/etcd:3.3.13
        imagePullPolicy:  Always
        ports:
        - name: http
          containerPort: 2379 #容器對外暴露的端口
        - name: peer 
          containerPort: 2380
---
apiVersion: v1
kind: Service
metadata:
  name: etcd-service
  namespace: saas
spec:
  type: ClusterIP
  selector:
    app: etcd-server
  ports:
  - name: http
    port: 12379  #etcd的service暴露給其他服務的端口
    targetPort: 2379 #指向容器暴露的端口
#通過template.metadata.labels字段爲即將新建的Pod附加Label。在上面的例子中，新建了一個名稱爲nginx的Pod，它擁有一個鍵值對爲app:nginx的Label。
#通過spec.selector字段來指定這個RC管理哪些Pod。在上面的例子中，新建的RC會管理所有擁有app:nginxLabel的Pod。這樣的spec.selector在Kubernetes中被稱作Label Selector。

namespace爲saas

apisix的編排文件：

apiVersion: v1
kind: Namespace
metadata:
  name: saas
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apisix-server
  namespace: saas
  labels:
    app: apisix-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: apisix-server
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: apisix-server
    spec:
      containers:
      - name: apisix-server
        image: harbor.aipo.lenovo.com/apisix/apisix:v1.1
        imagePullPolicy:  Always
        ports:
        - name: http
          containerPort: 9080
        - name: https
          containerPort: 9443
---
apiVersion: v1
kind: Service
metadata:
  name: apisix-service
  namespace: saas
spec:
  type: NodePort #選擇NodePort，暴露給k8s外部請求
  selector:
    app: apisix-server
  ports:
  - port: 80
    targetPort: 9080
    nodePort: 31191 #暴露給k8s外部的服務端口

啓動服務

啓動etcd服務

[root@k8s-1 k8s]# kubectl apply -f etcd.yaml
namespace/saas created
deployment.apps/etcd-server created
service/etcd-service created

[root@k8s-1 k8s]# kubectl get pods -n saas
NAME                           READY   STATUS    RESTARTS   AGE
etcd-server-69d9fbbcd7-dmxr2   1/1     Running   0          19s

可以看到etcd的pod已經啓動了

啓動apisix服務

[root@k8s-1 k8s]# kubectl apply -f apisix.yaml
namespace/saas unchanged
deployment.apps/apisix-server created
service/apisix-service created

[root@k8s-1 k8s]# kubectl get pods -n saas
NAME                            READY   STATUS    RESTARTS   AGE
apisix-server-8b4cb7ff9-7mst5   1/1     Running   0          3m41s
etcd-server-69d9fbbcd7-dmxr2    1/1     Running   0          4m12s

  可以看到apisix的pod已經啓動了

驗證

瀏覽器打開k8s集羣的任意一臺ip地址，如http://10.110.149.172:31191/apisix/dashboard
可以在頁面上面看到

頁面上面可以正常添加route和upstream，表明搭建成功！

其他

附調試常用命令行
在調試k8s的調試過程中，用到了好多常用的k8s的命令，貼出來，供大家參考：

cd /root/username/workspace/docker/etcd
docker build -t harbor.aipo.lenovo.com/apisix/etcd:3.3.13 -f etcd.dockerfile .
docker rm -f etcd-server
docker run -it --name etcd-server \
 -p 23791:2379 \
 -p 23801:2380 \
 -d harbor.aipo.lenovo.com/apisix/etcd:3.3.13
docker ps |grep etcd-server
docker logs etcd-server
 -------------------------------------------

kubectl delete -f apisix.yaml
kubectl apply -f etcd.yaml
kubectl get pods -n saas -o wide
kubectl logs pods/etcd-server-746d5b4cf7-th44j -c etcd-server --namespace=saas # --previous 該參數可以查看上一次啓動的容器的日誌，非常有用
kubectl exec -it etcd-server-746d5b4cf7-k4bhv -c etcd-server -n saas /bin/bash 

cd /root/username/workspace/docker/apisix
docker build --no-cache -t harbor.aipo.lenovo.com/apisix/apisix:v1.1 -f apisix.dockerfile .
docker push harbor.aipo.lenovo.com/apisix/apisix:v1.1

cd /root/caoyong/apisix/k8s
kubectl get pods -n saas -o wide
kubectl get svc -n saas -o wide
kubectl delete svc apisix-service -n saas
kubectl delete pod apisix-server-88fbc7f99-gvnr4  -n saas
kubectl delete -f apisix.yaml
kubectl get endpoints -n saas -o wide
kubectl apply -f apisix.yaml
kubectl get pods -n saas -o wide
kubectl get svc -n saas -o wide
kubectl logs pods/apisix-server-8b4cb7ff9-7mst5 -c apisix-server  --namespace=saas
kubectl exec -it  apisix-server-88fbc7f99-pf8sj -c apisix-server -n saas /bin/bash

由service尋找pod的過程

kubectl get svc -n kube-system
kubectl -n kube-system describe svc/metrics-server	
kubectl -n kube-system get pod -l app=metrics-server	
kubectl -n kube-system describe pod/kubernetes-dashboard-7844b55485-c2dxm	
kubectl -n kube-system exec -it kubernetes-dashboard-7844b55485-c2dxm /bin/sh

附可調試的etcd鏡像

需要下載etcd按照源碼到本地，參考：https://github.com/etcd-io/etcd/releases/tag/v3.4.5
比如下在版本爲v3.4.5的etcd，如下將ETCD_VER替換成v3.4.5即可

GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GOOGLE_URL}

rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test

curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz

/tmp/etcd-download-test/etcd --version
/tmp/etcd-download-test/etcdctl version

創建一個工具比較齊全的docker，附centos.tools.dockerfile：

FROM centos:7.6.1810
LABEL maintainer=caoyong1
WORKDIR /root
COPY *.repo /etc/yum.repos.d/
#指定時區以及安裝各種插件
RUN cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && yum install kde-l10n-Chinese -y && yum install glibc-common -y && localedef -c -f UTF-8 -i zh_CN zh_CN.utf8 && mkdir services && yum clean all && yum makecache && yum install -y epel-release && yum clean all && yum makecache && yum -y install iputils && yum -y install net-tools.x86_64 && yum install -y redhat-lsb && yum -y install bridge-utils && yum -y install traceroute && yum -y install vim*
#指定字符集
ENV LANG zh_CN.UTF-8
ENV LANGUAGE zh_CN.UTF-8
ENV LC_ALL zh_CN.UTF-8

附本目錄中的鏡像源：wget http://mirrors.163.com/.help/CentOS7-Base-163.repo

docker build -t harbor.aipo.lenovo.com/apisix/centos:tools -f centos.tools.dockerfile .
docker push harbor.aipo.lenovo.com/apisix/centos:tools

以上面的鏡像爲基礎，創建工具比較全的etcd鏡像源，附etcd3.4.5.dockerfile：

FROM harbor.aipo.lenovo.com/apisix/centos:tools
WORKDIR /root
COPY etcd.conf.yml /opt/bitnami/etcd/conf/etcd.conf.yml
ADD  etcd-v3.4.5-linux-amd64.tar.gz .
WORKDIR /root/etcd-v3.4.5-linux-amd64
EXPOSE 2379 2380
ENV ALLOW_NONE_AUTHENTICATION=yes
ENTRYPOINT [ "./etcd","--config-file=/opt/bitnami/etcd/conf/etcd.conf.yml"

在k8s上面搭建apisix網關

本地安裝apisix

本地安裝dashboard

工作目錄結構：

製作apisix的docker鏡像

製作etcd的docker鏡像

獲取集羣dns信息

重新制作鏡像

創建k8s的編排文件

啓動服務

驗證

其他

docker網絡模式macvlan驗證

在k8s上面搭建apisix網關

kubectl使用命令記錄

Jenkins 常用 REST API介紹(Java 客戶端)

k8s CI/CD板書

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結