kolla-ansible部署all-in-one單節點openstack

kolla ansible簡介

kolla 的使命是爲 openstack 雲平臺提供生產級別的、開箱即用的交付能力。kolla 的基本思想是一切皆容器，將所有服務基於 Docker 運行，並且保證一個容器只跑一個服務（進程），做到最小粒度的運行 docker。

kolla 要實現 openetack 部署總體上分爲兩步，第一步是製作 docker 鏡像，第二步是編排部署。因此，kolla 項目又被分爲兩個小項目：kolla、kolla-ansible 。

kolla-ansible項目
https://github.com/openstack/kolla-ansible

kolla項目
https://tarballs.opendev.org/openstack/kolla/

dockerhub鏡像地址
https://hub.docker.com/u/kolla/

安裝環境準備

官方部署文檔：
https://docs.openstack.org/kolla-ansible/train/user/quickstart.html

kolla安裝節點要求：

2 network interfaces
8GB main memory
40GB disk space

本次部署train版all-in-one單節點，使用一臺centos7.8 minimal節點進行部署，該節點同時作爲控制節點、計算節點、網絡節點和cinder存儲節點使用，同時也是kolla ansible的部署節點。

kolla的安裝要求目標機器至少兩塊網卡，本次安裝使用2塊網卡對應管理網絡和外部網絡兩個網絡平面，在vmware workstation虛擬機新增一塊網卡ens37：

ens33，NAT模式，管理網絡，正常配置靜態IP即可。租戶網絡與該網絡複用，租戶vm網絡不單獨創建網卡
ens37，橋接模式，外部網絡，無需配置IP地址，這個其實是讓neutron的br-ex 綁定使用，虛擬機通過這塊網卡訪問外網。

網卡配置信息，大部分爲默認參數

[root@kolla ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=a41355ae-f475-39d7-9e61-eb5f8f19f881
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.93.30
PREFIX=24
GATEWAY=192.168.93.2
DNS1=114.114.114.114
DNS2=8.8.8.8
IPV6_PRIVACY=no

[root@kolla ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens37
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
NAME=ens37
UUID=553a2dd0-b53e-417e-98a9-9a7a6a44a53c
DEVICE=ens37
ONBOOT=yes

如果啓用cinder還需要額外添加磁盤，這裏添加一塊sdb盤並創建爲pv和vg，使用lvm作爲cinder的後端存儲：

pvcreate /dev/sdb
vgcreate cinder-volumes /dev/sdb

注意卷組名稱爲cinder-volumes，默認與後面的globals.yml中定義一致。

[root@kolla ~]# cat /etc/kolla/globals.yml | grep cinder_volume_group
#cinder_volume_group: "cinder-volumes"

初次嘗試kolla ansible可能會遇到很多報錯，下面的部署流程可能與官方有一定出入，但是規避了大部分問題。

部署kolla ansible

配置主機名,kolla預檢查時rabbitmq可能需要能夠進行主機名解析

hostnamectl set-hostname kolla

安裝依賴

yum install -y python-devel libffi-devel gcc openssl-devel libselinux-python

安裝 Ansible，注意版本，默認2.9應該可以滿足要求

yum install -y ansible

配置阿里雲pip源，否則pip安裝時會很慢

mkdir ~/.pip
cat > ~/.pip/pip.conf << EOF 
[global]
trusted-host=mirrors.aliyun.com
index-url=https://mirrors.aliyun.com/pypi/simple/
EOF

安裝 kolla-ansible

kolla版本與openstack版本對應關係：https://releases.openstack.org/teams/kolla.html

yum install -y epel-release
yum install -y python-pip
pip install -U pip
pip install kolla-ansible==9.1.0 --ignore-installed PyYAML

複製 kolla-ansible配置文件到當前環境

mkdir -p /etc/kolla
chown $USER:$USER /etc/kolla

##Copy globals.yml and passwords.yml
cp -r /usr/share/kolla-ansible/etc_examples/kolla/* /etc/kolla

##Copy all-in-one and multinode inventory files
cp /usr/share/kolla-ansible/ansible/inventory/* .

修改ansible配置文件

$ vim /etc/ansible/ansible.cfg
[defaults]
host_key_checking=False
pipelining=True
forks=100

默認有all-in-one和multinode兩個inventory文件，這裏使用all-in-one，來規劃集羣角色，可以看到所有節點都是同一個節點kolla。

# sed -i 's#localhost       ansible_connection=local#kolla#g' all-in-one 

#查看修改後的配置，其他默認即可
# cat all-in-one | more
[control]
kolla

[network]
kolla

[compute]
kolla

[storage]
kolla

[monitoring]
kolla

[deployment]
kolla
...

配置主機名解析，實際在環境預配置時kolla會自動添加解析到/etc/hosts

cat >> /etc/hosts <<EOF
192.168.93.30 kolla
EOF

配置ssh免密

ssh-keygen
ssh-copy-id root@kolla

檢查inventory配置是否正確，執行：

ansible -i all-in-one all -m ping

生成openstack組件用到的密碼，該操作會填充/etc/kolla/passwords.yml，該文件中默認參數爲空。

kolla-genpwd

修改keystone_admin_password，可以修改爲自定義的密碼方便後續horizon登錄，這裏改爲kolla。

$ sed -i 's#keystone_admin_password:.*#keystone_admin_password: kolla#g' /etc/kolla/passwords.yml 

$ cat /etc/kolla/passwords.yml | grep keystone_admin_password
keystone_admin_password: kolla

修改全局配置文件globals.yml，該文件用來控制安裝哪些組件，以及如何配置組件，由於全部是註釋，這裏直接追加進去，也可以逐個找到對應項進行修改。

cp /etc/kolla/globals.yml{,.bak}

cat >> /etc/kolla/globals.yml <<EOF

#version
kolla_base_distro: "centos"
kolla_install_type: "binary"
openstack_release: "train"

#vip
kolla_internal_vip_address: "192.168.93.100"

#docker registry
docker_registry: "registry.cn-shenzhen.aliyuncs.com"
docker_namespace: "kollaimage"

#network
network_interface: "ens33"
neutron_external_interface: "ens37"
neutron_plugin_agent: "openvswitch"
enable_neutron_provider_networks: "yes"

#storage
enable_cinder: "yes"
enable_cinder_backend_lvm: "yes"

#virt_type
nova_compute_virt_type: "qemu"
EOF

參數說明：

kolla_base_distro: kolla鏡像基於不同linux髮型版構建，主機使用centos這裏對應使用centos類型的docker鏡像即可。
kolla_install_type: kolla鏡像基於binary二進制和source源碼兩種類型構建，實際部署使用binary即可。
openstack_release: openstack版本可自定義，會從dockerhub拉取對應版本的鏡像
kolla_internal_vip_address: 單節點部署kolla也會啓用haproxy和keepalived，方便後續擴容爲高可用集羣，該地址是ens33網卡網絡中的一個可用IP。
docker_registry: 默認從dockerhub拉取鏡像，這裏使用阿里雲鏡像倉庫，也可以本地搭建倉庫，提前推送鏡像上去。但該倉庫目前只有train和ussuri版本的鏡像，如何自己推送鏡像參考該博客的其他文章。
docker_namespace: 阿里雲kolla鏡像倉庫所在的命名空間，dockerhub官網默認是kolla。
network_interface: 管理網絡的網卡
neutron_external_interface: 外部網絡的網卡
neutron_plugin_agent: 默認啓用openvswitch
enable_neutron_provider_networks: 啓用外部網絡
enable_cinder: 啓用cinder
enable_cinder_backend_lvm: 指定cinder後端存儲爲lvm
nova_compute_virt_type: 由於使用vmware安裝，要改爲qemu，否則創建虛擬機失敗，生產部署默認使用kvm。

上面部分參數可能有默認配置，也可以不用明確開啓，比如neutron_plugin_agent。

有些參數也可以在部署後配置，比如nova_compute_virt_type，找到配置文件修改，並重啓對應組件容器即可：

[root@kolla ~]# cat /etc/kolla/nova-compute/nova.conf |grep virt_type
#virt_type = kvm
virt_type = qemu

[root@kolla ~]# docker restart nova_compute

修改docker官方yum源爲阿里雲yum源，另外配置docker鏡像加速，指定使用阿里雲鏡像加速。

$ vim /usr/share/kolla-ansible/ansible/roles/baremetal/defaults/main.yaml
docker_yum_url: "https://mirrors.aliyun.com/docker-ce/linux/{{ ansible_distribution | lower }}"
docker_custom_config: {"registry-mirrors": ["https://uyah70su.mirror.aliyuncs.com"]}

部署openstack組件

部署openstack

#預配置，安裝docker、docker sdk、關閉防火牆、配置時間同步等
kolla-ansible -i ./all-in-one bootstrap-servers

#部署前環境檢查
kolla-ansible -i ./all-in-one prechecks

#拉取鏡像，也可省略該步驟，默認會自動拉取
kolla-ansible -i ./all-in-one pull

#執行實際部署，拉取鏡像，運行對應組件容器
kolla-ansible -i ./all-in-one deploy

以上部署沒有報錯中斷說明部署成功，所有openstack組件以容器方式運行，查看容器

[root@kolla ~]# docker ps -a
CONTAINER ID        IMAGE                                                                                        COMMAND                  CREATED             STATUS              PORTS               NAMES
325c17a52c79        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-chrony:train                      "dumb-init --single-…"   36 hours ago        Up 25 hours                             chrony
6218d98755ee        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-cron:train                        "dumb-init --single-…"   36 hours ago        Up 25 hours                             cron
02b6598c1089        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-kolla-toolbox:train               "dumb-init --single-…"   36 hours ago        Up 25 hours                             kolla_toolbox
8572e445abad        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-fluentd:train                     "dumb-init --single-…"   36 hours ago        Up 25 hours                             fluentd
f11a103c5ade        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-openstack-base:train              "dumb-init --single-…"   44 hours ago        Up 25 hours                             client
5c91def3c963        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-horizon:train                     "dumb-init --single-…"   44 hours ago        Up 25 hours                             horizon
e024bd4f5dd3        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-heat-engine:train                 "dumb-init --single-…"   44 hours ago        Up 25 hours                             heat_engine
2d1491bd9e1a        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-heat-api-cfn:train                "dumb-init --single-…"   44 hours ago        Up 25 hours                             heat_api_cfn
eeefcfb31a61        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-heat-api:train                    "dumb-init --single-…"   44 hours ago        Up 25 hours                             heat_api
9b51b53448fc        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-neutron-metadata-agent:train      "dumb-init --single-…"   44 hours ago        Up 25 hours                             neutron_metadata_agent
9f88a6c0cf31        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-neutron-l3-agent:train            "dumb-init --single-…"   44 hours ago        Up 25 hours                             neutron_l3_agent
a419cb3270a6        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-neutron-dhcp-agent:train          "dumb-init --single-…"   44 hours ago        Up 25 hours                             neutron_dhcp_agent
959f6faba972        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-neutron-openvswitch-agent:train   "dumb-init --single-…"   44 hours ago        Up 25 hours                             neutron_openvswitch_agent
cc1b081cf876        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-neutron-server:train              "dumb-init --single-…"   44 hours ago        Up 25 hours                             neutron_server
eea1a87feb43        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-openvswitch-vswitchd:train        "dumb-init --single-…"   44 hours ago        Up 25 hours                             openvswitch_vswitchd
376f81bf75a2        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-openvswitch-db-server:train       "dumb-init --single-…"   44 hours ago        Up 25 hours                             openvswitch_db
c68fd9a92d73        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-nova-compute:train                "dumb-init --single-…"   44 hours ago        Up 25 hours                             nova_compute
2492e2a32c80        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-nova-libvirt:train                "dumb-init --single-…"   44 hours ago        Up 25 hours                             nova_libvirt
3802d199b29f        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-nova-ssh:train                    "dumb-init --single-…"   44 hours ago        Up 25 hours                             nova_ssh
1281c311ecd4        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-nova-novncproxy:train             "dumb-init --single-…"   44 hours ago        Up 25 hours                             nova_novncproxy
2e8c8478116b        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-nova-conductor:train              "dumb-init --single-…"   44 hours ago        Up 25 hours                             nova_conductor
950feb59b549        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-nova-api:train                    "dumb-init --single-…"   44 hours ago        Up 25 hours                             nova_api
49497e664922        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-nova-scheduler:train              "dumb-init --single-…"   44 hours ago        Up 25 hours                             nova_scheduler
f5eb37b48f7d        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-placement-api:train               "dumb-init --single-…"   44 hours ago        Up 25 hours                             placement_api
54cd0e3be101        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-cinder-backup:train               "dumb-init --single-…"   44 hours ago        Up 25 hours                             cinder_backup
b4efa4449e7f        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-cinder-volume:train               "dumb-init --single-…"   44 hours ago        Up 25 hours                             cinder_volume
159b669d2fd3        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-cinder-scheduler:train            "dumb-init --single-…"   44 hours ago        Up 25 hours                             cinder_scheduler
9fc7e6a4cb25        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-cinder-api:train                  "dumb-init --single-…"   44 hours ago        Up 25 hours                             cinder_api
b3f8f711f2b1        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-glance-api:train                  "dumb-init --single-…"   44 hours ago        Up 25 hours                             glance_api
760e92d698e2        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-keystone-fernet:train             "dumb-init --single-…"   44 hours ago        Up 25 hours                             keystone_fernet
95f235c4ac10        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-keystone-ssh:train                "dumb-init --single-…"   44 hours ago        Up 25 hours                             keystone_ssh
03306334ce19        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-keystone:train                    "dumb-init --single-…"   44 hours ago        Up 25 hours                             keystone
5173d4191567        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-rabbitmq:train                    "dumb-init --single-…"   44 hours ago        Up 25 hours                             rabbitmq
eb6bca26f6ce        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-tgtd:train                        "dumb-init --single-…"   44 hours ago        Up 25 hours                             tgtd
79fac2ca1b19        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-iscsid:train                      "dumb-init --single-…"   44 hours ago        Up 25 hours                             iscsid
4a3fcefc7009        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-memcached:train                   "dumb-init --single-…"   44 hours ago        Up 25 hours                             memcached
0773eaf446e4        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-mariadb:train                     "dumb-init -- kolla_…"   44 hours ago        Up 25 hours                             mariadb
77f0beaa28e5        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-keepalived:train                  "dumb-init --single-…"   44 hours ago        Up 25 hours                             keepalived
b02b744d2da3        registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-haproxy:train                     "dumb-init --single-…"   44 hours ago        Up 25 hours                             haproxy

確認沒有Exited等異常狀態的容器

[root@kolla ~]# docker ps -a  | grep -v Up

本次部署運行了39個容器

[root@localhost kolla-env]# docker ps -a | wc -l
39

查看拉取的鏡像，發現鏡像數量與容器數量是一致的。

[root@kolla ~]# docker images
REPOSITORY                                                                             TAG                 IMAGE ID            CREATED             SIZE
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-glance-api                  train               aec757c5908a        2 days ago          1.05GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-keystone-ssh                train               2c95619322ed        2 days ago          1.04GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-keystone-fernet             train               918564aa9c01        2 days ago          1.04GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-keystone                    train               8d5f3ca2a73c        2 days ago          1.04GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-cinder-api                  train               500910236e85        2 days ago          1.19GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-cinder-volume               train               f76ebe1e133d        2 days ago          1.14GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-cinder-backup               train               19342786a92c        2 days ago          1.13GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-cinder-scheduler            train               920630f0ea6c        2 days ago          1.11GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-heat-api                    train               517f6a0643ee        2 days ago          1.07GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-heat-api-cfn                train               2d46b91d44ef        2 days ago          1.07GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-heat-engine                 train               ab570c135dbc        2 days ago          1.07GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-horizon                     train               a00ddb359ea5        2 days ago          1.2GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-fluentd                     train               6a5b7be2551b        2 days ago          697MB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-cron                        train               0f784cd532e2        2 days ago          408MB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-chrony                      train               374dabc62868        2 days ago          408MB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-iscsid                      train               575873f9e4b8        2 days ago          413MB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-haproxy                     train               9cf840548535        2 days ago          433MB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-keepalived                  train               b2a20ccd7d6a        2 days ago          414MB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-openstack-base              train               c35001fb182b        3 days ago          920MB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-nova-compute                train               93be43a73a3e        5 days ago          1.85GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-placement-api               train               26f8c88c3c50        5 days ago          1.05GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-nova-api                    train               2a9d3ea95254        5 days ago          1.08GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-nova-novncproxy             train               e6acfbe47b2b        5 days ago          1.05GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-nova-conductor              train               836a9f775263        5 days ago          1.05GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-nova-ssh                    train               f89a813f3902        5 days ago          1.05GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-nova-scheduler              train               8061eaa33d21        5 days ago          1.05GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-openvswitch-vswitchd        train               2b780c8075c6        5 days ago          425MB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-openvswitch-db-server       train               86168147b086        5 days ago          425MB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-rabbitmq                    train               19cd34b4f503        5 days ago          487MB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-mariadb                     train               882472a192b5        6 days ago          593MB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-neutron-dhcp-agent          train               a007b53f0507        7 days ago          1.04GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-neutron-metadata-agent      train               8bcff22221bd        7 days ago          1.04GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-nova-libvirt                train               539673da5c25        7 days ago          1.25GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-kolla-toolbox               train               a18a474c65ea        7 days ago          842MB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-tgtd                        train               ad5380187ca9        7 days ago          383MB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-memcached                   train               1fcf18645254        7 days ago          408MB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-neutron-server              train               539cfb7c1fd2        8 days ago          1.08GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-neutron-openvswitch-agent   train               95113c0f5b8c        8 days ago          1.08GB
registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-neutron-l3-agent            train               fbe9385f49ca        8 days ago          1.08GB

查看cinder使用的卷，自動創建了lvm

[root@kolla ~]# lsblk | grep cinder
├─cinder--volumes-cinder--volumes--pool_tmeta 253:3    0   20M  0 lvm  
│ └─cinder--volumes-cinder--volumes--pool     253:5    0   19G  0 lvm  
└─cinder--volumes-cinder--volumes--pool_tdata 253:4    0   19G  0 lvm  
  └─cinder--volumes-cinder--volumes--pool     253:5    0   19G  0 lvm  
                         
[root@kolla ~]# lvs | grep cinder
  cinder-volumes-pool cinder-volumes twi-a-tz--  19.00g             0.00   10.55

查看網卡狀態

[root@kolla ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:0c:4e:fe brd ff:ff:ff:ff:ff:ff
    inet 192.168.93.30/24 brd 192.168.93.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.93.100/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::7a6c:d06c:ee49:4cd5/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP group default qlen 1000
    link/ether 00:0c:29:0c:4e:08 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::20c:29ff:fe0c:4e08/64 scope link 
       valid_lft forever preferred_lft forever
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:2a:d9:93:52 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:2aff:fed9:9352/64 scope link 
       valid_lft forever preferred_lft forever
6: veth0c46c6a@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default 
    link/ether 1a:ce:d7:61:d0:cc brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::18ce:d7ff:fe61:d0cc/64 scope link 
       valid_lft forever preferred_lft forever
7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether de:e5:b7:4d:e8:b8 brd ff:ff:ff:ff:ff:ff
11: br-int: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
    link/ether 52:14:05:ba:ce:4c brd ff:ff:ff:ff:ff:ff
13: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d2:5b:76:f5:01:49 brd ff:ff:ff:ff:ff:ff
14: br-ex: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0c:29:0c:4e:08 brd ff:ff:ff:ff:ff:ff
22: qbr2749f64b-1f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 3a:0d:ad:56:9d:9d brd ff:ff:ff:ff:ff:ff
23: qvo2749f64b-1f@qvb2749f64b-1f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether c2:c5:8b:a6:72:8b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c0c5:8bff:fea6:728b/64 scope link 
       valid_lft forever preferred_lft forever
24: qvb2749f64b-1f@qvo2749f64b-1f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master qbr2749f64b-1f state UP group default qlen 1000
    link/ether 3a:0d:ad:56:9d:9d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::380d:adff:fe56:9d9d/64 scope link 
       valid_lft forever preferred_lft forever
25: tap2749f64b-1f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master qbr2749f64b-1f state UNKNOWN group default qlen 1000
    link/ether fe:16:3e:94:b5:71 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe94:b571/64 scope link 
       valid_lft forever preferred_lft forever
26: qbr0a14e63d-2e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 02:f9:32:c0:f4:b7 brd ff:ff:ff:ff:ff:ff
27: qvo0a14e63d-2e@qvb0a14e63d-2e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 76:86:46:4c:4f:61 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7486:46ff:fe4c:4f61/64 scope link 
       valid_lft forever preferred_lft forever
28: qvb0a14e63d-2e@qvo0a14e63d-2e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master qbr0a14e63d-2e state UP group default qlen 1000
    link/ether 02:f9:32:c0:f4:b7 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f9:32ff:fec0:f4b7/64 scope link 
       valid_lft forever preferred_lft forever
29: tap0a14e63d-2e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master qbr0a14e63d-2e state UNKNOWN group default qlen 1000
    link/ether fe:16:3e:ee:08:6b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:feee:86b/64 scope link 
       valid_lft forever preferred_lft forever

另外需要注意，不要在該節點安裝libvirt等工具，這些工具安裝後可能會啓用libvirtd和iscsid.sock等服務，kolla已經在容器中運行了這些服務，這些服務會調用節點上的sock文件，如果節點上也啓用這些服務去搶佔這些文件，會導致容器異常。默認kolla在預配置時也會主動禁用節點上的相關服務。

安裝OpenStack客戶端

執行openstack相關命令和操作，需要本地安裝openstack客戶端，但是本次在kolla節點安裝客戶端報錯，所以直接啓動一個官方的base容器，該容器默認帶有客戶端命令，使用時將admin-openrc.sh掛載進容器即可。

安裝OpenStack CLI客戶端(可能報錯，略過該步驟)

pip install python-openstackclient

生成openrc文件，該文件中設置了管理員用戶的憑據

kolla-ansible post-deploy

cat /etc/kolla/admin-openrc.sh

kolla ansible提供了一個快速創建cirros demo實例的腳本(可能報錯，略過該步驟)

source /etc/kolla/admin-openrc.sh 
/usr/share/kolla-ansible/init-runonce

訪問openstack horizon

訪問openstack horizon需要使用vip地址，節點上可以看到由keepalived容器生成的vip

[root@kolla ~]# ip a |grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.93.30/24 brd 192.168.93.255 scope global ens33
    inet 192.168.93.100/32 scope global ens33

瀏覽器直接訪問該地址即可登錄到horizon

http://192.168.93.100

我這裏的用戶名密碼爲admin/kolla，信息可以從admin-openrc.sh中獲取

[root@kolla ~]# cat /etc/kolla/admin-openrc.sh
# Clear any old environment that may conflict.
for key in $( set | awk '{FS="="}  /^OS_/ {print $1}' ); do unset $key ; done
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=kolla
export OS_AUTH_URL=http://192.168.93.100:35357/v3
export OS_INTERFACE=internal
export OS_ENDPOINT_TYPE=internalURL
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne
export OS_AUTH_PLUGIN=password

默認登錄後如下

容器運行openstack客戶端

由於在kolla主機節點沒有成功安裝openstack客戶端，這裏在容器裏使用客戶端

拉取官方鏡像地址是kolla/centos-binary-openstack-base:train，這裏從阿里雲拉取

docker pull registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-openstack-base:train

啓動一個客戶端容器，使用sleep infinity命令讓容器一直處於運行狀態。

docker run -d --name client \
  --restart always \
  -v /etc/kolla/admin-openrc.sh:/admin-openrc.sh:ro \
  -v /usr/share/kolla-ansible/init-runonce:/init-runonce:rw \
  registry.cn-shenzhen.aliyuncs.com/kollaimage/centos-binary-openstack-base:train sleep infinity

進入容器就可以正常執行openstack相關命令了

[root@kolla ~]# docker exec -it client bash

()[root@f11a103c5ade /]# source /admin-openrc.sh

()[root@f11a103c5ade /]# openstack service list
+----------------------------------+-------------+----------------+
| ID                               | Name        | Type           |
+----------------------------------+-------------+----------------+
| 2aed09dc3dbd450599042edd9badcc17 | nova_legacy | compute_legacy |
| 2c26e8f09c20455bb67e1df58e7f5ab5 | nova        | compute        |
| 2ec7dd7cd3ce4298931e7272a6e0abd4 | glance      | image          |
| 47062da43fd644eabaa21ae3ec3189da | keystone    | identity       |
| 567057b208ae4a3bb2e3e8e3e7b80bd8 | neutron     | network        |
| 63418bb02ffd449f940c886e640162a1 | heat        | orchestration  |
| 652da566d85c47eb8d38465fe54c232e | cinderv2    | volumev2       |
| 9c8acd17ecbf457fb8b4f29cfc7859da | heat-cfn    | cloudformation |
| d1ef13894f2e44688a1bb117e64d8715 | placement   | placement      |
| d35e629c03794b4c87c6dc2670f3f00a | cinderv3    | volumev3       |
+----------------------------------+-------------+----------------+

示例demo腳本也被掛載到了容器，修改init-runonce示例腳本外部網絡部分的配置，然後執行該shell腳本

()[root@f11a103c5ade /]# cat init-runonce
# This EXT_NET_CIDR is your public network,that you want to connect to the internet via.
ENABLE_EXT_NET=${ENABLE_EXT_NET:-1}
EXT_NET_CIDR=${EXT_NET_CIDR:-'192.168.1.0/24'}
EXT_NET_RANGE=${EXT_NET_RANGE:-'start=192.168.1.200,end=192.168.1.250'}
EXT_NET_GATEWAY=${EXT_NET_GATEWAY:-'192.168.1.1'}

$ bash init-runonce

參數說明：

EXT_NET_CIDR 指定外部網絡，由於使用橋接模式，直接橋接到了電腦的無線網卡，所以這裏網絡就是無線網卡的網段。
EXT_NET_RANGE 指定從外部網絡取出一個地址範圍，作爲外部網絡的地址池
EXT_NET_GATEWAY 外部網絡網關，這裏與wifi網絡使用的網關一致

腳本會創建一些資源，如下載cirros鏡像並上傳，創建外部和內部網絡等，另外腳本執行過程中會創建ssh key，直接回車即可，默認保存在容器中的/root/.ssh目錄下，其中的id_rsa私鑰可以用來遠程連接實例使用。

該腳本首先會從github下載cirros鏡像，如果網絡較慢可以提前下載到cache目錄

docker exec -it client mkdir -p /opt/cache/files/
wget https://github.com/cirros-dev/cirros/releases/download/0.4.0/cirros-0.4.0-x86_64-disk.img
docker cp cirros-0.4.0-x86_64-disk.img client:/opt/cache/files/

最後根據提示手動運行一個實例

openstack server create \
    --image cirros \
    --flavor m1.tiny \
    --key-name mykey \
    --network demo-net \
    demo1

在horizion查看創建的網絡和實例

登錄實例控制檯，驗證實例與外網的連通性，cirros用戶密碼在初次登錄時有提示：

爲實例綁定浮動IP地址，方便從外部ssh遠程連接到實例

點擊+隨機分配一個浮動IP

在kolla節點上ssh連接實例浮動IP，cirros鏡像默認用戶密碼爲cirros/gocubsgo，該鏡像信息官網有介紹：
https://docs.openstack.org/image-guide/obtain-images.html#cirros-test

[root@kolla ~]# ssh [email protected]
[email protected]'s password: 
$ 
$

或者在集羣外部使用SecureCRT連接到實例。

運行CentOS實例

centos官方維護有相關cloud image，如果不需要進行定製，可以直接下載來運行實例。

參考：https://docs.openstack.org/image-guide/obtain-images.html

CentOS官方維護的鏡像下載地址：
http://cloud.centos.org/centos/7/images/

也可以使用命令直接下載鏡像，但是下載可能較慢，建議下載好在進行上傳。以centos7.8爲例：

wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-2003.qcow2c

下載完成後上傳鏡像到openstack，直接在horizon上傳即可。也可以使用命令上傳。

注意：默認該鏡像運行的實例只能使用ssh key以centos用戶身份登錄，如果需要使用root遠程ssh連接到實例需要在上傳前爲鏡像配置root免密並開啓ssh訪問。

參考：https://blog.csdn.net/networken/article/details/106713658

另外我們的命令客戶端在容器中，所有這裏有些不方便，首先要將鏡像複製到容器中，然後使用openstack命令上傳。

這裏複製到client容器的根目錄下。

[root@kolla ~]# docker cp CentOS-7-x86_64-GenericCloud-2003.qcow2c client:/

[root@kolla ~]# docker exec -it client bash
()[root@f11a103c5ade /]# 
()[root@f11a103c5ade /]# source /admin-openrc.sh 

()[root@f11a103c5ade /]# ls | grep CentOS
CentOS-7-x86_64-GenericCloud-2003.qcow2c

執行以下openstack命令上傳鏡像

openstack image create "CentOS78-image" \
  --file CentOS-7-x86_64-GenericCloud-2003.qcow2c \
  --disk-format qcow2 --container-format bare \
  --public

命令執行結果

()[root@f11a103c5ade ~]# openstack image create "CentOS78-image" \
>   --file CentOS-7-x86_64-GenericCloud-2003.qcow2c \
>   --disk-format qcow2 --container-format bare \
>   --public
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                                                                                                      |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum         | 362d1e07d42bcbc61b839fb4269b173b                                                                                                                                                           |
| container_format | bare                                                                                                                                                                                       |
| created_at       | 2020-06-13T03:23:16Z                                                                                                                                                                       |
| disk_format      | qcow2                                                                                                                                                                                      |
| file             | /v2/images/2d95d8a0-6fba-4ca8-9dde-8696eb7ebdbf/file                                                                                                                                       |
| id               | 2d95d8a0-6fba-4ca8-9dde-8696eb7ebdbf                                                                                                                                                       |
| min_disk         | 0                                                                                                                                                                                          |
| min_ram          | 0                                                                                                                                                                                          |
| name             | CentOS78-image                                                                                                                                                                           |
| owner            | 65850af146fe478ab13f59f7edf838ec                                                                                                                                                           |
| properties       | os_hash_algo='sha512', os_hash_value='aefa398f69e1746b420c44e5650f0dcf15926fb6f8c75f746bb2f48a04f7b140fdc745090f3d06b68fa0fe711ded7d822150765414e2a23f351efd2e181eb7b9', os_hidden='False' |
| protected        | False                                                                                                                                                                                      |
| schema           | /v2/schemas/image                                                                                                                                                                          |
| size             | 385941504                                                                                                                                                                                  |
| status           | active                                                                                                                                                                                     |
| tags             |                                                                                                                                                                                            |
| updated_at       | 2020-06-13T03:23:20Z                                                                                                                                                                       |
| virtual_size     | None                                                                                                                                                                                       |
| visibility       | public                                                                                                                                                                                     |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

在horizon中使用該鏡像創建實例，其他信息直接使用之前demo創建的資源即可。創建完成後爲實例綁定浮動IP。

如果實例創建失敗可以查看相關組件報錯日誌

[root@kolla ~]# tail -100f /var/log/kolla/nova/nova-compute.log

未配置root密碼連接實例
如果沒有提前定製鏡像修改root密碼，只能使用centos用戶及sshkey登錄，由於是在容器中運行的demo示例，ssh私鑰也保存在容器的默認目錄下，在容器中連接實例浮動IP測試

[root@kolla ~]# docker exec -it client bash
()[root@f11a103c5ade /]#  ssh -i /root/.ssh/id_rsa [email protected]
Last login: Sat Jun 13 05:47:49 2020 from 192.168.1.100
[centos@centos78 ~]$ 
[centos@centos78 ~]$

或者取出該id_rsa私鑰，使用SecureCRT登錄：

配置root密碼連接實例

如果提前修改了鏡像配置root密碼，直接使用root用戶密碼登錄實例即可，

[root@kolla ~]# ssh [email protected]
[email protected]'s password: 
Last login: Sat Jun 13 05:51:53 2020 from 192.168.1.100
[root@centos78 ~]# 
[root@centos78 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:94:b5:71 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.193/24 brd 10.0.0.255 scope global dynamic eth0
       valid_lft 84215sec preferred_lft 84215sec
    inet6 fe80::f816:3eff:fe94:b571/64 scope link 
       valid_lft forever preferred_lft forever
[root@centos78 ~]# 
[root@centos78 ~]# cat /etc/redhat-release 
CentOS Linux release 7.8.2003 (Core)

運行Ubuntu實例

下載鏡像

wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img

上傳鏡像

openstack image create "Ubuntu1804" \
  --file bionic-server-cloudimg-amd64.img \
  --disk-format qcow2 --container-format bare \
  --public

按照正常流程創建實例即可，ubuntu鏡像默認用戶爲ubuntu，首次登陸使用sshkey方式，然後執行以下命令即可直接切換到root用戶（centos無法使用該方式）

$ sudo -i

kolla配置和日誌文件

各個組件配置文件目錄： /etc/kolla/
各個組件日誌文件目錄：/var/log/kolla/

清理kolla ansilbe集羣

kolla-ansible destroy --include-images --yes-i-really-really-mean-it

#重置cinder卷，謹慎操作
vgremove cinder-volume

kolla-ansible部署all-in-one單節點openstack

kolla ansible簡介

安裝環境準備

部署kolla ansible

部署openstack組件

安裝OpenStack客戶端

訪問openstack horizon

容器運行openstack客戶端

運行CentOS實例

運行Ubuntu實例

kolla配置和日誌文件

清理kolla ansilbe集羣

CentOS安裝GUI圖形界面

手動部署ceph集羣

ceph ansible部署ceph集羣

Windows WSL2安裝

centos安裝openjdk

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結