常用自動化部署方案比較
名稱 | 實現語言 | 發佈時間 | 架構 | DSL (區域特定語言) |
---|---|---|---|---|
SaltStack | Python | 2011 | C/S | yaml |
Ansible | Python | 2012 | Agentless | yaml |
Puppet | Ruby | 2009 | C/S | Puppet Language |
Chef | Ruby | 2005 | C/S | 接近Ruby語法的Dsl |
環境準備
在部署 Ceph 存儲集羣之前,需要對Ceph 客戶端和Ceph 節點進行一些基本的配置。安裝一個ceph-ansible 管理節點和若干節點的Ceph存儲集羣來研究 Ceph 的基本特性。此處準備四個Ceph節點(其中adm節點充當ceph-ansible管理節點),以此構成 Ceph存儲集羣。
主機名稱 | IP地址 | 操作系統 | 備註 |
---|---|---|---|
adm | 100.10.10.163 | ubuntu-18.04.3-server-amd64.iso | ceph-ansible管理節點 |
N1 | 100.10.10.160/100.10.11.160 | ubuntu-18.04.3-server-amd64.iso | 專用節點 |
N2 | 100.10.10.161/100.10.11.161 | ubuntu-18.04.3-server-amd64.iso | 專用節點 |
N3 | 100.10.10.162/100.10.11.162 | ubuntu-18.04.3-server-amd64.iso | 專用節點 |
未特別說明,則操作需在每個節點上執行
添加keys
將密鑰添加到系統的受信任密鑰列表中,以避免出現安全警告。 對於主要發行版(例如luminous, mimic, nautilus)和開發發行版(release-name-rc1, release-name-rc2),請使用release.asc密鑰。
要安裝release.asc密鑰,請執行以下操作:
# wget -q -O- 'http://100.10.10.163/ceph/keys/release.asc' | sudo apt-key add -
# wget -q -O- 'http://100.10.10.163/os/18.04_LTS/debs.asc' | sudo apt-key add -
# apt-key list
/etc/apt/trusted.gpg
--------------------
pub rsa4096 2015-09-15 [SC]
08B7 3419 AC32 B4E9 66C1 A330 E84A C2C0 460F 3994
uid [ unknown] Ceph.com (release key) <[email protected]>
添加CEPH
將Ceph軟件包庫添加到系統的APT來源列表中。 對於較新版本的Debian/Ubuntu,請在命令行上調用lsb_release -sc以獲取簡短的代號。
# cp /etc/apt/sources.list /etc/apt/sources.list.org
# > /etc/apt/sources.list
# sudo apt-add-repository 'deb http://100.10.10.163/ceph/debian-nautilus/ bionic main'
# apt-get update -y
修改/etc/hosts文件
# vim /etc/hosts
127.0.0.1 localhost
100.10.10.160 N1
100.10.10.161 N2
100.10.10.162 N3
100.10.10.163 adm
配置ssh免密登錄
在adm節點執行以下命令,
# ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:w3exfRZVT5yUhWGDS0KlJFSXkGOBRB52DY2wXJZfNdI
The key's randomart image is:
+---[RSA 2048]----+
| +O=@@o+*B@|
| +.OBo*.+E=|
| +..=.o ..|
| . o+ .|
| S . o . o|
| o . o |
| |
| |
| |
+----[SHA256]-----+
# ssh-copy-id 100.10.10.160
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '100.10.10.160'"
and check to make sure that only the key(s) you wanted were added.
# ssh-copy-id 100.10.10.161
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '100.10.10.161'"
and check to make sure that only the key(s) you wanted were added.
# ssh-copy-id 100.10.10.162
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '100.10.10.162'"
and check to make sure that only the key(s) you wanted were added.
# ssh-copy-id 100.10.10.163
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '100.10.10.163 (100.10.10.163)' can't be established.
ECDSA key fingerprint is SHA256:krdx3rwkZITeNgQBIDo4eY70tOTrMFhJ4c24RSGPQYI.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '100.10.10.163'"
and check to make sure that only the key(s) you wanted were added.
# scp /root/.ssh/id_rsa* 100.10.10.160:/root/.ssh/
id_rsa 100% 1679 1.6MB/s 00:00
id_rsa.pub 100% 393 481.8KB/s 00:00
# scp /root/.ssh/id_rsa* 100.10.10.161:/root/.ssh/
id_rsa 100% 1679 1.6MB/s 00:00
id_rsa.pub 100% 393 527.1KB/s 00:00
# scp /root/.ssh/id_rsa* 100.10.10.162:/root/.ssh/
id_rsa 100% 1679 1.5MB/s 00:00
id_rsa.pub
在每個節點上執行
# ssh 100.10.10.160
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Thu Jan 9 12:35:33 CST 2020
System load: 0.04 Processes: 311
Usage of /: 3.2% of 136.36GB Users logged in: 2
Memory usage: 0% IP address for enp3s0f0: 100.10.10.160
Swap usage: 0% IP address for enp3s0f1: 100.10.11.160
* Canonical Livepatch is available for installation.
- Reduce system reboots and improve kernel security. Activate at:
https://ubuntu.com/livepatch
0 packages can be updated.
0 updates are security updates.
Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings
Last login: Thu Jan 9 12:34:37 2020 from 100.10.10.162
root@N1:~# exit
logout
Connection to 100.10.10.160 closed.
# ssh 100.10.10.161
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Thu Jan 9 12:35:38 CST 2020
System load: 0.08 Processes: 305
Usage of /: 3.2% of 136.36GB Users logged in: 2
Memory usage: 0% IP address for enp4s0f0: 100.10.10.161
Swap usage: 0% IP address for enp4s0f1: 100.10.11.161
* Canonical Livepatch is available for installation.
- Reduce system reboots and improve kernel security. Activate at:
https://ubuntu.com/livepatch
0 packages can be updated.
0 updates are security updates.
Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings
Last login: Thu Jan 9 12:35:04 2020 from 100.10.10.162
root@N2:~# exit
logout
Connection to 100.10.10.161 closed.
# ssh 100.10.10.162
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Thu Jan 9 12:41:26 CST 2020
System load: 0.05 Processes: 299
Usage of /: 3.2% of 136.36GB Users logged in: 1
Memory usage: 0% IP address for enp3s0f0: 100.10.10.162
Swap usage: 0% IP address for enp3s0f1: 100.10.11.162
* Canonical Livepatch is available for installation.
- Reduce system reboots and improve kernel security. Activate at:
https://ubuntu.com/livepatch
0 packages can be updated.
0 updates are security updates.
Last login: Thu Jan 9 12:40:55 2020 from 100.10.10.162
root@N3:~# exit
logout
Connection to 100.10.10.162 closed.
# ssh 100.10.10.163
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Thu Jan 9 12:41:32 CST 2020
System load: 0.0 Processes: 320
Usage of /: 6.1% of 136.36GB Users logged in: 2
Memory usage: 1% IP address for enp3s0f0: 100.10.10.163
Swap usage: 0% IP address for enp3s0f1: 172.10.27.163
* Overheard at KubeCon: "microk8s.status just blew my mind".
https://microk8s.io/docs/commands#microk8s.status
* Canonical Livepatch is available for installation.
- Reduce system reboots and improve kernel security. Activate at:
https://ubuntu.com/livepatch
141 packages can be updated.
89 updates are security updates.
*** System restart required ***
Last login: Thu Jan 9 12:41:07 2020 from 100.10.10.162
root@ceph-n:~# exit
logout
Connection to 100.10.10.163 closed.
安裝 CHRONY
在Ceph節點上安裝CHRONY以防止時鐘漂移引起的問題。
# timedatectl set-ntp false
# apt-get install chrony -y
# chronyd --version
# cp /etc/chrony/chrony.conf /etc/chrony/chrony.conf.org
設置chrony服務器
此處將adm節點作爲CHRONY服務器
# vim /etc/chrony/chrony.conf
.....
pool ntp.ubuntu.com iburst maxsources 4
pool 0.ubuntu.pool.ntp.org iburst maxsources 1
pool 1.ubuntu.pool.ntp.org iburst maxsources 1
pool 2.ubuntu.pool.ntp.org iburst maxsources 2
.......
allow 100.10 ##允許同步時間的客戶端所在網段
local stratum 10 ##設置源時鐘等級
將配置文件中上述4個pool註釋掉,在其中增加允許同步時間的IP網段段以及設置源時鐘等級。
# systemctl enable chrony.service
# systemctl restart chrony.service
# systemctl status chrony.service
● chrony.service - chrony, an NTP client/server
Loaded: loaded (/lib/systemd/system/chrony.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-01-09 15:51:54 CST; 8s ago
Docs: man:chronyd(8)
man:chronyc(1)
man:chrony.conf(5)
Process: 25427 ExecStartPost=/usr/lib/chrony/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
Process: 25398 ExecStart=/usr/lib/systemd/scripts/chronyd-starter.sh $DAEMON_OPTS (code=exited, status=0/SUCCESS)
Main PID: 25422 (chronyd)
Tasks: 1 (limit: 4915)
CGroup: /system.slice/chrony.service
└─25422 /usr/sbin/chronyd
設置chrony客戶端
此處N1、N2和N3節點爲CHRONY客戶端
# vim /etc/chrony/chrony.conf
.....
# pool ntp.ubuntu.com iburst maxsources 4
# pool 0.ubuntu.pool.ntp.org iburst maxsources 1
# pool 1.ubuntu.pool.ntp.org iburst maxsources 1
# pool 2.ubuntu.pool.ntp.org iburst maxsources 2
server 100.10.10.163 iburst
.......
其中100.10.10.163爲CHRONY服務器的IP地址。
啓動並確認chronyd服務
# systemctl enable chrony.service
Synchronizing state of chrony.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable chrony
# systemctl restart chrony.service
# systemctl status chrony.service
● chrony.service - chrony, an NTP client/server
Loaded: loaded (/lib/systemd/system/chrony.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-01-09 15:51:54 CST; 8s ago
Docs: man:chronyd(8)
man:chronyc(1)
man:chrony.conf(5)
Process: 25427 ExecStartPost=/usr/lib/chrony/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
Process: 25398 ExecStart=/usr/lib/systemd/scripts/chronyd-starter.sh $DAEMON_OPTS (code=exited, status=0/SUCCESS)
Main PID: 25422 (chronyd)
Tasks: 1 (limit: 4915)
CGroup: /system.slice/chrony.service
└─25422 /usr/sbin/chronyd
確認時間同步狀態
## 查看 chrony_servers 狀態
# chronyc sources -v
## 查看 chrony_sync 狀態
# chronyc sourcestats -v
## 查看 chrony_servers 是否在線
# chronyc activity -v
## 查看 chrony 詳細信息
# chronyc tracking -v
Ceph-Ansible環境配置
下載Ceph-ansible
# wget -q -O ceph-ansible-stable-4.0.zip https://codeload.github.com/ceph/ceph-ansible/zip/stable-4.0
# unzip ceph-ansible-stable-4.0.zip
下載ansible
本次使用的是ansible-2.8.7,安裝ansible需要一些依賴包,依賴包及其版本如下:
asn1crypto==0.24.0
cffi==1.13.2
cryptography==2.8
dnspython==1.15.0
enum34==1.1.6
httplib2==0.11.3
idna==2.6
ipaddress==1.0.23
Jinja2==2.10.3
MarkupSafe==1.1.1
netaddr==0.7.19
ply==3.11
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.19
pycrypto==2.6.1
PyYAML==5.3
six==1.13.0
真實的軟件包如下:
-rw-r--r-- 1 root root 125326 Jan 8 15:23 Jinja2-2.10.3-py2.py3-none-any.whl
-rw-r--r-- 1 root root 24348 Jan 8 15:24 MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root 268214 Jan 8 15:23 PyYAML-5.3.tar.gz
-rw-r--r-- 1 root root 12697144 Jan 8 15:23 ansible-2.8.7.tar.gz
-rw-r--r-- 1 root root 384920 Jan 8 15:24 cffi-1.13.2-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root 2260524 Jan 8 15:23 cryptography-2.8-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root 12427 Jan 8 15:23 enum34-1.1.6-py2-none-any.whl
-rw-r--r-- 1 root root 18159 Jan 8 15:23 ipaddress-1.0.23-py2.py3-none-any.whl
-rw-r--r-- 1 root root 1628451 Jan 8 15:23 netaddr-0.7.19-py2.py3-none-any.whl
-rw-r--r-- 1 root root 158295 Jan 8 15:24 pycparser-2.19.tar.gz
-rw-r--r-- 1 root root 10747 Jan 8 15:24 six-1.13.0-py2.py3-none-any.whl
不再贅述如何安裝軟件包
安裝Ceph-N
1.創建主機清單
# cd ceph-ansible-stable-4.0
# vim hosts
[mons]
N1 monitor_address=100.10.10.160
N2 monitor_address=100.10.10.161
N3 monitor_address=100.10.10.162
[osds]
N[1:3]
[rgws]
N1 radosgw_address=100.10.10.160
N2 radosgw_address=100.10.10.161
N3 radosgw_address=100.10.10.162
[grafana-server]
N[1:3]
[mdss]
N[1:3]
[clients]
N[1:3]
[mgrs]
N[1:3]
2.拷貝group_vars/all.yml.sample 到group_vars/all.yml,並修改all.yml 文件,添加如下參數:
ceph_origin: repository
ceph_repository: community
ceph_mirror: http://100.10.10.163/ceph/
ceph_stable_key: "{{ ceph_mirror }}/keys/release.asc"
ceph_stable_release: nautilus
ceph_stable_repo: "{{ ceph_mirror }}/debian-{{ ceph_stable_release }}"
ceph_stable_distro_source: bionic
cephx: "true"
public_network: 100.10.10.0/24
cluster_network: 100.10.11.0/24
mon_host: 100.10.10.160,100.10.10.161,100.10.10.162
mon_initial_members: N1,N2,N3
monitor_interface: enp3s0f0
rbd_cache: "true"
rbd_cache_writethrough_until_flush: "true"
rbd_concurrent_management_ops: 20
rbd_client_directories: true
osd_objectstore: bluestore
osd_auto_discovery: true
osd_auto_discovery_exclude: "dm-*|loop*|md*|rbd*"
mds_max_mds: 1
radosgw_frontend_type: beast
radosgw_thread_pool_size: 512
radosgw_interface: "{{ monitor_interface }}"
email_address: [email protected]
dashboard_enabled: True
dashboard_protocol: http
dashboard_port: 8443
dashboard_admin_user: admin
dashboard_admin_password: toyou@123
grafana_admin_user: admin
grafana_admin_password: admin
grafana_uid: 472
grafana_datasource: Dashboard
grafana_dashboard_version: nautilus
grafana_port: 3000
grafana_allow_embedding: True
3.拷貝site.yml.sample 到site.yml(註釋掉一些hosts)
# Defines deployment design and assigns role to server groups
- hosts:
- mons
- osds
- mdss
- rgws
# - nfss
# - rbdmirrors
- clients
- mgrs
# - iscsigws
# - iscsi-gws # for backward compatibility only!
- grafana-server
# - rgwloadbalancers
4.執行部署操作
# ansible-playbook -i hosts site.yml