Ceph-Ansible部署Ceph-N(基於ubuntu 18.04 LTS)

常用自動化部署方案比較

名稱 實現語言 發佈時間 架構 DSL (區域特定語言)
SaltStack Python 2011 C/S yaml
Ansible Python 2012 Agentless yaml
Puppet Ruby 2009 C/S Puppet Language
Chef Ruby 2005 C/S 接近Ruby語法的Dsl

環境準備

在部署 Ceph 存儲集羣之前,需要對Ceph 客戶端和Ceph 節點進行一些基本的配置。安裝一個ceph-ansible 管理節點和若干節點的Ceph存儲集羣來研究 Ceph 的基本特性。此處準備四個Ceph節點(其中adm節點充當ceph-ansible管理節點),以此構成 Ceph存儲集羣。

主機名稱 IP地址 操作系統 備註
adm 100.10.10.163 ubuntu-18.04.3-server-amd64.iso ceph-ansible管理節點
N1 100.10.10.160/100.10.11.160 ubuntu-18.04.3-server-amd64.iso 專用節點
N2 100.10.10.161/100.10.11.161 ubuntu-18.04.3-server-amd64.iso 專用節點
N3 100.10.10.162/100.10.11.162 ubuntu-18.04.3-server-amd64.iso 專用節點

未特別說明,則操作需在每個節點上執行

添加keys

將密鑰添加到系統的受信任密鑰列表中,以避免出現安全警告。 對於主要發行版(例如luminous, mimic, nautilus)和開發發行版(release-name-rc1, release-name-rc2),請使用release.asc密鑰。

要安裝release.asc密鑰,請執行以下操作:

# wget -q -O- 'http://100.10.10.163/ceph/keys/release.asc' | sudo apt-key add -
# wget -q -O- 'http://100.10.10.163/os/18.04_LTS/debs.asc' | sudo apt-key add -
# apt-key list
/etc/apt/trusted.gpg
--------------------
pub   rsa4096 2015-09-15 [SC]
      08B7 3419 AC32 B4E9 66C1  A330 E84A C2C0 460F 3994
uid           [ unknown] Ceph.com (release key) <[email protected]>

添加CEPH

將Ceph軟件包庫添加到系統的APT來源列表中。 對於較新版本的Debian/Ubuntu,請在命令行上調用lsb_release -sc以獲取簡短的代號。

# cp /etc/apt/sources.list /etc/apt/sources.list.org
# > /etc/apt/sources.list
# sudo apt-add-repository 'deb http://100.10.10.163/ceph/debian-nautilus/ bionic main'
# apt-get update -y

修改/etc/hosts文件

# vim /etc/hosts
127.0.0.1	localhost
100.10.10.160	N1
100.10.10.161	N2
100.10.10.162	N3
100.10.10.163	adm

配置ssh免密登錄

在adm節點執行以下命令,

#  ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:w3exfRZVT5yUhWGDS0KlJFSXkGOBRB52DY2wXJZfNdI
The key's randomart image is:
+---[RSA 2048]----+
|       +O=@@o+*B@|
|       +.OBo*.+E=|
|        +..=.o ..|
|       .    o+  .|
|        S . o . o|
|         o .   o |
|                 |
|                 |
|                 |
+----[SHA256]-----+

# ssh-copy-id 100.10.10.160
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '100.10.10.160'"
and check to make sure that only the key(s) you wanted were added.

# ssh-copy-id 100.10.10.161
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '100.10.10.161'"
and check to make sure that only the key(s) you wanted were added.

# ssh-copy-id 100.10.10.162
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '100.10.10.162'"
and check to make sure that only the key(s) you wanted were added.

# ssh-copy-id 100.10.10.163
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '100.10.10.163 (100.10.10.163)' can't be established.
ECDSA key fingerprint is SHA256:krdx3rwkZITeNgQBIDo4eY70tOTrMFhJ4c24RSGPQYI.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '100.10.10.163'"
and check to make sure that only the key(s) you wanted were added.

# scp /root/.ssh/id_rsa* 100.10.10.160:/root/.ssh/
id_rsa                                                                                                                                   100% 1679     1.6MB/s   00:00    
id_rsa.pub                                                                                                                               100%  393   481.8KB/s   00:00    
# scp /root/.ssh/id_rsa* 100.10.10.161:/root/.ssh/
id_rsa                                                                                                                                   100% 1679     1.6MB/s   00:00    
id_rsa.pub                                                                                                                               100%  393   527.1KB/s   00:00    
# scp /root/.ssh/id_rsa* 100.10.10.162:/root/.ssh/
id_rsa                                                                                                                                    100% 1679     1.5MB/s   00:00    
id_rsa.pub                                  

在每個節點上執行

# ssh 100.10.10.160
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Jan  9 12:35:33 CST 2020

  System load:  0.04               Processes:               311
  Usage of /:   3.2% of 136.36GB   Users logged in:         2
  Memory usage: 0%                 IP address for enp3s0f0: 100.10.10.160
  Swap usage:   0%                 IP address for enp3s0f1: 100.10.11.160


 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

0 packages can be updated.
0 updates are security updates.

Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings


Last login: Thu Jan  9 12:34:37 2020 from 100.10.10.162
root@N1:~# exit
logout
Connection to 100.10.10.160 closed.
# ssh 100.10.10.161
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Jan  9 12:35:38 CST 2020

  System load:  0.08               Processes:               305
  Usage of /:   3.2% of 136.36GB   Users logged in:         2
  Memory usage: 0%                 IP address for enp4s0f0: 100.10.10.161
  Swap usage:   0%                 IP address for enp4s0f1: 100.10.11.161


 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

0 packages can be updated.
0 updates are security updates.

Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings


Last login: Thu Jan  9 12:35:04 2020 from 100.10.10.162
root@N2:~# exit
logout
Connection to 100.10.10.161 closed.
# ssh 100.10.10.162
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Jan  9 12:41:26 CST 2020

  System load:  0.05               Processes:               299
  Usage of /:   3.2% of 136.36GB   Users logged in:         1
  Memory usage: 0%                 IP address for enp3s0f0: 100.10.10.162
  Swap usage:   0%                 IP address for enp3s0f1: 100.10.11.162


 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

0 packages can be updated.
0 updates are security updates.


Last login: Thu Jan  9 12:40:55 2020 from 100.10.10.162
root@N3:~# exit
logout
Connection to 100.10.10.162 closed.
# ssh 100.10.10.163
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Jan  9 12:41:32 CST 2020

  System load:  0.0                Processes:               320
  Usage of /:   6.1% of 136.36GB   Users logged in:         2
  Memory usage: 1%                 IP address for enp3s0f0: 100.10.10.163
  Swap usage:   0%                 IP address for enp3s0f1: 172.10.27.163

 * Overheard at KubeCon: "microk8s.status just blew my mind".

     https://microk8s.io/docs/commands#microk8s.status

 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

141 packages can be updated.
89 updates are security updates.


*** System restart required ***
Last login: Thu Jan  9 12:41:07 2020 from 100.10.10.162
root@ceph-n:~# exit
logout
Connection to 100.10.10.163 closed.

安裝 CHRONY

在Ceph節點上安裝CHRONY以防止時鐘漂移引起的問題。

# timedatectl set-ntp false
# apt-get install chrony -y
# chronyd --version
# cp /etc/chrony/chrony.conf /etc/chrony/chrony.conf.org

設置chrony服務器

此處將adm節點作爲CHRONY服務器

# vim /etc/chrony/chrony.conf
.....
pool ntp.ubuntu.com        iburst maxsources 4
pool 0.ubuntu.pool.ntp.org iburst maxsources 1
pool 1.ubuntu.pool.ntp.org iburst maxsources 1
pool 2.ubuntu.pool.ntp.org iburst maxsources 2
.......
allow 100.10 ##允許同步時間的客戶端所在網段
local stratum 10 ##設置源時鐘等級

將配置文件中上述4個pool註釋掉,在其中增加允許同步時間的IP網段段以及設置源時鐘等級。

# systemctl enable chrony.service
# systemctl restart chrony.service
# systemctl status chrony.service
● chrony.service - chrony, an NTP client/server
   Loaded: loaded (/lib/systemd/system/chrony.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-01-09 15:51:54 CST; 8s ago
     Docs: man:chronyd(8)
           man:chronyc(1)
           man:chrony.conf(5)
  Process: 25427 ExecStartPost=/usr/lib/chrony/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
  Process: 25398 ExecStart=/usr/lib/systemd/scripts/chronyd-starter.sh $DAEMON_OPTS (code=exited, status=0/SUCCESS)
 Main PID: 25422 (chronyd)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/chrony.service
           └─25422 /usr/sbin/chronyd

設置chrony客戶端

此處N1、N2和N3節點爲CHRONY客戶端

# vim /etc/chrony/chrony.conf
.....
# pool ntp.ubuntu.com        iburst maxsources 4
# pool 0.ubuntu.pool.ntp.org iburst maxsources 1
# pool 1.ubuntu.pool.ntp.org iburst maxsources 1
# pool 2.ubuntu.pool.ntp.org iburst maxsources 2
server 100.10.10.163 iburst 
.......

其中100.10.10.163爲CHRONY服務器的IP地址。

啓動並確認chronyd服務

# systemctl enable chrony.service
Synchronizing state of chrony.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable chrony
# systemctl restart chrony.service
# systemctl status chrony.service
● chrony.service - chrony, an NTP client/server
   Loaded: loaded (/lib/systemd/system/chrony.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-01-09 15:51:54 CST; 8s ago
     Docs: man:chronyd(8)
           man:chronyc(1)
           man:chrony.conf(5)
  Process: 25427 ExecStartPost=/usr/lib/chrony/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
  Process: 25398 ExecStart=/usr/lib/systemd/scripts/chronyd-starter.sh $DAEMON_OPTS (code=exited, status=0/SUCCESS)
 Main PID: 25422 (chronyd)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/chrony.service
           └─25422 /usr/sbin/chronyd

確認時間同步狀態

## 查看 chrony_servers 狀態
# chronyc sources -v

## 查看 chrony_sync 狀態
# chronyc sourcestats -v

## 查看 chrony_servers 是否在線
# chronyc activity -v

## 查看 chrony 詳細信息
# chronyc tracking -v

Ceph-Ansible環境配置

下載Ceph-ansible

# wget -q -O ceph-ansible-stable-4.0.zip https://codeload.github.com/ceph/ceph-ansible/zip/stable-4.0
# unzip ceph-ansible-stable-4.0.zip

下載ansible

本次使用的是ansible-2.8.7,安裝ansible需要一些依賴包,依賴包及其版本如下:

asn1crypto==0.24.0
cffi==1.13.2
cryptography==2.8
dnspython==1.15.0
enum34==1.1.6
httplib2==0.11.3
idna==2.6
ipaddress==1.0.23
Jinja2==2.10.3
MarkupSafe==1.1.1
netaddr==0.7.19
ply==3.11
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.19
pycrypto==2.6.1
PyYAML==5.3
six==1.13.0

真實的軟件包如下:

-rw-r--r-- 1 root root   125326 Jan  8 15:23 Jinja2-2.10.3-py2.py3-none-any.whl
-rw-r--r-- 1 root root    24348 Jan  8 15:24 MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root   268214 Jan  8 15:23 PyYAML-5.3.tar.gz
-rw-r--r-- 1 root root 12697144 Jan  8 15:23 ansible-2.8.7.tar.gz
-rw-r--r-- 1 root root   384920 Jan  8 15:24 cffi-1.13.2-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root  2260524 Jan  8 15:23 cryptography-2.8-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root    12427 Jan  8 15:23 enum34-1.1.6-py2-none-any.whl
-rw-r--r-- 1 root root    18159 Jan  8 15:23 ipaddress-1.0.23-py2.py3-none-any.whl
-rw-r--r-- 1 root root  1628451 Jan  8 15:23 netaddr-0.7.19-py2.py3-none-any.whl
-rw-r--r-- 1 root root   158295 Jan  8 15:24 pycparser-2.19.tar.gz
-rw-r--r-- 1 root root    10747 Jan  8 15:24 six-1.13.0-py2.py3-none-any.whl

不再贅述如何安裝軟件包

安裝Ceph-N

1.創建主機清單

# cd ceph-ansible-stable-4.0
# vim hosts
[mons]
N1 monitor_address=100.10.10.160
N2 monitor_address=100.10.10.161
N3 monitor_address=100.10.10.162

[osds]
N[1:3]

[rgws]
N1 radosgw_address=100.10.10.160
N2 radosgw_address=100.10.10.161
N3 radosgw_address=100.10.10.162

[grafana-server]
N[1:3]

[mdss]
N[1:3]

[clients]
N[1:3]

[mgrs]
N[1:3]

2.拷貝group_vars/all.yml.sample 到group_vars/all.yml,並修改all.yml 文件,添加如下參數:

ceph_origin: repository
ceph_repository: community
ceph_mirror: http://100.10.10.163/ceph/
ceph_stable_key: "{{ ceph_mirror }}/keys/release.asc"
ceph_stable_release: nautilus
ceph_stable_repo: "{{ ceph_mirror }}/debian-{{ ceph_stable_release }}"
ceph_stable_distro_source: bionic

cephx: "true"


public_network: 100.10.10.0/24
cluster_network: 100.10.11.0/24

mon_host: 100.10.10.160,100.10.10.161,100.10.10.162
mon_initial_members: N1,N2,N3
monitor_interface: enp3s0f0

rbd_cache: "true"
rbd_cache_writethrough_until_flush: "true"
rbd_concurrent_management_ops: 20
rbd_client_directories: true
osd_objectstore: bluestore
osd_auto_discovery: true
osd_auto_discovery_exclude: "dm-*|loop*|md*|rbd*"
mds_max_mds: 1
radosgw_frontend_type: beast
radosgw_thread_pool_size: 512
radosgw_interface: "{{ monitor_interface }}"
email_address: [email protected]
dashboard_enabled: True
dashboard_protocol: http
dashboard_port: 8443
dashboard_admin_user: admin
dashboard_admin_password: toyou@123
grafana_admin_user: admin
grafana_admin_password: admin
grafana_uid: 472
grafana_datasource: Dashboard
grafana_dashboard_version: nautilus
grafana_port: 3000
grafana_allow_embedding: True

3.拷貝site.yml.sample 到site.yml(註釋掉一些hosts)

# Defines deployment design and assigns role to server groups

- hosts:
  - mons
  - osds
  - mdss
  - rgws
#  - nfss
#  - rbdmirrors
  - clients
  - mgrs
#  - iscsigws
#  - iscsi-gws # for backward compatibility only!
  - grafana-server
#  - rgwloadbalancers

4.執行部署操作

# ansible-playbook  -i hosts  site.yml
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章