Ceph-Ansible部署Ceph-N（基于ubuntu 18.04 LTS）

常用自动化部署方案比较

名称	实现语言	发布时间	架构	DSL （区域特定语言）
SaltStack	Python	2011	C/S	yaml
Ansible	Python	2012	Agentless	yaml
Puppet	Ruby	2009	C/S	Puppet Language
Chef	Ruby	2005	C/S	接近Ruby语法的Dsl

环境准备

在部署 Ceph 存储集群之前，需要对Ceph 客户端和Ceph 节点进行一些基本的配置。安装一个ceph-ansible 管理节点和若干节点的Ceph存储集群来研究 Ceph 的基本特性。此处准备四个Ceph节点（其中adm节点充当ceph-ansible管理节点），以此构成 Ceph存储集群。

主机名称	IP地址	操作系统	备注
adm	100.10.10.163	ubuntu-18.04.3-server-amd64.iso	ceph-ansible管理节点
N1	100.10.10.160/100.10.11.160	ubuntu-18.04.3-server-amd64.iso	专用节点
N2	100.10.10.161/100.10.11.161	ubuntu-18.04.3-server-amd64.iso	专用节点
N3	100.10.10.162/100.10.11.162	ubuntu-18.04.3-server-amd64.iso	专用节点

未特别说明，则操作需在每个节点上执行

添加keys

将密钥添加到系统的受信任密钥列表中，以避免出现安全警告。对于主要发行版（例如luminous, mimic, nautilus）和开发发行版（release-name-rc1, release-name-rc2），请使用release.asc密钥。

要安装release.asc密钥，请执行以下操作：

# wget -q -O- 'http://100.10.10.163/ceph/keys/release.asc' | sudo apt-key add -
# wget -q -O- 'http://100.10.10.163/os/18.04_LTS/debs.asc' | sudo apt-key add -
# apt-key list
/etc/apt/trusted.gpg
--------------------
pub   rsa4096 2015-09-15 [SC]
      08B7 3419 AC32 B4E9 66C1  A330 E84A C2C0 460F 3994
uid           [ unknown] Ceph.com (release key) <[email protected]>

添加CEPH

将Ceph软件包库添加到系统的APT来源列表中。对于较新版本的Debian/Ubuntu，请在命令行上调用lsb_release -sc以获取简短的代号。

# cp /etc/apt/sources.list /etc/apt/sources.list.org
# > /etc/apt/sources.list
# sudo apt-add-repository 'deb http://100.10.10.163/ceph/debian-nautilus/ bionic main'
# apt-get update -y

修改/etc/hosts文件

# vim /etc/hosts
127.0.0.1	localhost
100.10.10.160	N1
100.10.10.161	N2
100.10.10.162	N3
100.10.10.163	adm

配置ssh免密登录

在adm节点执行以下命令，

#  ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:w3exfRZVT5yUhWGDS0KlJFSXkGOBRB52DY2wXJZfNdI
The key's randomart image is:
+---[RSA 2048]----+
|       +O=@@o+*B@|
|       +.OBo*.+E=|
|        +..=.o ..|
|       .    o+  .|
|        S . o . o|
|         o .   o |
|                 |
|                 |
|                 |
+----[SHA256]-----+

# ssh-copy-id 100.10.10.160
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '100.10.10.160'"
and check to make sure that only the key(s) you wanted were added.

# ssh-copy-id 100.10.10.161
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '100.10.10.161'"
and check to make sure that only the key(s) you wanted were added.

# ssh-copy-id 100.10.10.162
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '100.10.10.162'"
and check to make sure that only the key(s) you wanted were added.

# ssh-copy-id 100.10.10.163
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '100.10.10.163 (100.10.10.163)' can't be established.
ECDSA key fingerprint is SHA256:krdx3rwkZITeNgQBIDo4eY70tOTrMFhJ4c24RSGPQYI.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '100.10.10.163'"
and check to make sure that only the key(s) you wanted were added.

# scp /root/.ssh/id_rsa* 100.10.10.160:/root/.ssh/
id_rsa                                                                                                                                   100% 1679     1.6MB/s   00:00    
id_rsa.pub                                                                                                                               100%  393   481.8KB/s   00:00    
# scp /root/.ssh/id_rsa* 100.10.10.161:/root/.ssh/
id_rsa                                                                                                                                   100% 1679     1.6MB/s   00:00    
id_rsa.pub                                                                                                                               100%  393   527.1KB/s   00:00    
# scp /root/.ssh/id_rsa* 100.10.10.162:/root/.ssh/
id_rsa                                                                                                                                    100% 1679     1.5MB/s   00:00    
id_rsa.pub

在每个节点上执行

# ssh 100.10.10.160
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Jan  9 12:35:33 CST 2020

  System load:  0.04               Processes:               311
  Usage of /:   3.2% of 136.36GB   Users logged in:         2
  Memory usage: 0%                 IP address for enp3s0f0: 100.10.10.160
  Swap usage:   0%                 IP address for enp3s0f1: 100.10.11.160


 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

0 packages can be updated.
0 updates are security updates.

Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings


Last login: Thu Jan  9 12:34:37 2020 from 100.10.10.162
root@N1:~# exit
logout
Connection to 100.10.10.160 closed.
# ssh 100.10.10.161
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Jan  9 12:35:38 CST 2020

  System load:  0.08               Processes:               305
  Usage of /:   3.2% of 136.36GB   Users logged in:         2
  Memory usage: 0%                 IP address for enp4s0f0: 100.10.10.161
  Swap usage:   0%                 IP address for enp4s0f1: 100.10.11.161


 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

0 packages can be updated.
0 updates are security updates.

Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings


Last login: Thu Jan  9 12:35:04 2020 from 100.10.10.162
root@N2:~# exit
logout
Connection to 100.10.10.161 closed.
# ssh 100.10.10.162
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Jan  9 12:41:26 CST 2020

  System load:  0.05               Processes:               299
  Usage of /:   3.2% of 136.36GB   Users logged in:         1
  Memory usage: 0%                 IP address for enp3s0f0: 100.10.10.162
  Swap usage:   0%                 IP address for enp3s0f1: 100.10.11.162


 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

0 packages can be updated.
0 updates are security updates.


Last login: Thu Jan  9 12:40:55 2020 from 100.10.10.162
root@N3:~# exit
logout
Connection to 100.10.10.162 closed.
# ssh 100.10.10.163
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Jan  9 12:41:32 CST 2020

  System load:  0.0                Processes:               320
  Usage of /:   6.1% of 136.36GB   Users logged in:         2
  Memory usage: 1%                 IP address for enp3s0f0: 100.10.10.163
  Swap usage:   0%                 IP address for enp3s0f1: 172.10.27.163

 * Overheard at KubeCon: "microk8s.status just blew my mind".

     https://microk8s.io/docs/commands#microk8s.status

 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

141 packages can be updated.
89 updates are security updates.


*** System restart required ***
Last login: Thu Jan  9 12:41:07 2020 from 100.10.10.162
root@ceph-n:~# exit
logout
Connection to 100.10.10.163 closed.

安装 CHRONY

在Ceph节点上安装CHRONY以防止时钟漂移引起的问题。

# timedatectl set-ntp false
# apt-get install chrony -y
# chronyd --version
# cp /etc/chrony/chrony.conf /etc/chrony/chrony.conf.org

设置chrony服务器

此处将adm节点作为CHRONY服务器

# vim /etc/chrony/chrony.conf
.....
pool ntp.ubuntu.com        iburst maxsources 4
pool 0.ubuntu.pool.ntp.org iburst maxsources 1
pool 1.ubuntu.pool.ntp.org iburst maxsources 1
pool 2.ubuntu.pool.ntp.org iburst maxsources 2
.......
allow 100.10 ##允许同步时间的客户端所在网段
local stratum 10 ##设置源时钟等级

将配置文件中上述4个pool注释掉，在其中增加允许同步时间的IP网段段以及设置源时钟等级。

# systemctl enable chrony.service
# systemctl restart chrony.service
# systemctl status chrony.service
● chrony.service - chrony, an NTP client/server
   Loaded: loaded (/lib/systemd/system/chrony.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-01-09 15:51:54 CST; 8s ago
     Docs: man:chronyd(8)
           man:chronyc(1)
           man:chrony.conf(5)
  Process: 25427 ExecStartPost=/usr/lib/chrony/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
  Process: 25398 ExecStart=/usr/lib/systemd/scripts/chronyd-starter.sh $DAEMON_OPTS (code=exited, status=0/SUCCESS)
 Main PID: 25422 (chronyd)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/chrony.service
           └─25422 /usr/sbin/chronyd

设置chrony客户端

此处N1、N2和N3节点为CHRONY客户端

# vim /etc/chrony/chrony.conf
.....
# pool ntp.ubuntu.com        iburst maxsources 4
# pool 0.ubuntu.pool.ntp.org iburst maxsources 1
# pool 1.ubuntu.pool.ntp.org iburst maxsources 1
# pool 2.ubuntu.pool.ntp.org iburst maxsources 2
server 100.10.10.163 iburst 
.......

其中100.10.10.163为CHRONY服务器的IP地址。

启动并确认chronyd服务

# systemctl enable chrony.service
Synchronizing state of chrony.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable chrony
# systemctl restart chrony.service
# systemctl status chrony.service
● chrony.service - chrony, an NTP client/server
   Loaded: loaded (/lib/systemd/system/chrony.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-01-09 15:51:54 CST; 8s ago
     Docs: man:chronyd(8)
           man:chronyc(1)
           man:chrony.conf(5)
  Process: 25427 ExecStartPost=/usr/lib/chrony/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
  Process: 25398 ExecStart=/usr/lib/systemd/scripts/chronyd-starter.sh $DAEMON_OPTS (code=exited, status=0/SUCCESS)
 Main PID: 25422 (chronyd)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/chrony.service
           └─25422 /usr/sbin/chronyd

确认时间同步状态

## 查看 chrony_servers 状态
# chronyc sources -v

## 查看 chrony_sync 状态
# chronyc sourcestats -v

## 查看 chrony_servers 是否在线
# chronyc activity -v

## 查看 chrony 详细信息
# chronyc tracking -v

Ceph-Ansible环境配置

下载Ceph-ansible

# wget -q -O ceph-ansible-stable-4.0.zip https://codeload.github.com/ceph/ceph-ansible/zip/stable-4.0
# unzip ceph-ansible-stable-4.0.zip

下载ansible

本次使用的是ansible-2.8.7，安装ansible需要一些依赖包，依赖包及其版本如下：

asn1crypto==0.24.0
cffi==1.13.2
cryptography==2.8
dnspython==1.15.0
enum34==1.1.6
httplib2==0.11.3
idna==2.6
ipaddress==1.0.23
Jinja2==2.10.3
MarkupSafe==1.1.1
netaddr==0.7.19
ply==3.11
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.19
pycrypto==2.6.1
PyYAML==5.3
six==1.13.0

真实的软件包如下：

-rw-r--r-- 1 root root   125326 Jan  8 15:23 Jinja2-2.10.3-py2.py3-none-any.whl
-rw-r--r-- 1 root root    24348 Jan  8 15:24 MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root   268214 Jan  8 15:23 PyYAML-5.3.tar.gz
-rw-r--r-- 1 root root 12697144 Jan  8 15:23 ansible-2.8.7.tar.gz
-rw-r--r-- 1 root root   384920 Jan  8 15:24 cffi-1.13.2-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root  2260524 Jan  8 15:23 cryptography-2.8-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root    12427 Jan  8 15:23 enum34-1.1.6-py2-none-any.whl
-rw-r--r-- 1 root root    18159 Jan  8 15:23 ipaddress-1.0.23-py2.py3-none-any.whl
-rw-r--r-- 1 root root  1628451 Jan  8 15:23 netaddr-0.7.19-py2.py3-none-any.whl
-rw-r--r-- 1 root root   158295 Jan  8 15:24 pycparser-2.19.tar.gz
-rw-r--r-- 1 root root    10747 Jan  8 15:24 six-1.13.0-py2.py3-none-any.whl

不再赘述如何安装软件包

安装Ceph-N

1.创建主机清单

# cd ceph-ansible-stable-4.0
# vim hosts
[mons]
N1 monitor_address=100.10.10.160
N2 monitor_address=100.10.10.161
N3 monitor_address=100.10.10.162

[osds]
N[1:3]

[rgws]
N1 radosgw_address=100.10.10.160
N2 radosgw_address=100.10.10.161
N3 radosgw_address=100.10.10.162

[grafana-server]
N[1:3]

[mdss]
N[1:3]

[clients]
N[1:3]

[mgrs]
N[1:3]

2.拷贝group_vars/all.yml.sample 到group_vars/all.yml，并修改all.yml 文件，添加如下参数：

ceph_origin: repository
ceph_repository: community
ceph_mirror: http://100.10.10.163/ceph/
ceph_stable_key: "{{ ceph_mirror }}/keys/release.asc"
ceph_stable_release: nautilus
ceph_stable_repo: "{{ ceph_mirror }}/debian-{{ ceph_stable_release }}"
ceph_stable_distro_source: bionic

cephx: "true"


public_network: 100.10.10.0/24
cluster_network: 100.10.11.0/24

mon_host: 100.10.10.160,100.10.10.161,100.10.10.162
mon_initial_members: N1,N2,N3
monitor_interface: enp3s0f0

rbd_cache: "true"
rbd_cache_writethrough_until_flush: "true"
rbd_concurrent_management_ops: 20
rbd_client_directories: true
osd_objectstore: bluestore
osd_auto_discovery: true
osd_auto_discovery_exclude: "dm-*|loop*|md*|rbd*"
mds_max_mds: 1
radosgw_frontend_type: beast
radosgw_thread_pool_size: 512
radosgw_interface: "{{ monitor_interface }}"
email_address: [email protected]
dashboard_enabled: True
dashboard_protocol: http
dashboard_port: 8443
dashboard_admin_user: admin
dashboard_admin_password: toyou@123
grafana_admin_user: admin
grafana_admin_password: admin
grafana_uid: 472
grafana_datasource: Dashboard
grafana_dashboard_version: nautilus
grafana_port: 3000
grafana_allow_embedding: True

3.拷贝site.yml.sample 到site.yml（注释掉一些hosts）

# Defines deployment design and assigns role to server groups

- hosts:
  - mons
  - osds
  - mdss
  - rgws
#  - nfss
#  - rbdmirrors
  - clients
  - mgrs
#  - iscsigws
#  - iscsi-gws # for backward compatibility only!
  - grafana-server
#  - rgwloadbalancers

4.执行部署操作

# ansible-playbook  -i hosts  site.yml

Ceph-Ansible部署Ceph-N（基于ubuntu 18.04 LTS）

常用自动化部署方案比较

环境准备

添加keys

添加CEPH

修改/etc/hosts文件

配置ssh免密登录

安装 CHRONY

设置chrony服务器

设置chrony客户端

Ceph-Ansible环境配置

下载Ceph-ansible

下载ansible

安装Ceph-N

NFS/故障排除

無法刪除image報rbd error image still has watchers解決方法

Ceph集羣配置系列《一》配置yum源

Ceph集羣配置系列《二》環境準備

Python之JSON庫的使用（二）

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結