國慶在家折騰了一把openstack manila,看了下現網還沒有中文的manila入門介紹,於是決定寫個筆記貼出來
一、manila簡介
和我們傳統存儲服務器一樣,openstack的存儲也分爲3種:塊存儲(cinder),對象存儲(swift),文件存儲(manila).manila 提供的是帶有完整文件系統的存儲服務.可以提供的類型有:nfs,cifs,glusterfs,hdfs等.雲主機可以直接在系統裏面掛載manila啓動的實例
下面就是已經被虛擬機掛載的實例
[root@centos centos]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda1 xfs 10G 1.2G 8.9G 12% /
devtmpfs devtmpfs 900M 0 900M 0% /dev
10.254.0.7:/shares/share-f898d9b0-77b8-4231-9919-27b12e34cfa3 nfs4 976M 1.3M 908M 1% /mnt
二 manila的組件
默認情況下manila有3個組件
manila-api 接受並驗證REST請求，通過客戶端及路由進行轉發
manila-scheduler 決定共享創建的後端（後端及pool）
manila-share 基於後端創建共享的服務 (類似於cinder volume 只是創建服務,並不一定提供服務)
以上3行 引用http://blog.csdn.net/Miss_yang_Cloud/article/details/51376480
二、manila-share的典型模式
(一)單獨一個或若干個manila-share節點上通過lvm劃分並格式化出一個ext4的卷通過external網絡給雲主機訪問
典型架構如下
(二) 虛擬機模式
具體流程看第三段
架構圖如下
(三)nas存儲
manila share 節點通過管理網絡連接nas存儲的api管理口
用戶雲主機通過vrouter(可選)連接nas
三、虛擬機模式架構
筆者主要對openstack中vm as a service的模式有着比較大的興趣,折騰完trove(虛擬機當數據庫實例)和octavia(虛擬機當lb)之後花了幾天時間折騰了一下manila的虛擬機+nfs模式
(一) 流程
系統會爲manila創建一個manila networks,
用戶創建manila 實例時會進行如下步驟,其中任何一個步驟出錯都會導致創建實例出錯
1 將manila網絡接入和指定網絡同一個vrouter下
2 調用nova 創建一個manila雲主機,綁定安全組,創建cinder 卷
3 manila share會在5分鐘內不停的 通過 manila networks 嘗試ssh 連接manila虛擬機,
4 連接上之後進行nfs初始化
5 用戶虛擬機 用過用戶網絡 到vrouter 到manila網絡 實現對nfs的訪問
(二)關鍵點的代碼
1 檢查manila網絡有沒有和用戶指定網絡有沒有接入同一個vrouter
manila/share/drivers/service_instance.py
def _get_private_router(self, neutron_net_id, neutron_subnet_id):
"""Returns router attached to private subnet gateway."""
private_subnet = self.neutron_api.get_subnet(neutron_subnet_id)
if not private_subnet['gateway_ip']:
raise exception.ServiceInstanceException(
_('Subnet must have gateway.'))
private_network_ports = [p for p in self.neutron_api.list_ports(
network_id=neutron_net_id)]
for p in private_network_ports:
fixed_ip = p['fixed_ips'][0]
if (fixed_ip['subnet_id'] == private_subnet['id'] and
fixed_ip['ip_address'] == private_subnet['gateway_ip']):
private_subnet_gateway_port = p
break
else:
raise exception.ServiceInstanceException(
_('Subnet gateway is not attached to the router.'))
private_subnet_router = self.neutron_api.show_router(
private_subnet_gateway_port['device_id'])
return private_subnet_router先檢查subnet 有沒有gateway 有的話在檢查有沒有接入路由器
2 檢查 manila share 節點 有沒有打通網絡
前面所述manila share 進程是可以直接ssh 進mannila vm的,因此需要打通manila share 節點和manila networks 這個租戶網絡
network/linux/interface.py
def plug(self, device_name, port_id, mac_address,
bridge=None, namespace=None, prefix=None):
"""Plugin the interface."""
ip = ip_lib.IPWrapper()
if prefix:
tap_name = device_name.replace(prefix, 'tap')
else:
tap_name = device_name.replace(self.DEV_NAME_PREFIX, 'tap')
if not ip_lib.device_exists(device_name,
namespace=namespace):
# Create ns_veth in a namespace if one is configured.
root_veth, ns_veth = ip.add_veth(tap_name, device_name,
namespace2=namespace)
ns_veth.link.set_address(mac_address)
else:
ns_veth = ip.device(device_name)
root_veth = ip.device(tap_name)
LOG.warning("Device %s already exists.", device_name)
root_veth.link.set_up()
ns_veth.link.set_up()先檢查 是否存在這樣的設備,如果不存在,就創建一個
最終會發現manila share 節點上生成一個 ns開頭的tap設備
ns-7a203bcb-40: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.254.0.10 netmask 255.255.255.240 broadcast 10.254.0.15
inet6 fe80::f816:3eff:fe56:b6f7 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:56:b6:f7 txqueuelen 1000 (Ethernet)
RX packets 1190 bytes 193963 (189.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2236 bytes 135968 (132.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0通過他 我們可以直接在manila share 節點 訪問 manila vm
3 主要創建過程
share/drivers/generic.py
def create_share(self, context, share, share_server=None):
"""Creates share."""
return self._create_share(
context, share,
snapshot=None,
share_server=share_server,
)
def _create_share(self, context, share, snapshot, share_server=None):
helper = self._get_helper(share)
server_details = share_server['backend_details']
volume = self._allocate_container(
self.admin_context, share, snapshot=snapshot)
volume = self._attach_volume(
self.admin_context, share, server_details['instance_id'], volume)
if not snapshot:
self._format_device(server_details, volume)
self._mount_device(share, server_details, volume)
export_locations = helper.create_exports(
server_details, share['name'])
return export_locations總體過程就是 :
(1)創建虛擬機
(2)創建volume
(3)掛載 volume
4 虛擬機創建後的操作
(1)manila share 在虛擬機被創建後不同的嘗試ssh連接 虛擬機
share/drivers/service_instance.p
def _check_server_availability(self, instance_details):
t = time.time()
while time.time() - t < self.max_time_to_build_instance:
LOG.debug('Checking server availability.')
if not self._test_server_connection(instance_details):
time.sleep(5)
else:
return True
return False
def _test_server_connection(self, server):
try:
socket.socket().connect((server['ip'], 22))
LOG.debug('Server %s is available via SSH.',
server['ip'])
return True
except socket.error as e:
LOG.debug(e)
LOG.debug("Server %s is not available via SSH. Waiting...",
server['ip'])
return False如果規定時間連接不成功直接把 狀態由creating變成error
def set_up_service_instance(self, context, network_info):
"""Finds or creates and sets up service vm.
:param context: defines context, that should be used
:param network_info: network info for getting allocations
:returns: dict with service instance details
:raises: exception.ServiceInstanceException
"""
instance_name = network_info['server_id']
server = self._create_service_instance(
context, instance_name, network_info)
instance_details = self._get_new_instance_details(server)
if not self._check_server_availability(instance_details):
e = exception.ServiceInstanceException(
_('%(conn_proto)s connection has not been '
'established to %(server)s in %(time)ss. Giving up.') % {
'conn_proto': self._INSTANCE_CONNECTION_PROTO,
'server': server['ip'],
'time': self.max_time_to_build_instance})
e.detail_data = {'server_details': instance_details}
raise e
return instance_detailsssh連上之後的操作
檢查cinder volume 是否存在
def _is_device_file_available(self, server_details, volume):
"""Checks whether the device file is available"""
command = ['sudo', 'test', '-b', volume['mountpoint']]
self._ssh_exec(server_details, command)檢測到存在之後進行格式化
def _format_device(self, server_details, volume):
"""Formats device attached to the service vm."""
self._is_device_file_available(server_details, volume)
command = ['sudo', 'mkfs.%s' % self.configuration.share_volume_fstype,
volume['mountpoint']]
self._ssh_exec(server_details, command)格式化後進行掛載
def _is_device_mounted(self, mount_path, server_details, volume=None):
"""Checks whether volume already mounted or not."""
log_data = {
'mount_path': mount_path,
'server_id': server_details['instance_id'],
}
if volume and volume.get('mountpoint', ''):
log_data['volume_id'] = volume['id']
log_data['dev_mount_path'] = volume['mountpoint']
msg = ("Checking whether volume '%(volume_id)s' with mountpoint "
"'%(dev_mount_path)s' is mounted on mount path '%(mount_p"
"ath)s' on server '%(server_id)s' or not." % log_data)
else:
msg = ("Checking whether mount path '%(mount_path)s' exists on "
"server '%(server_id)s' or not." % log_data)
LOG.debug(msg)
mounts_list_cmd = ['sudo', 'mount']
output, __ = self._ssh_exec(server_details, mounts_list_cmd)
mounts = output.split('\n')
for mount in mounts:
mount_elements = mount.split(' ')
if (len(mount_elements) > 2 and mount_path == mount_elements[2]):
if volume:
# Mount goes with device path and mount path
if (volume.get('mountpoint', '') == mount_elements[0]):
return True
else:
# Unmount goes only by mount path
return True
return False5 權限控制
虛擬機模式的權限控制主要依靠nfs 自己的權限控制來進行
manila access-allow demo-share1 ip 172.16.1.11share 節點 會ssh連接上 demo-share1所在虛擬機 的/etc/export加上一行
manila@ubuntu:~$ cat /etc/exports
/shares/share-f898d9b0-77b8-4231-9919-27b12e34cfa3 172.16.1.11(rw,sync,wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,anonuid=65534,anongid=65534,sec=sys,rw,no_root_squash,no_all_squash)上文說的安全組規則只是是通用規則,mannila 並不會會對每臺虛擬機應用不同的安全組規則
share/drivers/helpers.py
def update_access(self, server, share_name, access_rules, add_rules,
delete_rules):
local_path = os.path.join(self.configuration.share_mount_path,
share_name)
out, err = self._ssh_exec(server, ['sudo', 'exportfs'])
# Recovery mode
if not (add_rules or delete_rules):
self.validate_access_rules(
access_rules, ('ip',),
(const.ACCESS_LEVEL_RO, const.ACCESS_LEVEL_RW))
hosts = self.get_host_list(out, local_path)
for host in hosts:
self._ssh_exec(server, ['sudo', 'exportfs', '-u',
':'.join((host, local_path))])
self._sync_nfs_temp_and_perm_files(server)
for access in access_rules:
rules_options = '%s,no_subtree_check'
if access['access_level'] == const.ACCESS_LEVEL_RW:
rules_options = ','.join((rules_options, 'no_root_squash'))
access_to = self._get_parsed_address_or_cidr(
access['access_to'])
self._ssh_exec(
server,
['sudo', 'exportfs', '-o',
rules_options % access['access_level'],
':'.join((access_to, local_path))])
self._sync_nfs_temp_and_perm_files(server)主要是通過 exportfs命令來進行 nfs權限管理
四 對manila目前狀態的評價
堪用.在流程上沒有大的bug,但是在安全,容錯,高可用,文檔,代碼可讀性等方面還有很多路要走.其實以上這些也是整個openstack項目的通病,但是反過來一想,如果沒有這些問題,openstack還能吸引這麼多人關注嗎?
我的博客即將同步至 OSCHINA 社區，這是我的 OSCHINA ID：osc_76674613，邀請大家一同入駐：https://www.oschina.net/sharing-plan/apply