oslo.memcache如何支持SASL詳解

backend = oslo_cache.memcache_pool
the backend is <oslo_cache.backends.memcache_pool.PooledMemcachedBackend object at 0x7f46bb3b2d68>
		

前面可以看到,如果.conf文件中的cache 組中的backend = oslo_cache.memcache_pool

那麼得到的backend就是oslo_cache.backends.memcache_pool.PooledMemcachedBackend

然後,我又在代碼中將backend.set這個方法打印出來:

the backend.set is 
<bound method GenericMemcachedBackend.set of 
<oslo_cache.backends.memcache_pool.PooledMemcachedBackend object at 0x7fad77e497b8>>

可以看到backend.set是GenericMemcachedBackend.set中的方法,在dogpile/cache/backends/memcached.py文件中定義的

GenericMemcachedBackend.set中調用的是self.client.set(key, value, **self.set_arguments),而self.client又是一個屬性method

oslo_cache.backends.memcache_pool.PooledMemcachedBackend在oslo_cache/backends/memcache_pool.py中定義的,繼承的是GenericMemcachedBackend,所有可以調用其中的set方法。這個類又對client屬性方法進行二次重載

the backend.client is 
<oslo_cache.backends.memcache_pool.ClientProxy object at 0x7fce5f48a710>
backend.client返回的是ClientProxy類對象

在ClientProxy類中,傳入的是oslo_cache._memcache_pool.MemcacheClientPool參數

class ClientProxy(object):
    def __init__(self, client_pool):
        self.client_pool = client_pool

    def _run_method(self, __name, *args, **kwargs):
        with self.client_pool.acquire() as client:
            return getattr(client, __name)(*args, **kwargs)

    def __getattr__(self, name):
        return functools.partial(self._run_method, name)

這個函數暫時沒有看懂,應該在調用ClientProxy(self.client_pool)的時候,直接調用__getattr__(self, name)函數吧,然後把client_pool中的conn對象直接返回了,這個時候,PooledMemcachedBackend.client就直接是client_pool中的conn的了,就可以直接調用其中的set、get等方法了!

現在的應用場景是想打開memcached服務的SALA安全認證,前面已經說了如何開啓這個SALA安全認證的!

現在的想法是:

由於oslo_cache/_memcache_pool.py文件中的_MemcacheClient類繼承的是memcache類
想着能不能創建一個_bmemcacge_pool.py文件,在這個文件中建立一個_bMemcacheClient類,讓這個類繼承bmemcache類
然後在memcache_pool.py文件中,用conf.sala_enable來判斷調用哪個類!

現在在centos環境下進行驗證:

把原來的demo代碼copy到centos環境下進行驗證,由於該環境沒有安裝openstack,因此並沒有對應的庫,需要執行:

pip install oslo_cache

執行:pip install python-memcached

安裝memcache

執行demo之前,需要確認一下memcached服務是否已經開啓了

netstat -an |grep 11211

如果沒有開啓的話,則執行命令

cd /opt/system/portaluser/memcached/bin
bash start

然後執行 python app.py

執行結果如下:

[jorh@localhost cache]$ python app.py 
the caching is True and the cache_time is 7200
_get_default_cache_region
<dogpile.cache.region.CacheRegion object at 0x7fd9f951fd50>
_CACHE is <cacheutils.CacheClient instance at 0x7fda0790bc20>
value is none!
Traceback (most recent call last):
  File "app.py", line 80, in <module>
    value = id_to_glance_id(context,image_id)
  File "app.py", line 26, in memorizer
    val = _CACHE.set(key,value)
  File "/home/jorh/cache/cacheutils.py", line 78, in set
    return self.region.set(key,value)
  File "/home/jorh/.local/lib/python2.7/site-packages/dogpile/cache/region.py", line 1112, in set
    self.backend.set(key, self._value(value))
  File "/home/jorh/.local/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 179, in set
    self.client.set(key, value, **self.set_arguments)
  File "/home/jorh/.local/lib/python2.7/site-packages/oslo_cache/backends/memcache_pool.py", line 32, in _run_method
    return getattr(client, __name)(*args, **kwargs)
  File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 727, in set
    return self._set("set", key, val, time, min_compress_len, noreply)
  File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 1016, in _set
    server, key = self._get_server(key)
  File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 433, in _get_server
    if server.connect():
  File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 1381, in connect
    if self._get_socket():
  File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 1413, in _get_socket
    self.flush()
  File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 1488, in flush
    self.expect(b'OK')
  File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 1463, in expect
    line = self.readline(raise_exception)
  File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 1449, in readline
    data = recv(4096)
socket.timeout: timed out

可以看到上面沒有讀出相應的值的,並且出現錯誤!

如果關閉SASL之後,再執行代碼會出現什麼情況呢!

首先說明一下,如何關閉memcached的SALA認證,將start文件sudo ./memcached -d -f 1.1 -L -S -m 2048 -u os-user中的-S去掉,然後再將memcached這個服務重啓一下就行了!有可能不會成功,這個時候,可能需要重啓一下電腦纔行!

執行python app.py命令,發現可以正常讀出數據

[jorh@localhost cache]$ python app.py 
the caching is True and the cache_time is 7200
_get_default_cache_region
<dogpile.cache.region.CacheRegion object at 0x7fc649207d50>
_CACHE is <cacheutils.CacheClient instance at 0x7fc6575f3c20>
value is none!
the set return value is None
the value is this is image_id:1234
_get_default_cache_region
<dogpile.cache.region.CacheRegion object at 0x7fc64719ca10>
cl.add return value: val = 90
key = mykey
VAL1 is 90
VAL2 is jihhhhsnna
_get_default_cache_region
<dogpile.cache.region.CacheRegion object at 0x7fc649207dd0>
cl1 VAL3 is jihhhhsnna
_get_default_cache_region
<dogpile.cache.region.CacheRegion object at 0x7fc6471a6250>
key = mykey
value2 is jihhhhsnna

走到這裏,就非常高興了,說明可以正常開啓和關閉帶有SALA認證的memcached服務

下面,我們將修改代碼,讓oslo_cache模塊支持開啓SALA認證的memcached服務!

首先修改cache.conf文件,裏面增加如下的三個參數

memcache_sasl_enable = true
memcache_usrname = username
memcache_password = Password

之後,通過位於oslo_cache/core.py文件中的configure(conf)函數,將conf中的參數進行配置:

在configure函數中添加如下的內容(下面內容在oslo_cache/_opts.py文件中):

FILE_OPTIONS = {
    'cache': [
        ...........
        cfg.BoolOpt('memcache_sasl_enable',default=False,
                    help='Enable the SASL(Simple Authentication and Security Layer) if the sasl_enable is true, '
                         'else disable!'),
        cfg.StrOpt('memcache_usrname',default='usrname',
                   help='the user name for the SASL'),
        cfg.StrOpt('memcache_password',default='password',
                   help="the user's password for SASL!"),
    ],
}

之後,configure函數會將cache組中的所有配置項進行註冊,然後就可以通過CONF.cache.SALA_enable, CONF.cache.os_user以及CONF.cache.passwd來調用相應的參數!

之後,會創建一個region對象,然後將這個CONF和region進行結合

region = cache.create_region()
cache.configure_cache_region(CONF,region)

下面我們看一下configure_cache_region(CONF,region)這個函數是如何修改(oslo_cache/core.py文件中):

在這個函數中主要調用兩個函數:

config_dict = _build_cache_config(conf)
region.configure_from_config(config_dict,'%s.' % conf.cache.config_prefix)

第一個函數主要用於配置config_dict這個字典參數。

然後修改並添加如下的代碼:

'''
在括號內,添加如下的三個關鍵字,'sala_enable', 'os_user', 'passwd'
'''    
for arg in ('dead_retry', 'socket_timeout', 'pool_maxsize',
                'pool_unused_timeout', 'pool_connection_get_timeout',
                'sasl_enable', 'usrname', 'password'):
        value = getattr(conf.cache, 'memcache_' + arg)
        conf_dict['%s.arguments.%s' % (prefix, arg)] = value

後面就會將這個conf_dict返回,然後我們的三個參數就在這個字典中了!

雖然現在看起來字典中的key = oslo.cache.arguments,不知道爲什麼一定要加上這個prefix

後面當這個字典傳到backend類中的時候,會將字典中key中的這個前綴去掉(感覺上面多此一舉!)

region.configure_from_config這個函數主要用來確定存儲後端backend的,這裏不用進行改動!

現在工作已經完成一半了,因爲我們已經得到了我們想要的參數,並且將這個參數放到字典中,後面只要能得到這個字典,我們就可以得到我們需要的這三個參數!

因爲我們這裏設置的後端存儲是:backend=oslo_cache.memcache_pool,上面也有打印相應的參數實際對應的類對象實例!

因此對應的方法是:oslo_cache.backends.memcache_pool.PooledMemcachedBackend

下面看一下PooledMemcachedBackend中是如何調用的,可以看如下的代碼:

    from oslo_cache import _bmemcache_pool
    def __init__(self, arguments):
        #初始化父類中的__init__函數,後面可以直接調用父類中的屬性
        super(PooledMemcachedBackend, self).__init__(arguments)
        if arguments.get()
        self.client_pool = _memcache_pool.MemcacheClientPool(
            self.url,#這個參數是在GenericMemcachedBackend中__init__的時候得到的!
            arguments={
                'dead_retry': arguments.get('dead_retry', 5 * 60),
                'socket_timeout': arguments.get('socket_timeout', 3.0),
                'server_max_value_length':
                    arguments.get('server_max_value_length'),
            },
            maxsize=arguments.get('pool_maxsize', 10),
            unused_timeout=arguments.get('pool_unused_timeout', 60),
            conn_get_timeout=arguments.get('pool_connection_get_timeout', 10),
        )

那麼,現在我們只需要重新構造一個類,讓他繼承bmemcached就可以了,然後根據arguments中的sala_enable是否爲True來判斷執行那個flow!

下面修改方案:增加一個_bmemcache_pool.py文件,在裏面導入bmemcached模塊,之後所有的代碼都和_memcache_pool.py中的一樣,只不過其中的類前面都添加b字符,以示區分;

    def __init__(self, arguments):
        #初始化父類中的__init__函數,後面可以直接調用父類中的屬性
        super(PooledMemcachedBackend, self).__init__(arguments)
        if arguments.get('sala_enable'):
            self.client_pool = _bmemcache_pool.bMemcacheClientPool(
                self.url,  # 這個參數是在GenericMemcachedBackend中__init__的時候得到的!
                username=arguments.get('os_name'),
                password=arguments.get('passwd'),
                arguments={
                    'socket_timeout': arguments.get('socket_timeout', 3.0),
                },
                maxsize=arguments.get('pool_maxsize', 10),
                unused_timeout=arguments.get('pool_unused_timeout', 60),
                conn_get_timeout=arguments.get('pool_connection_get_timeout', 10),
            )
        else:
            self.client_pool = _memcache_pool.MemcacheClientPool(
                self.url,#這個參數是在GenericMemcachedBackend中__init__的時候得到的!
                arguments={
                    'dead_retry': arguments.get('dead_retry', 5 * 60),
                    'socket_timeout': arguments.get('socket_timeout', 3.0),
                    'server_max_value_length':
                    arguments.get('server_max_value_length'),
                },
                maxsize=arguments.get('pool_maxsize', 10),
                unused_timeout=arguments.get('pool_unused_timeout', 60),
                conn_get_timeout=arguments.get('pool_connection_get_timeout', 10),
            )

_bmemcache_pool.py文件的改動如下:

import bmemcached

.......

'''
class _MemcacheClient類改爲class _bMemcacheClient,繼承bmemcached.Client
'''
class _bMemcacheClient(bmemcached.Client): #
...........

............
#將class ConnectionPool(queue.Queue)改爲class bConnectionPool(queue.Queue)
class bConnectionPool(queue.Queue):
...............

#將class MemcacheClientPool改爲class bMemcacheClientPool,並繼承bConnectionPool
class bMemcacheClientPool(bConnectionPool):
#在這個類中添加兩個變量
    def __init__(self, urls,username,password, arguments, **kwargs):
        # super() cannot be used here because Queue in stdlib is an
        # old-style class
        ConnectionPool.__init__(self, **kwargs)
        self.urls = urls
        #添加如下兩個變量!
        self.username = username
        self.password = password
        self._arguments = arguments
        # NOTE(morganfainberg): The host objects expect an int for the
        # deaduntil value. Initialize this at 0 for each host with 0 indicating
        # the host is not dead.
        self._hosts_deaduntil = [0] * len(urls)
    def _create_connection(self):   #這裏返回bmemcached對象!
        return _bMemcacheClient(self.urls, self.username,self.password,
                               **self._arguments)
    def _get(self):
        # super() cannot be used here because Queue in stdlib is an
        # old-style class
        conn = bConnectionPool._get(self)
      ..........
            bConnectionPool._put(self, conn)
            raise
        return conn
   def _put(self, conn):
        try:
          ......
            bConnectionPool._put(self, conn)


這樣整個代碼就改動完成了,下面開始正式測試流程!

測試出現如下的錯誤:

status = 32
Traceback (most recent call last):
  File "app.py", line 85, in <module>
    myfun2('mykey')
  File "app.py", line 73, in myfun2
    cl2.set(key,'jihhhhsnna')
  File "/home/jorh/cache/cacheutils.py", line 82, in set
    return self.region.set(key,value)
  File "/home/jorh/.local/lib/python2.7/site-packages/dogpile/cache/region.py", line 1112, in set
    self.backend.set(key, self._value(value))
  File "/home/jorh/.local/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 179, in set
    self.client.set(key, value, **self.set_arguments)
  File "/home/jorh/.local/lib/python2.7/site-packages/oslo_cache/backends/memcache_pool.py", line 33, in _run_method
    print("getattr(client, __name)(*args, **kwargs) = %s"%(getattr(client, __name)(*args, **kwargs)))
  File "/usr/lib/python2.7/site-packages/bmemcached/client/replicating.py", line 112, in set
    returns.append(server.set(key, value, time, compress_level=compress_level))
  File "/usr/lib/python2.7/site-packages/bmemcached/protocol.py", line 612, in set
    return self._set_add_replace('set', key, value, time, compress_level=compress_level)
  File "/usr/lib/python2.7/site-packages/bmemcached/protocol.py", line 591, in _set_add_replace
    raise MemcachedException('Code: %d Message: %s' % (status, extra_content), status)
bmemcached.exceptions.MemcachedException

現在發現在關閉SALA的時候,執行是可以的,打開SALA之後,就出現上述的錯誤,現在的可能原因是username和password參數可能傳入不對!

在_bmemcache_pool.py文件中,直接將__init__函數中的self._os-name和self._password設置爲對應的值之後,發現讀取正常了,那麼出現上述現象的原因就是傳入的認證參數不對!

添加代碼鏈接:https://download.csdn.net/download/qq_33970713/12558596

 

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章