backend = oslo_cache.memcache_pool
the backend is <oslo_cache.backends.memcache_pool.PooledMemcachedBackend object at 0x7f46bb3b2d68>
前面可以看到,如果.conf文件中的cache 組中的backend = oslo_cache.memcache_pool
那麼得到的backend就是oslo_cache.backends.memcache_pool.PooledMemcachedBackend
然後,我又在代碼中將backend.set這個方法打印出來:
the backend.set is
<bound method GenericMemcachedBackend.set of
<oslo_cache.backends.memcache_pool.PooledMemcachedBackend object at 0x7fad77e497b8>>
可以看到backend.set是GenericMemcachedBackend.set中的方法,在dogpile/cache/backends/memcached.py文件中定義的
GenericMemcachedBackend.set中調用的是self.client.set(key, value, **self.set_arguments),而self.client又是一個屬性method
oslo_cache.backends.memcache_pool.PooledMemcachedBackend在oslo_cache/backends/memcache_pool.py中定義的,繼承的是GenericMemcachedBackend,所有可以調用其中的set方法。這個類又對client屬性方法進行二次重載
the backend.client is
<oslo_cache.backends.memcache_pool.ClientProxy object at 0x7fce5f48a710>
backend.client返回的是ClientProxy類對象
在ClientProxy類中,傳入的是oslo_cache._memcache_pool.MemcacheClientPool參數
class ClientProxy(object):
def __init__(self, client_pool):
self.client_pool = client_pool
def _run_method(self, __name, *args, **kwargs):
with self.client_pool.acquire() as client:
return getattr(client, __name)(*args, **kwargs)
def __getattr__(self, name):
return functools.partial(self._run_method, name)
這個函數暫時沒有看懂,應該在調用ClientProxy(self.client_pool)的時候,直接調用__getattr__(self, name)函數吧,然後把client_pool中的conn對象直接返回了,這個時候,PooledMemcachedBackend.client就直接是client_pool中的conn的了,就可以直接調用其中的set、get等方法了!
現在的應用場景是想打開memcached服務的SALA安全認證,前面已經說了如何開啓這個SALA安全認證的!
現在的想法是:
由於oslo_cache/_memcache_pool.py文件中的_MemcacheClient類繼承的是memcache類
想着能不能創建一個_bmemcacge_pool.py文件,在這個文件中建立一個_bMemcacheClient類,讓這個類繼承bmemcache類
然後在memcache_pool.py文件中,用conf.sala_enable來判斷調用哪個類!
現在在centos環境下進行驗證:
把原來的demo代碼copy到centos環境下進行驗證,由於該環境沒有安裝openstack,因此並沒有對應的庫,需要執行:
pip install oslo_cache
執行:pip install python-memcached
安裝memcache
執行demo之前,需要確認一下memcached服務是否已經開啓了
netstat -an |grep 11211
如果沒有開啓的話,則執行命令
cd /opt/system/portaluser/memcached/bin
bash start
然後執行 python app.py
執行結果如下:
[jorh@localhost cache]$ python app.py
the caching is True and the cache_time is 7200
_get_default_cache_region
<dogpile.cache.region.CacheRegion object at 0x7fd9f951fd50>
_CACHE is <cacheutils.CacheClient instance at 0x7fda0790bc20>
value is none!
Traceback (most recent call last):
File "app.py", line 80, in <module>
value = id_to_glance_id(context,image_id)
File "app.py", line 26, in memorizer
val = _CACHE.set(key,value)
File "/home/jorh/cache/cacheutils.py", line 78, in set
return self.region.set(key,value)
File "/home/jorh/.local/lib/python2.7/site-packages/dogpile/cache/region.py", line 1112, in set
self.backend.set(key, self._value(value))
File "/home/jorh/.local/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 179, in set
self.client.set(key, value, **self.set_arguments)
File "/home/jorh/.local/lib/python2.7/site-packages/oslo_cache/backends/memcache_pool.py", line 32, in _run_method
return getattr(client, __name)(*args, **kwargs)
File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 727, in set
return self._set("set", key, val, time, min_compress_len, noreply)
File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 1016, in _set
server, key = self._get_server(key)
File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 433, in _get_server
if server.connect():
File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 1381, in connect
if self._get_socket():
File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 1413, in _get_socket
self.flush()
File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 1488, in flush
self.expect(b'OK')
File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 1463, in expect
line = self.readline(raise_exception)
File "/home/jorh/.local/lib/python2.7/site-packages/memcache.py", line 1449, in readline
data = recv(4096)
socket.timeout: timed out
可以看到上面沒有讀出相應的值的,並且出現錯誤!
如果關閉SASL之後,再執行代碼會出現什麼情況呢!
首先說明一下,如何關閉memcached的SALA認證,將start文件sudo ./memcached -d -f 1.1 -L -S -m 2048 -u os-user中的-S去掉,然後再將memcached這個服務重啓一下就行了!有可能不會成功,這個時候,可能需要重啓一下電腦纔行!
執行python app.py命令,發現可以正常讀出數據
[jorh@localhost cache]$ python app.py
the caching is True and the cache_time is 7200
_get_default_cache_region
<dogpile.cache.region.CacheRegion object at 0x7fc649207d50>
_CACHE is <cacheutils.CacheClient instance at 0x7fc6575f3c20>
value is none!
the set return value is None
the value is this is image_id:1234
_get_default_cache_region
<dogpile.cache.region.CacheRegion object at 0x7fc64719ca10>
cl.add return value: val = 90
key = mykey
VAL1 is 90
VAL2 is jihhhhsnna
_get_default_cache_region
<dogpile.cache.region.CacheRegion object at 0x7fc649207dd0>
cl1 VAL3 is jihhhhsnna
_get_default_cache_region
<dogpile.cache.region.CacheRegion object at 0x7fc6471a6250>
key = mykey
value2 is jihhhhsnna
走到這裏,就非常高興了,說明可以正常開啓和關閉帶有SALA認證的memcached服務
下面,我們將修改代碼,讓oslo_cache模塊支持開啓SALA認證的memcached服務!
首先修改cache.conf文件,裏面增加如下的三個參數
memcache_sasl_enable = true
memcache_usrname = username
memcache_password = Password
之後,通過位於oslo_cache/core.py文件中的configure(conf)函數,將conf中的參數進行配置:
在configure函數中添加如下的內容(下面內容在oslo_cache/_opts.py文件中):
FILE_OPTIONS = {
'cache': [
...........
cfg.BoolOpt('memcache_sasl_enable',default=False,
help='Enable the SASL(Simple Authentication and Security Layer) if the sasl_enable is true, '
'else disable!'),
cfg.StrOpt('memcache_usrname',default='usrname',
help='the user name for the SASL'),
cfg.StrOpt('memcache_password',default='password',
help="the user's password for SASL!"),
],
}
之後,configure函數會將cache組中的所有配置項進行註冊,然後就可以通過CONF.cache.SALA_enable, CONF.cache.os_user以及CONF.cache.passwd來調用相應的參數!
之後,會創建一個region對象,然後將這個CONF和region進行結合
region = cache.create_region()
cache.configure_cache_region(CONF,region)
下面我們看一下configure_cache_region(CONF,region)這個函數是如何修改(oslo_cache/core.py文件中):
在這個函數中主要調用兩個函數:
config_dict = _build_cache_config(conf)
region.configure_from_config(config_dict,'%s.' % conf.cache.config_prefix)
第一個函數主要用於配置config_dict這個字典參數。
然後修改並添加如下的代碼:
'''
在括號內,添加如下的三個關鍵字,'sala_enable', 'os_user', 'passwd'
'''
for arg in ('dead_retry', 'socket_timeout', 'pool_maxsize',
'pool_unused_timeout', 'pool_connection_get_timeout',
'sasl_enable', 'usrname', 'password'):
value = getattr(conf.cache, 'memcache_' + arg)
conf_dict['%s.arguments.%s' % (prefix, arg)] = value
後面就會將這個conf_dict返回,然後我們的三個參數就在這個字典中了!
雖然現在看起來字典中的key = oslo.cache.arguments,不知道爲什麼一定要加上這個prefix
後面當這個字典傳到backend類中的時候,會將字典中key中的這個前綴去掉(感覺上面多此一舉!)
region.configure_from_config這個函數主要用來確定存儲後端backend的,這裏不用進行改動!
現在工作已經完成一半了,因爲我們已經得到了我們想要的參數,並且將這個參數放到字典中,後面只要能得到這個字典,我們就可以得到我們需要的這三個參數!
因爲我們這裏設置的後端存儲是:backend=oslo_cache.memcache_pool,上面也有打印相應的參數實際對應的類對象實例!
因此對應的方法是:oslo_cache.backends.memcache_pool.PooledMemcachedBackend
下面看一下PooledMemcachedBackend中是如何調用的,可以看如下的代碼:
from oslo_cache import _bmemcache_pool
def __init__(self, arguments):
#初始化父類中的__init__函數,後面可以直接調用父類中的屬性
super(PooledMemcachedBackend, self).__init__(arguments)
if arguments.get()
self.client_pool = _memcache_pool.MemcacheClientPool(
self.url,#這個參數是在GenericMemcachedBackend中__init__的時候得到的!
arguments={
'dead_retry': arguments.get('dead_retry', 5 * 60),
'socket_timeout': arguments.get('socket_timeout', 3.0),
'server_max_value_length':
arguments.get('server_max_value_length'),
},
maxsize=arguments.get('pool_maxsize', 10),
unused_timeout=arguments.get('pool_unused_timeout', 60),
conn_get_timeout=arguments.get('pool_connection_get_timeout', 10),
)
那麼,現在我們只需要重新構造一個類,讓他繼承bmemcached就可以了,然後根據arguments中的sala_enable是否爲True來判斷執行那個flow!
下面修改方案:增加一個_bmemcache_pool.py文件,在裏面導入bmemcached模塊,之後所有的代碼都和_memcache_pool.py中的一樣,只不過其中的類前面都添加b字符,以示區分;
def __init__(self, arguments):
#初始化父類中的__init__函數,後面可以直接調用父類中的屬性
super(PooledMemcachedBackend, self).__init__(arguments)
if arguments.get('sala_enable'):
self.client_pool = _bmemcache_pool.bMemcacheClientPool(
self.url, # 這個參數是在GenericMemcachedBackend中__init__的時候得到的!
username=arguments.get('os_name'),
password=arguments.get('passwd'),
arguments={
'socket_timeout': arguments.get('socket_timeout', 3.0),
},
maxsize=arguments.get('pool_maxsize', 10),
unused_timeout=arguments.get('pool_unused_timeout', 60),
conn_get_timeout=arguments.get('pool_connection_get_timeout', 10),
)
else:
self.client_pool = _memcache_pool.MemcacheClientPool(
self.url,#這個參數是在GenericMemcachedBackend中__init__的時候得到的!
arguments={
'dead_retry': arguments.get('dead_retry', 5 * 60),
'socket_timeout': arguments.get('socket_timeout', 3.0),
'server_max_value_length':
arguments.get('server_max_value_length'),
},
maxsize=arguments.get('pool_maxsize', 10),
unused_timeout=arguments.get('pool_unused_timeout', 60),
conn_get_timeout=arguments.get('pool_connection_get_timeout', 10),
)
_bmemcache_pool.py文件的改動如下:
import bmemcached
.......
'''
class _MemcacheClient類改爲class _bMemcacheClient,繼承bmemcached.Client
'''
class _bMemcacheClient(bmemcached.Client): #
...........
............
#將class ConnectionPool(queue.Queue)改爲class bConnectionPool(queue.Queue)
class bConnectionPool(queue.Queue):
...............
#將class MemcacheClientPool改爲class bMemcacheClientPool,並繼承bConnectionPool
class bMemcacheClientPool(bConnectionPool):
#在這個類中添加兩個變量
def __init__(self, urls,username,password, arguments, **kwargs):
# super() cannot be used here because Queue in stdlib is an
# old-style class
ConnectionPool.__init__(self, **kwargs)
self.urls = urls
#添加如下兩個變量!
self.username = username
self.password = password
self._arguments = arguments
# NOTE(morganfainberg): The host objects expect an int for the
# deaduntil value. Initialize this at 0 for each host with 0 indicating
# the host is not dead.
self._hosts_deaduntil = [0] * len(urls)
def _create_connection(self): #這裏返回bmemcached對象!
return _bMemcacheClient(self.urls, self.username,self.password,
**self._arguments)
def _get(self):
# super() cannot be used here because Queue in stdlib is an
# old-style class
conn = bConnectionPool._get(self)
..........
bConnectionPool._put(self, conn)
raise
return conn
def _put(self, conn):
try:
......
bConnectionPool._put(self, conn)
這樣整個代碼就改動完成了,下面開始正式測試流程!
測試出現如下的錯誤:
status = 32
Traceback (most recent call last):
File "app.py", line 85, in <module>
myfun2('mykey')
File "app.py", line 73, in myfun2
cl2.set(key,'jihhhhsnna')
File "/home/jorh/cache/cacheutils.py", line 82, in set
return self.region.set(key,value)
File "/home/jorh/.local/lib/python2.7/site-packages/dogpile/cache/region.py", line 1112, in set
self.backend.set(key, self._value(value))
File "/home/jorh/.local/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 179, in set
self.client.set(key, value, **self.set_arguments)
File "/home/jorh/.local/lib/python2.7/site-packages/oslo_cache/backends/memcache_pool.py", line 33, in _run_method
print("getattr(client, __name)(*args, **kwargs) = %s"%(getattr(client, __name)(*args, **kwargs)))
File "/usr/lib/python2.7/site-packages/bmemcached/client/replicating.py", line 112, in set
returns.append(server.set(key, value, time, compress_level=compress_level))
File "/usr/lib/python2.7/site-packages/bmemcached/protocol.py", line 612, in set
return self._set_add_replace('set', key, value, time, compress_level=compress_level)
File "/usr/lib/python2.7/site-packages/bmemcached/protocol.py", line 591, in _set_add_replace
raise MemcachedException('Code: %d Message: %s' % (status, extra_content), status)
bmemcached.exceptions.MemcachedException
現在發現在關閉SALA的時候,執行是可以的,打開SALA之後,就出現上述的錯誤,現在的可能原因是username和password參數可能傳入不對!
在_bmemcache_pool.py文件中,直接將__init__函數中的self._os-name和self._password設置爲對應的值之後,發現讀取正常了,那麼出現上述現象的原因就是傳入的認證參數不對!
添加代碼鏈接:https://download.csdn.net/download/qq_33970713/12558596