python logging模塊註冊流程

python logging模塊註冊流程(以logging.config.dictConfig流程爲例)

最近在查看python logging相關的模塊，用到了dictConfig這一函數，嘗試着跟蹤了一下，捋一捋在調用dictConfig之後，這些都發生了什麼,我想，用fileConfig的流程也差不多，先記載dictConfig的加載流程，後面跟蹤源碼了之後再做更新。

logging.config.dictconfig源碼：:

dictConfigClass = DictConfigurator

def dictConfig(config):
    """Configure logging using a dictionary."""
    dictConfigClass(config).configure()

1. 代碼中可以看到執行config操作的實際是DictConfigurator這一實例的configure方法.那麼就涉及到DictConfigurator實例的初始化：

class DictConfigurator(BaseConfigurator):
    ...

1.1 DictConfigurator繼承自BaseConfigurator,查看其__init__方法:

class BaseConfigurator(object):
    ....

    def __init__(self, config):
        self.config = ConvertingDict(config)
        self.config.configurator = self

代碼中有兩點: ConvertingDict的實例self.config, 以及self.config.configurator引用了當前的Baseconfigurator實例,
那麼， 代碼中的ConvertingDict(config)是個什麼？繼續跟蹤:

1.2 ConvertingDict解析

“`
class ConvertingDict(dict, ConvertingMixin):
“”“A converting dictionary wrapper.”“”

def __getitem__(self, key):
    value = dict.__getitem__(self, key)
    return self.convert_with_key(key, value)

def get(self, key, default=None):
    value = dict.get(self, key, default)
    return self.convert_with_key(key, value)

def pop(self, key, default=None):
    value = dict.pop(self, key, default)
    return self.convert_with_key(key, value, replace=False)

``
可以看到，Convertingdict繼承自dict, 重寫了dict中的getitem,get,pop三個方法，先調用dict原有的方法，然後對獲取的值使用self.convert_with_key進行轉換,其代碼在ConvertingMixin`類中。

1.3 ConvertingMixin類

“`
class ConvertingMixin(object):
“”“For ConvertingXXX’s, this mixin class provides common functions”“”

def convert_with_key(self, key, value, replace=True):
    result = self.configurator.convert(value)
    #If the converted value is different, save for next time
    if value is not result:
        if replace:
            self[key] = result
        if type(result) in (ConvertingDict, ConvertingList,
                           ConvertingTuple):
            result.parent = self
            result.key = key
    return result

def convert(self, value):
    result = self.configurator.convert(value)
    if value is not result:
        if type(result) in (ConvertingDict, ConvertingList,
                           ConvertingTuple):
            result.parent = self
    return result

``
看到convert_with_key方法中，每次都會調用它所引用的self.configurator的convert方法, 在這個流程中，self.configurator就是當前的Baseconfigurator`實例.

1.4 Baseconfigurator的convert方法：

 def convert(self, value):

        """
        Convert values to an appropriate type. dicts, lists and tuples are
        replaced by their converting alternatives. Strings are checked to
        see if they have a conversion format and are converted if they do.
        """
        if not isinstance(value, ConvertingDict) and isinstance(value, dict):
            value = ConvertingDict(value)
            value.configurator = self
        elif not isinstance(value, ConvertingList) and isinstance(value, list):
            value = ConvertingList(value)
            value.configurator = self
        elif not isinstance(value, ConvertingTuple) and\
                 isinstance(value, tuple):
            value = ConvertingTuple(value)
            value.configurator = self
        elif isinstance(value, str): # str for py3k
            m = self.CONVERT_PATTERN.match(value)
            if m:
                d = m.groupdict()
                prefix = d['prefix']
                converter = self.value_converters.get(prefix, None)
                if converter:
                    suffix = d['suffix']
                    converter = getattr(self, converter)
                    value = converter(suffix)
        return value
 ```

 前面都是在遞歸，只有當value類型爲str類型時，纔會去進行轉換，其他都是遞歸轉換類型

##### 1.4.1 str數據的轉化

CONVERT_PATTERN = re.compile(r'^(?P<prefix>[a-z]+)://(?P<suffix>.*)$')

`CONVERT_PATTERN`是一個正則，匹配類似於一個協議，如：`prefix://suffix`.

接着`value_converters`：

value_converters = {
    'ext' : 'ext_convert',
    'cfg' : 'cfg_convert',
}

可以看到，這對應了兩種協議，`ext`和`cfg`. 看`ext://`協議對應的`ext_convert`方法：

def ext_convert(self, value):
    """Default converter for the ext:// protocol."""
    return self.resolve(value)

它其實調用了`self.resolve(value)`, `self`是當前的的`Baseconfigurator`, 查看resolve方法：

##### 1.4.1 `Baseconfigurator`的`resolve`方法:

def resolve(self, s):
    """
    Resolve strings to objects using standard import and attribute
    syntax.
    """
    name = s.split('.')
    used = name.pop(0)
    try:
        found = self.importer(used)
        for frag in name:
            used += '.' + frag
            try:
                found = getattr(found, frag)
            except AttributeError:
                self.importer(used)
                found = getattr(found, frag)
        return found
    except ImportError:
        e, tb = sys.exc_info()[1:]
        v = ValueError('Cannot resolve %r: %s' % (s, e))
        v.__cause__, v.__traceback__ = e, tb
        raise v

根據官方文檔，`ext`協議主要是用來引用其他的模塊，例如自定義模塊`ext://a.b.c`, 分析源碼：
先通過`.`號分割出各個小段，`self.importer`其實就是`__import__`函數, `resolve`方法會一直去循環加載模塊,最後把模塊返回。


#### 1.5 總結一下：
1. `Baseconfigurator`初始化過程是：初始化一個`ConvertingDict`實例, 並將該實例的`configurator`指向`Baseconfigurator`自身,
2. `ConvertingDict`的作用: 以`ext`爲例，在獲取對應的key:value值時，轉化爲協議對應的模塊，以供使用。


### 2. `DictConfigurator`的`configure`方法：
源碼如下：

“”“Do the configuration.”“”

    config = self.config
    if 'version' not in config:
        raise ValueError("dictionary doesn't specify a version")
    if config['version'] != 1:
        raise ValueError("Unsupported version: %s" % config['version'])
    incremental = config.pop('incremental', False)
    EMPTY_DICT = {}
    logging._acquireLock()
    try:
        if incremental:
            handlers = config.get('handlers', EMPTY_DICT)
            for name in handlers:
                if name not in logging._handlers:
                    raise ValueError('No handler found with '
                                     'name %r'  % name)
                else:
                    try:
                        handler = logging._handlers[name]
                        handler_config = handlers[name]
                        level = handler_config.get('level', None)
                        if level:
                            handler.setLevel(logging._checkLevel(level))
                    except Exception as e:
                        raise ValueError('Unable to configure handler '
                                         '%r: %s' % (name, e))
            loggers = config.get('loggers', EMPTY_DICT)
            for name in loggers:
                try:
                    self.configure_logger(name, loggers[name], True)
                except Exception as e:
                    raise ValueError('Unable to configure logger '
                                     '%r: %s' % (name, e))
            root = config.get('root', None)
            if root:
                try:
                    self.configure_root(root, True)
                except Exception as e:
                    raise ValueError('Unable to configure root '
                                     'logger: %s' % e)
        else:
            disable_existing = config.pop('disable_existing_loggers', True)

            logging._handlers.clear()
            del logging._handlerList[:]

            # Do formatters first - they don't refer to anything else
            formatters = config.get('formatters', EMPTY_DICT)
            for name in formatters:
                try:
                    formatters[name] = self.configure_formatter(
                                                        formatters[name])
                except Exception as e:
                    raise ValueError('Unable to configure '
                                     'formatter %r: %s' % (name, e))
            # Next, do filters - they don't refer to anything else, either
            filters = config.get('filters', EMPTY_DICT)
            for name in filters:
                try:
                    filters[name] = self.configure_filter(filters[name])
                except Exception as e:
                    raise ValueError('Unable to configure '
                                     'filter %r: %s' % (name, e))

            # Next, do handlers - they refer to formatters and filters
            # As handlers can refer to other handlers, sort the keys
            # to allow a deterministic order of configuration
            handlers = config.get('handlers', EMPTY_DICT)
            deferred = []
            for name in sorted(handlers):
                try:
                    handler = self.configure_handler(handlers[name])
                    handler.name = name
                    handlers[name] = handler
                except Exception as e:
                    if 'target not configured yet' in str(e):
                        deferred.append(name)
                    else:
                        raise ValueError('Unable to configure handler '
                                         '%r: %s' % (name, e))

            # Now do any that were deferred
            for name in deferred:
                try:
                    handler = self.configure_handler(handlers[name])
                    handler.name = name
                    handlers[name] = handler
                except Exception as e:
                    raise ValueError('Unable to configure handler '
                                     '%r: %s' % (name, e))

            # Next, do loggers - they refer to handlers and filters

            #we don't want to lose the existing loggers,
            #since other threads may have pointers to them.
            #existing is set to contain all existing loggers,
            #and as we go through the new configuration we
            #remove any which are configured. At the end,
            #what's left in existing is the set of loggers
            #which were in the previous configuration but
            #which are not in the new configuration.
            root = logging.root
            existing = list(root.manager.loggerDict.keys())
            #The list needs to be sorted so that we can
            #avoid disabling child loggers of explicitly
            #named loggers. With a sorted list it is easier
            #to find the child loggers.
            existing.sort()
            #We'll keep the list of existing loggers
            #which are children of named loggers here...
            child_loggers = []
            #now set up the new ones...
            loggers = config.get('loggers', EMPTY_DICT)
            for name in loggers:
                if name in existing:
                    i = existing.index(name) + 1 # look after name
                    prefixed = name + "."
                    pflen = len(prefixed)
                    num_existing = len(existing)
                    while i < num_existing:
                        if existing[i][:pflen] == prefixed:
                            child_loggers.append(existing[i])
                        i += 1
                    existing.remove(name)
                try:
                    self.configure_logger(name, loggers[name])
                except Exception as e:
                    raise ValueError('Unable to configure logger '
                                     '%r: %s' % (name, e))

            #Disable any old loggers. There's no point deleting
            #them as other threads may continue to hold references
            #and by disabling them, you stop them doing any logging.
            #However, don't disable children of named loggers, as that's
            #probably not what was intended by the user.
            #for log in existing:
            #    logger = root.manager.loggerDict[log]
            #    if log in child_loggers:
            #        logger.level = logging.NOTSET
            #        logger.handlers = []
            #        logger.propagate = True
            #    elif disable_existing:
            #        logger.disabled = True
            _handle_existing_loggers(existing, child_loggers,
                                     disable_existing)

            # And finally, do the root logger
            root = config.get('root', None)
            if root:
                try:
                    self.configure_root(root)
                except Exception as e:
                    raise ValueError('Unable to configure root '
                                     'logger: %s' % e)
    finally:
        logging._releaseLock()

#### 2.1 `incremental`
從代碼中可一個看到有個`incremental`參數:

incremental = config.pop('incremental', False)

這個參數的作用是什麼?
##### 2.1.1 假設`incremental`參數爲`True`其核心在這裏:

for name in loggers:
    try:
        self.configure_logger(name, loggers[name], True)
    except Exception as e:
        raise ValueError('Unable to configure logger '

這個例子中， `configure_logger`在`DictConfigurator`中， 代碼如下:

def common_logger_config(self, logger, config, incremental=False):
    """
    Perform configuration which is common to root and non-root loggers.
    """
    level = config.get('level', None)
    if level is not None:
        logger.setLevel(logging._checkLevel(level))
    if not incremental:
        #Remove any existing handlers
        for h in logger.handlers[:]:
            logger.removeHandler(h)
        handlers = config.get('handlers', None)
        if handlers:
            self.add_handlers(logger, handlers)
        filters = config.get('filters', None)
        if filters:
            self.add_filters(logger, filters)

def configure_logger(self, name, config, incremental=False):
    """Configure a non-root logger from a dictionary."""
    logger = logging.getLogger(name)
    self.common_logger_config(logger, config, incremental)
    propagate = config.get('propagate', None)
    if propagate is not None:
        logger.propagate = propagate

從以上代碼中可以看到， 當`incremental`爲`True`時，只會將日誌的等級`level`修改.

#### 2.2 `incremental`參數爲`False`時：

###### 2.2.1 清除logging中的`_handler`和`_handlerList`信息

logging._handlers.clear()
del logging._handlerList[:]

###### 2.2.2 初始化formmaters, 源碼如下

            formatters = config.get('formatters', EMPTY_DICT)
            for name in formatters:
                try:
                    formatters[name] = self.configure_formatter(
                                                        formatters[name])
                except Exception as e:
                    raise ValueError('Unable to configure '
                                     'formatter %r: %s' % (name, e))

先會獲取所有的formatter,然後通過configure_formatter方法來初始化.

def configure_formatter(self, config):
    """Configure a formatter from a dictionary."""
    if '()' in config:
        factory = config['()'] # for use in exception handler
        try:
            result = self.configure_custom(config)
        except TypeError as te:
            if "'format'" not in str(te):
                raise
            #Name of parameter changed from fmt to format.
            #Retry with old name.
            #This is so that code can be used with older Python versions
            #(e.g. by Django)
            config['fmt'] = config.pop('format')
            config['()'] = factory
            result = self.configure_custom(config)
    else:
        fmt = config.get('format', None)
        dfmt = config.get('datefmt', None)
        style = config.get('style', '%')
        cname = config.get('class', None)
        if not cname:
            c = logging.Formatter
        else:
            c = _resolve(cname)
        result = c(fmt, dfmt, style)
    return result

可以看到，參數當formatter配置中，有兩種情況：
1. 有名爲`()`的key

此時，調用configure_custom方法:

def configure_custom(self, config):
    """Configure an object with a user-supplied factory."""
    c = config.pop('()')
    if not callable(c):
        c = self.resolve(c)
    props = config.pop('.', None)
    # Check for valid identifiers
    kwargs = dict([(k, config[k]) for k in config if valid_ident(k)])
    result = c(**kwargs)
    if props:
        for name, value in props.items():
            setattr(result, name, value)
    return result

從代碼中可以看到，`config.pop('()')`的調用，在上文中有提到：

def pop(self, key, default=None):
    value = dict.pop(self, key, default)
    return self.convert_with_key(key, value, replace=False)

這就是來獲取需要調用的模塊, 最後調用對應的模塊，生成formatter,
注意一點： `props = config.pop('.', None)`, 配置`.`屬性可以設置對應實例的自身屬性，如:

    "console": {
        "class": "logging.StreamHandler",
        "level": "DEBUG",
        "formatter": "simple",
        ".": {
            "hei": 1
        }
    }

``
這個配置加上了.號屬性的設置，初始化之後，對應的Streamhandler就會有屬性hei`
2. 正常的key
直接調用對應的class模塊，生成formatter

2.2.3 生成filter、Handler，與formatter類似, Handler最後會加上對應的formmaters以及filters

2.2.4 生成logger

這些就是logging.config.dictConfig的初始化過程，一步步解析，通過自帶協議：ext://和cfg://協議來引用外部模塊或是已有的配置,初始化你自己需求的logger，也可自定義相關模塊，完成特殊化需求,粗糙之問，有點亂。有錯誤，麻煩之處，好更正。