快捷導(dǎo)航

說一說Python logging

更新時(shí)間：2016年04月15日 08:49:40 作者：正_午

這篇文章主要和大家聊一聊Python logging，Python logging是什么，Python logging的作用是什么，感興趣的小伙伴們可以參考一下

最近有個(gè)需求是把以前字符串輸出的log 改為json 格式,看了別人的例子,還是有些比較茫然,索性就把logging 整個(gè)翻了一邊,做點(diǎn)小總結(jié).

初看log

在程序中, log 的用處寫代碼的你用你知道,log 有等級(jí),DEBUG, INFO,...之類,還會(huì)記錄時(shí)間,log 發(fā)生的位置,在Python 中用的多的就是logging 這個(gè)標(biāo)準(zhǔn)庫(kù)中的包了.當(dāng)打log 的時(shí)候究竟發(fā)生了什么? 是如何把不同級(jí)別的log 輸出到不同文件里,還能在控制臺(tái)輸出.......

最簡(jiǎn)單的用法

import logging
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
1,第一行導(dǎo)入包 2,第二行利用basicConfig 對(duì)輸出的格式,和輸出級(jí)別做了限制 3, 后面分別輸出了三條不同級(jí)別的 log

Logging Levels

共有幾個(gè)等級(jí), 每個(gè)等級(jí)對(duì)應(yīng)一個(gè)Int 型整數(shù) ,每個(gè)等級(jí)都會(huì)有一個(gè)方法與之對(duì)應(yīng),這樣輸出的內(nèi)容就有了不同的等級(jí).

logger 流程,

整個(gè)過程,還是不是很詳細(xì),貼個(gè)圖吧, 現(xiàn)在看還太早,也說不清真?zhèn)€過程到底發(fā)生了什么,先放著,回頭來看會(huì)比較好懂. loger flow

讀代碼

代碼結(jié)構(gòu)

logging 在源碼中有三個(gè)文件,結(jié)構(gòu)如下:

├── config.py
├── handlers.py
└── __init__.py
_int.py中實(shí)現(xiàn)了基礎(chǔ)功能,主要的邏輯就在這個(gè)文件中 handlers.py 是一些Handlers (用處后面會(huì)明白)用起來很方便的. config.py 是對(duì)配置做處理的方法.

objects

LogRecord Objects

每一次log 都會(huì)實(shí)例化一個(gè)Record 對(duì)象，這個(gè)對(duì)象有很多屬性,最后對(duì)LogRecord 做一下format 就輸出了，格式化的log ，里面就基本就是這個(gè)對(duì)象的屬性了。

class LogRecord(object):
  def __init__(self, name, level, pathname, lineno,
         msg, args, exc_info, func=None):
    ct = time.time()
    self.name = name
    self.msg = msg
    if (args and len(args) == 1 and isinstance(args[0], collections.Mapping)
      and args[0]):
      args = args[0]
    self.args = args
    self.levelname = getLevelName(level)
    self.levelno = level
    self.pathname = pathname
    try:
      self.filename = os.path.basename(pathname)
      self.module = os.path.splitext(self.filename)[0]
    except (TypeError, ValueError, AttributeError):
      self.filename = pathname
      self.module = "Unknown module"
    self.exc_info = exc_info
    self.exc_text = None   # used to cache the traceback text
    self.lineno = lineno
    self.funcName = func
    self.created = ct
    self.msecs = (ct - long(ct)) * 1000
    self.relativeCreated = (self.created - _startTime) * 1000
    if logThreads and thread:
      self.thread = thread.get_ident()
      self.threadName = threading.current_thread().name
    else:
      self.thread = None
      self.threadName = None
    if not logMultiprocessing:
      self.processName = None
    else:
      self.processName = 'MainProcess'
      mp = sys.modules.get('multiprocessing')
      if mp is not None:
        try:
          self.processName = mp.current_process().name
        except StandardError:
          pass
    if logProcesses and hasattr(os, 'getpid'):
      self.process = os.getpid()
    else:
      self.process = None

  def __str__(self):
    return '<LogRecord: %s, %s, %s, %s, "%s">'%(self.name, self.levelno,
      self.pathname, self.lineno, self.msg)

  def getMessage(self):
     pass

看代碼就發(fā)現(xiàn)，這個(gè)類沒做什么事情，就是一個(gè)model 而已, 有一個(gè)得到msg 的方法

Formatter Objects

Formatter 就是對(duì)Record 專門格式化的對(duì)象，它有一個(gè)format 方法，我們實(shí)現(xiàn)這個(gè)方法就能做到不同的輸出，我的需求是做json 格式的log 其實(shí)關(guān)鍵就在寫一個(gè)Formatter 就好了

class Formatter(object):
  converter = time.localtime

  def __init__(self, fmt=None, datefmt=None):
    if fmt:
      self._fmt = fmt
    else:
      self._fmt = "%(message)s"
    self.datefmt = datefmt

  def formatTime(self, record, datefmt=None):
    pass


  def formatException(self, ei):
    pass

  def usesTime(self):
    return self._fmt.find("%(asctime)") >= 0

  def format(self, record):
    pass

刪掉源代碼中的實(shí)現(xiàn)細(xì)節(jié)，這個(gè)類里面主要的是format 方法，這是默認(rèn)最基本的Formater ，還有專門對(duì)exception ，時(shí)間做格式化的方法。具體是哪個(gè)，看方法名就很清楚了，具體每個(gè)方法怎么實(shí)現(xiàn)的，一眼也就懂了。fmt 是制定格式化的，具體怎么指定在最基礎(chǔ)的用法中就有例子，datefmt 是對(duì)時(shí)間格式的指定。

Filter Objects

這個(gè)類是Logger 和Handler 的基類，主要有一個(gè)Filter 方法，和一個(gè)filters 屬性

Handler Objects

叫Handler 的類還真的不少,在SocketServer 中也有看到,具體的功能都在Handler 中.在這里,組合所有的Formatter ,和控制log 的輸出的方向，繼承自Filter.

 def __init__(self, level=NOTSET):
    Filterer.__init__(self)
    self._name = None
    self.level = _checkLevel(level)
    self.formatter = None
    _addHandlerRef(self)
    self.createLock()

在init方法中看到，Handler 也有一個(gè)屬性，通過把自身的屬性和LogRecord 的level對(duì)比來決定是否處理這個(gè)LogRecord 的。每個(gè)Handler 都有一個(gè)Formatter 屬性，其實(shí)就是上面介紹的Formatter 。Handler 就是來控制LogRecord 和Formatter 的，它還可以控制輸出的方式，在后面會(huì)有，StreamHandler,FileHandler等。通過名稱也就能明白具體能干什么，這就是編程取名的智慧。

Logger Objects

這個(gè)類通常會(huì)通過getLogger()或者getLogger(name)來得到,不會(huì)直接new 一個(gè)出來.它會(huì)有info(msg, *args, kwargs) ,warn(msg, args, *kwargs)等方法,

  def __init__(self, name, level=NOTSET):
    Filterer.__init__(self)
    self.name = name
    self.level = _checkLevel(level)
    self.parent = Noneou
    self.handlers = []
    self.disabled = 0

從init方法中能看到handlers 屬性,這是一個(gè)list ，每個(gè)LogRecord 通過Handlers 不同的handlers 就能以不同的格式輸出到不同的地方了。每個(gè)Logger 可以通過addHandler(hdlr)方法來添加各種Handler, 知道這些你就基本可以隨意定制化了下面就是我實(shí)現(xiàn)的json 格式的Formater，支持控制臺(tái)顏色變化，當(dāng)然前提是你的控制終端支持(Ubuntu14.04測(cè)試通過)

import re
import logging
import socket
import json
import traceback
import datetime
import time

try:
  from collections import OrderedDict
except ImportError:
  pass


RESERVED_ATTRS = (
  'args', 'asctime', 'created', 'exc_info', 'exc_text', 'filename',
  'funcName', 'levelname', 'levelno', 'lineno', 'module',
  'msecs', 'message', 'msg', 'name', 'pathname', 'process',
  'processName', 'relativeCreated', 'stack_info', 'thread', 'threadName')


RESERVED_ATTR_HASH = dict(zip(RESERVED_ATTRS, RESERVED_ATTRS))

COLORS ={
  'HEADER' : '\033[95m',
  'INFO' : '\033[94m',
  'DEBUG' : '\033[92m',
  'WARNING' : '\033[93m',
  'ERROR' : '\033[91m',
  'ENDC' : '\033[0m',
}

def merge_record_extra(record, target, reserved=RESERVED_ATTR_HASH):
  for key, value in record.__dict__.items():
    if (key not in reserved
      and not (hasattr(key, "startswith")
           and key.startswith('_'))):
      target[key] = value
  return target

def get_host_info():
  host_name = ''
  local_ip = ''
  try:
    host_name = socket.gethostname()
    local_ip = socket.gethostbyname(host_name)
  except Exception, e:
    pass

  return host_name, local_ip

class JsonFormatterBase(logging.Formatter):

  def __init__(self, *args, **kwargs):

    logging.Formatter.__init__(self, *args, **kwargs)
    self._required_fields = self.parse()
    self._skip_fields = dict(zip(self._required_fields,self._required_fields))
    self._skip_fields.update(RESERVED_ATTR_HASH)
  def parse(self):
    standard_formatters = re.compile(r'\((.+?)\)', re.IGNORECASE)
    return standard_formatters.findall(self._fmt)


  def add_fields(self, record ):
    log_record = {}

    for field in self._required_fields:
      log_record[field] = record.__dict__.get(field)

    host_name , local_ip = get_host_info()

    log_record[u'@hostName'] = host_name
    log_record[u'@localIp'] = local_ip
    return log_record

    #merge_record_extra(record, log_record, reserved=self._skip_fields)


  def process_log_record(self, log_record):
    """
    Override this method to implement custom logic
    on the possibly ordered dictionary.
    """

    try:
      new_record = OrderedDict()
    except Exception, e:
      return log_record

    key_list = [
      'asctime',
      'levelname',
      '@hostName',
      '@localIp',
      'threadName',
      'thread',
      'name',
      'pathname',
      'lineno',
      'message',
    ]
    for k in key_list:
      new_record[k] = log_record.get(k)
    new_record.update(log_record)
    return new_record

  def jsonify_log_record(self, log_record):
    """Returns a json string of the log record."""

    return json.dumps(log_record, ensure_ascii=False)


  def format_col(self, message_str, level_name):
    """

    是否需要顏色
    """
    return message_str

  def formatTime(self, record, datefmt=None):
    ct = self.converter(record.created)
    if datefmt:
      s = time.strftime(datefmt, ct)
    else:
      t = time.strftime("%Y-%m-%d %H:%M:%S", ct)
      s = "%s.%03d" % (t, record.msecs)
    return s

  def format(self, record):


    if isinstance(record.msg, dict):
      record.message = record.msg

    elif isinstance(record.msg, list) or isinstance(record.msg, tuple):
      record.message = record.msg

    elif isinstance(record.msg, basestring):
      record.message = record.getMessage().split('\n')

    elif isinstance(record.msg, Exception):
      record.message = traceback.format_exc(record.msg).split('\n')

    else :
      record.message = repr(record.msg)

    if "asctime" in self._required_fields:
      record.asctime = self.formatTime(record, self.datefmt)

    #
    # if record.exc_info and not message_dict.get('exc_info'):
    #   message_dict['message'] = traceback.format_exception(*record.exc_info)
    log_record = self.add_fields(record)
    log_record = self.process_log_record(log_record)
    message_str = self.jsonify_log_record(log_record)
    message_str = self.format_col(message_str, level_name=record.levelname)
    return message_str



class ConsoleFormater(JsonFormatterBase):

  def __init__(self, *args, **kwargs):
    JsonFormatterBase.__init__(self, *args, **kwargs)

  def format_col(self, message_str, level_name):
    if level_name in COLORS.keys():
      message_str = COLORS.get(level_name) + message_str + COLORS.get('ENDC')
    return message_str

  def jsonify_log_record(self, log_record):
    return json.dumps(log_record, ensure_ascii=False, indent=4)


class JsonFileFormater(JsonFormatterBase):

  def __init__(self, *args, **kewars):
    JsonFormatterBase.__init__(self, *args, **kewars)

  def jsonify_log_record(self, log_record):
    return json.dumps(log_record, ensure_ascii=False)

配置

很多時(shí)候我們并不是這樣自己去實(shí)現(xiàn)一些Handler ，F(xiàn)ormater ,之類的代碼，用logging 提供的config 就能做到了，如何寫config下面舉個(gè)例子解釋下，

SC_LOGGING_CONF = {
  "version": 1,
  "disable_existing_loggers": False,
  "formatters": {
    "simple": {
      "format": "%(asctime)s [%(levelname)s] [%(threadName)s:%(thread)d] [%(name)s:%(lineno)d] - %(message)s"
    }
  },

  "handlers": {
    "console": {
      "class": "logging.StreamHandler",
      "level": "DEBUG",
      "formatter": "simple",
      "stream": "ext://sys.stdout"
    },
    "info_file_handler": {
      "class": "logging.handlers.RotatingFileHandler",
      "level": "INFO",
      "formatter": "simple",
      "filename": PATH + "info-" + date.today().isoformat() + ".log",
      "maxBytes": 10485760,
      "backupCount": 20,
      "encoding": "utf8"
    },
    "error_file_handler": {
      "class": "logging.handlers.RotatingFileHandler",
      "level": "ERROR",
      "formatter": "simple",
      "filename": PATH + "errors-" + date.today().isoformat() + ".log",
      "maxBytes": 10485760,
      "backupCount": 20,
      "encoding": "utf8"
    }
  },
    "": {
      "level": "INFO",
      "handlers": ["console", "info_file_handler", "error_file_handler"]
    }
  }
}

首先定義了一個(gè)formater 叫simaple , 然后定義了三個(gè)Handler ,分別是輸出到控制臺(tái)，輸出到文件和info,error的。

logging.config.dictConfig(CONFIG.SC_LOGGING_CONF)
通過這句就能讓這些配置產(chǎn)生效果了，這也是config.py做的事情，不需要寫很多代碼也能定制個(gè)性化的log.。

以上就是本文的全部?jī)?nèi)容，希望對(duì)大家的學(xué)習(xí)有所幫助。

您可能感興趣的文章: