Basic example on how setup a Python logger

log.py

import logging
import sys
from logging.handlers import TimedRotatingFileHandler

FORMATTER = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
LOG_FILE = "my_app.log"

def get_console_handler():
	console_handler = logging.StreamHandler(sys.stdout)
	console_handler.setFormatter(FORMATTER)
	return console_handler

def get_file_handler():
	file_handler = TimedRotatingFileHandler(LOG_FILE, when='midnight')
	file_handler.setFormatter(FORMATTER)
	return file_handler

def get_logger(logger_name):
	logger = logging.getLogger(logger_name)

	logger.setLevel(logging.DEBUG) # better to have too much log than not enough

	logger.addHandler(get_console_handler())
	logger.addHandler(get_file_handler())

	# with this pattern, it's rarely necessary to propagate the error up to parent
	logger.propagate = False

	return logger

how would you write it in a class? for using it in some other modules too

how would you write it in a class? for using it in some other modules too

You can try with this

import logging
import sys
from logging import Logger
from logging.handlers import TimedRotatingFileHandler


class MyLogger(Logger):
    def __init__(
        self,
        log_file=None,
        log_format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
        *args,
        **kwargs
    ):
        self.formatter = logging.Formatter(log_format)
        self.log_file = log_file

        Logger.__init__(self, *args, **kwargs)

        self.addHandler(self.get_console_handler())
        if log_file:
            self.addHandler(self.get_file_handler())

        # with this pattern, it's rarely necessary to propagate the| error up to parent
        self.propagate = False

    def get_console_handler(self):
        console_handler = logging.StreamHandler(sys.stdout)
        console_handler.setFormatter(self.formatter)
        return console_handler

    def get_file_handler(self):
        file_handler = TimedRotatingFileHandler(self.log_file, when="midnight")
        file_handler.setFormatter(self.formatter)
        return file_handler

How to use MyLogger class in custom classes ? for example have classes Dog() and Cat()

Hello @toren77,

As I understand we need to add logging.setLoggerClass(MyLogger) to start using our custom Logger implementation.

@nguyenkims Thank you so much for this amazing class. Would be great if you can share some usage example. Struggling with a couple of parameters in myLogger() class. Thanks again.

@urwa I'd suggest using the log.py directly, for example

import log

logger = log. get_logger("any_name")

logger.debug("this is a debug message")
logger.info("this is a debug message")

Hello @nguyenkims
To be honest I don't really understand how the construction with log class is going to work (class MyLogger).
For example, how to provide log_file to it during initialization ?

Doing something like won't allow to provide log_file name:

logging.setLoggerClass(MyLogger)
logger = logging.getLogger(__name__)

Am I missing something ?

@nykolab the class is there rather for illustrating how the first code snippet can be used as a class. I'd suggest using the log.py directly, for example:

import log

logger = log. get_logger("any_name")

logger.debug("this is a debug message")
logger.info("this is a debug message")

Hey @nguyenkims, tried the class out. A call-in one function for instance:

import log
custom_logger = log.getLogger(__name__)


def do_something():
    custom_logger.info(f"Random logs")

However:

 logger = log.get_logger(__name__)
AttributeError: module 'log' has no attribute 'get_logger'

Perhaps I'm looking at it wrong.

What's the license for this code?

@danielplatt: it's MIT or you can use it freely without any restriction :).

Thanks!

Where do you set logging level in class level logging

@Tamilvanantm: good catch! The class-based implementation is for illustrating the idea and I suggest using function-based implementation (log.py ) instead.

I too have been trying to wrap my head around what you're doing here, especially with the class. You've put so much time into this, why not write a complete, functional logging class?

Let's say you have multiple modules and in each one you do get_logger from the log.py. Isn't the log then configured again every time you include a module?

I was under the impression that the best practice would be to do something along the lines of what get_logger does while starting the application, and then do

import logging
logger = logging.getLogger(<logger_name>)

in each module where you wish to use it. Am I wrong?

I usually have a single logger that is shared for all modules in my app it's just my preference.

The above class can be executed as follows:
logger2 = MyLogger(log_file=, name = )

logger2.debug("this is a debug message")
logger2.info("this is a info message")

Great Logger!

Question: When I change the name of the LOG_FILE = "my_app.log" to a path + filename I get an error: "The process cannot access the file because it is being used by another process".

It seems that the TimedRotatingFileHandler() function can only handle a filename and not a path + filename. Do you know why this happens, and how I could fix this issue?

Thanks!

@dkooten hey I just tested with LOG_FILE = "/tmp/my_app.log" and it seems that the script works fine. I wonder if the path you use exist or you have the right access to it?

Why do I have to restart my project to create a new log file by date? Is there a way to automatically generate log files?

--- Logging error ---
Traceback (most recent call last):
File "D:\PYTHON SETUP\lib\logging\handlers.py", line 70, in emit
self.doRollover()
File "D:\PYTHON SETUP\lib\logging\handlers.py", line 394, in doRollover
self.rotate(self.baseFilename, dfn)
File "D:\PYTHON SETUP\lib\logging\handlers.py", line 111, in rotate
os.rename(source, dest)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\PYTHON\QTASolution\Backend\logs\QTASolutions.log' -> 'D:\PYTHON\QTASolution\Backend\logs\QTASolutions.log.2022-03-24_10_52_53.log'
Call stack:
File "D:\PYTHON SETUP\lib\threading.py", line 890, in _bootstrap
self._bootstrap_inner()
File "D:\PYTHON SETUP\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "D:\PYTHON SETUP\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "D:\PYTHON SETUP\lib\socketserver.py", line 650, in process_request_thread
self.finish_request(request, client_address)
File "D:\PYTHON SETUP\lib\socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "D:\PYTHON SETUP\lib\socketserver.py", line 720, in init
self.handle()
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 347, in handle
BaseHTTPRequestHandler.handle(self)
File "D:\PYTHON SETUP\lib\http\server.py", line 426, in handle
self.handle_one_request()
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 379, in handle_one_request
self.run_wsgi()
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 324, in run_wsgi
execute(self.server.app)
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 316, in execute
write(data)
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 274, in write
self.send_response(code, msg)
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 383, in send_response
self.log_request(code)
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 433, in log_request
self.log("info", '"%s" %s %s', msg, code, size)
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 442, in log
_log(
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug_internal.py", line 225, in _log
getattr(_logger, type)(message.rstrip(), *args, **kwargs)
Message: '192.168.100.143 - - [24/Mar/2022 10:54:30] "%s" %s %s'

'D:\PYTHON\QTASolution\Backend\logs\QTASolutions.log.2022-03-24_10_52_53.log'
This file has not been created, so it cannot be logged.

I have one suggestion, since I was getting repeating lines in the logs (one for each call of get_logger) and logger. propagate = False didn't help, adding if not logger.hasHandlers(): did:

import logging
import sys
from logging.handlers import TimedRotatingFileHandler

FORMATTER = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
LOG_FILE = "my_app.log"

def get_console_handler():
	console_handler = logging.StreamHandler(sys.stdout)
	console_handler.setFormatter(FORMATTER)
	return console_handler

def get_file_handler():
	file_handler = TimedRotatingFileHandler(LOG_FILE, when='midnight')
	file_handler.setFormatter(FORMATTER)
	return file_handler

def get_logger(logger_name):
    logger = logging.getLogger(logger_name)

    # better to have too much log than not enough
    logger.setLevel(logging.DEBUG)

    if not logger.hasHandlers():
        logger.addHandler(get_console_handler())
        logger.addHandler(get_file_handler())

    # with this pattern, it's rarely necessary to propagate the error up to parent
    logger.propagate = False

    return logger

@homoludens oh that's interesting, do you have a code that produces the code duplication using the get_logger()?