Basic example on how setup a Python logger

log.py

import logging
import sys
from logging.handlers import TimedRotatingFileHandler

FORMATTER = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
LOG_FILE = "my_app.log"

def get_console_handler():
	console_handler = logging.StreamHandler(sys.stdout)
	console_handler.setFormatter(FORMATTER)
	return console_handler

def get_file_handler():
	file_handler = TimedRotatingFileHandler(LOG_FILE, when='midnight')
	file_handler.setFormatter(FORMATTER)
	return file_handler

def get_logger(logger_name):
	logger = logging.getLogger(logger_name)

	logger.setLevel(logging.DEBUG) # better to have too much log than not enough

	logger.addHandler(get_console_handler())
	logger.addHandler(get_file_handler())

	# with this pattern, it's rarely necessary to propagate the error up to parent
	logger.propagate = False

	return logger

Thanks!

Where do you set logging level in class level logging

@Tamilvanantm: good catch! The class-based implementation is for illustrating the idea and I suggest using function-based implementation (log.py ) instead.

I too have been trying to wrap my head around what you're doing here, especially with the class. You've put so much time into this, why not write a complete, functional logging class?

Let's say you have multiple modules and in each one you do get_logger from the log.py. Isn't the log then configured again every time you include a module?

I was under the impression that the best practice would be to do something along the lines of what get_logger does while starting the application, and then do

import logging
logger = logging.getLogger(<logger_name>)

in each module where you wish to use it. Am I wrong?

I usually have a single logger that is shared for all modules in my app it's just my preference.

The above class can be executed as follows:
logger2 = MyLogger(log_file=, name = )

logger2.debug("this is a debug message")
logger2.info("this is a info message")

Great Logger!

Question: When I change the name of the LOG_FILE = "my_app.log" to a path + filename I get an error: "The process cannot access the file because it is being used by another process".

It seems that the TimedRotatingFileHandler() function can only handle a filename and not a path + filename. Do you know why this happens, and how I could fix this issue?

Thanks!

@dkooten hey I just tested with LOG_FILE = "/tmp/my_app.log" and it seems that the script works fine. I wonder if the path you use exist or you have the right access to it?

Why do I have to restart my project to create a new log file by date? Is there a way to automatically generate log files?

--- Logging error ---
Traceback (most recent call last):
File "D:\PYTHON SETUP\lib\logging\handlers.py", line 70, in emit
self.doRollover()
File "D:\PYTHON SETUP\lib\logging\handlers.py", line 394, in doRollover
self.rotate(self.baseFilename, dfn)
File "D:\PYTHON SETUP\lib\logging\handlers.py", line 111, in rotate
os.rename(source, dest)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\PYTHON\QTASolution\Backend\logs\QTASolutions.log' -> 'D:\PYTHON\QTASolution\Backend\logs\QTASolutions.log.2022-03-24_10_52_53.log'
Call stack:
File "D:\PYTHON SETUP\lib\threading.py", line 890, in _bootstrap
self._bootstrap_inner()
File "D:\PYTHON SETUP\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "D:\PYTHON SETUP\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "D:\PYTHON SETUP\lib\socketserver.py", line 650, in process_request_thread
self.finish_request(request, client_address)
File "D:\PYTHON SETUP\lib\socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "D:\PYTHON SETUP\lib\socketserver.py", line 720, in init
self.handle()
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 347, in handle
BaseHTTPRequestHandler.handle(self)
File "D:\PYTHON SETUP\lib\http\server.py", line 426, in handle
self.handle_one_request()
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 379, in handle_one_request
self.run_wsgi()
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 324, in run_wsgi
execute(self.server.app)
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 316, in execute
write(data)
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 274, in write
self.send_response(code, msg)
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 383, in send_response
self.log_request(code)
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 433, in log_request
self.log("info", '"%s" %s %s', msg, code, size)
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 442, in log
_log(
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug_internal.py", line 225, in _log
getattr(_logger, type)(message.rstrip(), *args, **kwargs)
Message: '192.168.100.143 - - [24/Mar/2022 10:54:30] "%s" %s %s'

'D:\PYTHON\QTASolution\Backend\logs\QTASolutions.log.2022-03-24_10_52_53.log'
This file has not been created, so it cannot be logged.

I have one suggestion, since I was getting repeating lines in the logs (one for each call of get_logger) and logger. propagate = False didn't help, adding if not logger.hasHandlers(): did:

import logging
import sys
from logging.handlers import TimedRotatingFileHandler

FORMATTER = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
LOG_FILE = "my_app.log"

def get_console_handler():
	console_handler = logging.StreamHandler(sys.stdout)
	console_handler.setFormatter(FORMATTER)
	return console_handler

def get_file_handler():
	file_handler = TimedRotatingFileHandler(LOG_FILE, when='midnight')
	file_handler.setFormatter(FORMATTER)
	return file_handler

def get_logger(logger_name):
    logger = logging.getLogger(logger_name)

    # better to have too much log than not enough
    logger.setLevel(logging.DEBUG)

    if not logger.hasHandlers():
        logger.addHandler(get_console_handler())
        logger.addHandler(get_file_handler())

    # with this pattern, it's rarely necessary to propagate the error up to parent
    logger.propagate = False

    return logger

@homoludens oh that's interesting, do you have a code that produces the code duplication using the get_logger()?