Skip to content

Instantly share code, notes, and snippets.

@nguyenkims
Last active February 13, 2024 07:59
Show Gist options
  • Save nguyenkims/e92df0f8bd49973f0c94bddf36ed7fd0 to your computer and use it in GitHub Desktop.
Save nguyenkims/e92df0f8bd49973f0c94bddf36ed7fd0 to your computer and use it in GitHub Desktop.
Basic example on how setup a Python logger
import logging
import sys
from logging.handlers import TimedRotatingFileHandler
FORMATTER = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
LOG_FILE = "my_app.log"
def get_console_handler():
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setFormatter(FORMATTER)
return console_handler
def get_file_handler():
file_handler = TimedRotatingFileHandler(LOG_FILE, when='midnight')
file_handler.setFormatter(FORMATTER)
return file_handler
def get_logger(logger_name):
logger = logging.getLogger(logger_name)
logger.setLevel(logging.DEBUG) # better to have too much log than not enough
logger.addHandler(get_console_handler())
logger.addHandler(get_file_handler())
# with this pattern, it's rarely necessary to propagate the error up to parent
logger.propagate = False
return logger
@danielplatt
Copy link

Thanks!

@Tamilvanantmv
Copy link

Where do you set logging level in class level logging

@nguyenkims
Copy link
Author

@Tamilvanantm: good catch! The class-based implementation is for illustrating the idea and I suggest using function-based implementation (log.py ) instead.

@StL-Jim
Copy link

StL-Jim commented Sep 18, 2020

I too have been trying to wrap my head around what you're doing here, especially with the class. You've put so much time into this, why not write a complete, functional logging class?

Copy link

ghost commented Sep 22, 2020

Let's say you have multiple modules and in each one you do get_logger from the log.py. Isn't the log then configured again every time you include a module?

I was under the impression that the best practice would be to do something along the lines of what get_logger does while starting the application, and then do

import logging
logger = logging.getLogger(<logger_name>)

in each module where you wish to use it. Am I wrong?

@nguyenkims
Copy link
Author

I usually have a single logger that is shared for all modules in my app it's just my preference.

@snehotosh
Copy link

The above class can be executed as follows:
logger2 = MyLogger(log_file=, name = )

logger2.debug("this is a debug message")
logger2.info("this is a info message")

@dkooten
Copy link

dkooten commented Nov 9, 2021

Great Logger!

Question: When I change the name of the LOG_FILE = "my_app.log" to a path + filename I get an error: "The process cannot access the file because it is being used by another process".

It seems that the TimedRotatingFileHandler() function can only handle a filename and not a path + filename. Do you know why this happens, and how I could fix this issue?

Thanks!

@nguyenkims
Copy link
Author

@dkooten hey I just tested with LOG_FILE = "/tmp/my_app.log" and it seems that the script works fine. I wonder if the path you use exist or you have the right access to it?

@vanquang872003
Copy link

vanquang872003 commented Mar 24, 2022

Why do I have to restart my project to create a new log file by date? Is there a way to automatically generate log files?

--- Logging error ---
Traceback (most recent call last):
File "D:\PYTHON SETUP\lib\logging\handlers.py", line 70, in emit
self.doRollover()
File "D:\PYTHON SETUP\lib\logging\handlers.py", line 394, in doRollover
self.rotate(self.baseFilename, dfn)
File "D:\PYTHON SETUP\lib\logging\handlers.py", line 111, in rotate
os.rename(source, dest)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\PYTHON\QTASolution\Backend\logs\QTASolutions.log' -> 'D:\PYTHON\QTASolution\Backend\logs\QTASolutions.log.2022-03-24_10_52_53.log'
Call stack:
File "D:\PYTHON SETUP\lib\threading.py", line 890, in _bootstrap
self._bootstrap_inner()
File "D:\PYTHON SETUP\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "D:\PYTHON SETUP\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "D:\PYTHON SETUP\lib\socketserver.py", line 650, in process_request_thread
self.finish_request(request, client_address)
File "D:\PYTHON SETUP\lib\socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "D:\PYTHON SETUP\lib\socketserver.py", line 720, in init
self.handle()
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 347, in handle
BaseHTTPRequestHandler.handle(self)
File "D:\PYTHON SETUP\lib\http\server.py", line 426, in handle
self.handle_one_request()
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 379, in handle_one_request
self.run_wsgi()
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 324, in run_wsgi
execute(self.server.app)
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 316, in execute
write(data)
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 274, in write
self.send_response(code, msg)
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 383, in send_response
self.log_request(code)
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 433, in log_request
self.log("info", '"%s" %s %s', msg, code, size)
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug\serving.py", line 442, in log
_log(
File "D:\PYTHON\QTASolution\Backend\env\lib\site-packages\werkzeug_internal.py", line 225, in _log
getattr(_logger, type)(message.rstrip(), *args, **kwargs)
Message: '192.168.100.143 - - [24/Mar/2022 10:54:30] "%s" %s %s'

'D:\PYTHON\QTASolution\Backend\logs\QTASolutions.log.2022-03-24_10_52_53.log'
This file has not been created, so it cannot be logged.

@homoludens
Copy link

I have one suggestion, since I was getting repeating lines in the logs (one for each call of get_logger) and logger. propagate = False didn't help, adding if not logger.hasHandlers(): did:

import logging
import sys
from logging.handlers import TimedRotatingFileHandler

FORMATTER = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
LOG_FILE = "my_app.log"

def get_console_handler():
	console_handler = logging.StreamHandler(sys.stdout)
	console_handler.setFormatter(FORMATTER)
	return console_handler

def get_file_handler():
	file_handler = TimedRotatingFileHandler(LOG_FILE, when='midnight')
	file_handler.setFormatter(FORMATTER)
	return file_handler

def get_logger(logger_name):
    logger = logging.getLogger(logger_name)

    # better to have too much log than not enough
    logger.setLevel(logging.DEBUG)

    if not logger.hasHandlers():
        logger.addHandler(get_console_handler())
        logger.addHandler(get_file_handler())

    # with this pattern, it's rarely necessary to propagate the error up to parent
    logger.propagate = False

    return logger

@nguyenkims
Copy link
Author

@homoludens oh that's interesting, do you have a code that produces the code duplication using the get_logger()?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment