Logging in Python

Setting Up Python Logging Properly

Setting Up Python Logging Properly

Python's logging module requires configuration before it works. This code prints nothing:

import logging
logging.info("Hello world")

By default, the root logger only shows WARNING and above. While logging.basicConfig() exists, here's a more flexible setup.

My Custom Logging Setup

This configuration routes INFO+ logs to stdout and ERROR logs to stderr, with readable formatting:

import logging
import sys

def setup_logging(level="INFO"):
    logger = logging.getLogger()
    logger.handlers.clear()  # Remove existing handlers
    logger.setLevel(level)

    formatter = logging.Formatter(
        "%(asctime)s %(name)-12s %(levelname)-8s %(message)s"
    )

    # INFO/DEBUG/WARNING to stdout
    stdout_handler = logging.StreamHandler(sys.stdout)
    stdout_handler.setLevel(logging.DEBUG)
    stdout_handler.addFilter(lambda record: record.levelno < logging.ERROR)
    stdout_handler.setFormatter(formatter)

    # ERROR/CRITICAL to stderr
    stderr_handler = logging.StreamHandler(sys.stderr)
    stderr_handler.setLevel(logging.ERROR)
    stderr_handler.setFormatter(formatter)

    logger.addHandler(stdout_handler)
    logger.addHandler(stderr_handler)

Why Split stdout and stderr?

Separating streams helps log aggregators (like Google Cloud Logging) correctly categorize error vs. info logs. Without this, all logs may be treated as the same severity level.

For Notebooks and Long-Running Jobs

For Jupyter notebooks or remote sessions, logging to a file is more reliable:

def file_logging(filename, level="INFO"):
    logger = logging.getLogger()
    logger.handlers.clear()
    logger.setLevel(level)

    formatter = logging.Formatter(
        "%(asctime)s %(name)-12s %(levelname)-8s %(message)s"
    )

    file_handler = logging.FileHandler(f"{filename}.log")
    file_handler.setFormatter(formatter)
    logger.addHandler(file_handler)

Temporarily Disabling Logs in Notebooks

When running tests or code that generates excessive logs, you can temporarily suppress output:

# Save current level
pre_disable = logging.getLogger().getEffectiveLevel()

# Disable all logging
logging.disable(logging.CRITICAL)

# no logging from libraries, etc.

# Re-enable logging
logging.disable(pre_disable)

This is particularly useful in Jupyter notebooks to keep output cells clean during testing or data processing.

Example Usage

logging.info("Before setup")  # Prints nothing

setup_logging()
logging.info("Goes to stdout")
logging.error("Goes to stderr")

file_logging("myapp")
logging.info("Goes to myapp.log")

Output:

$ python test.py
2025-01-24 10:15:22,331 root         INFO     Goes to stdout
2025-01-24 10:15:22,331 root         ERROR    Goes to stderr

$ cat myapp.log
2025-01-24 10:15:22,332 root         INFO     Goes to myapp.log

Note: handlers.clear() prevents duplicate log messages when reconfiguring the logger.