duplication of messages in stderr and stdout python logging

I would like to log info and warning level to stdout and error level and above to stderr how can I do it ?

code:

import os
import subprocess
from typing import Union
import logging
import yaml
import logging.config

with open("logging.conf", "r") as f:
    config = yaml.safe_load(f.read())
    logging.config.dictConfig(config)


logger = logging.getLogger(__name__)    


def run(
    cmd: Union[str, list[str]],
    *,
    cwd: Union[str, os.PathLike[str], None] = None,
    shell: bool = False,
    check: bool = True,
    capture_output: bool = True,
    text: bool = True,
) -> subprocess.CompletedProcess[str]:
    logger.info("start run!!")
    return subprocess.run(cmd, check=True, capture_output=capture_output, text=text)


cmd = ["lxs"]


def main():
    try:
        output = run(cmd)
    except Exception as e:
        logger.error(e, exc_info=False)


main()

logging.conf

version: 1
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers:
  outconsole:
    class: logging.StreamHandler
    level: INFO
    formatter: simple
    stream: ext://sys.stdout
  errconsole:
    class: logging.StreamHandler
    level: WARNING
    formatter: simple
    stream: ext://sys.stderr

  file_handler:
    class: logging.FileHandler
    level: INFO
    formatter: simple
    filename: info.log
    encoding: utf8
    mode: w

loggers:
  sampleLogger:
    level: DEBUG
    handlers: [file_handler]
    propagate: no
root:
  level: INFO
  handlers: [errconsole, outconsole, file_handler]
  propagate: no

and output after python3 -m tests 1> output.log 2> err.log:

err.log

2022-10-14 14:34:02,354 - __main__ - ERROR - [Errno 2] No such file or directory: 'lxs'

output.log

2022-10-14 14:34:02,353 - __main__ - INFO - start run!!
2022-10-14 14:34:02,354 - __main__ - ERROR - [Errno 2] No such file or directory: 'lxs'

Consequently, the output on the console is:

2022-10-14 14:48:33,178 - __main__ - INFO - start run!!
2022-10-14 14:48:33,179 - __main__ - ERROR - [Errno 2] No such file or directory: 'lxs'
2022-10-14 14:48:33,179 - __main__ - ERROR - [Errno 2] No such file or directory: 'lxs'

My question is how to make the message not repeat itself ? I want ERROR on strerr and INFO on stdout.

Answer

Because of the way levels work in logging, if you want to restrict stdout to WARNING and below and restrict stderr to ERROR and above, you’ll need to employ filters in the solution. One way of doing it is as follows.

First, add a filters entry parallel to formatters and handlers, to define a couple of filters:

filters:
  warnings_and_below:
    "()": __main__.filter_maker
    level: WARNING
    sense: below
  errors_and_above:
    "()": __main__.filter_maker
    level: ERROR
    sense: above

and refer to them in the handler configurations:

filters: [warnings_and_below]  # for outconsole

and

filters: [errors_and_above]  # for errconsole

Filters are just functions, and we’ll define filter_maker as follows:

def filter_maker(level, sense):
    level = getattr(logging, level)  # get the actual numeric value from the string
    if sense == 'below':  # return a function which only passes if level is at or below threshold
        def filter(record):
            return record.levelno <= level
    else:  # return a function which only passes if level is at or above threshold
        def filter(record):
            return record.levelno >= level
    return filter

I’m using a small script and running it directly, and so the filter_maker will be in the __main__ module (as indicated in the “()” lines above). You might need to change that to fit your specific situation.

Now, if I add some lines to log stuff:

logging.config.dictConfig(...)
logging.debug('A DEBUG message')
logging.info('An INFO message')
logging.warning('A WARNING message')
logging.error('An ERROR message')
logging.critical('A CRITICAL message')

and suppose all this is in main.py, then you can run

python main.py 2>stderr.log >stdout.log

and then you should see what’s expected:

$ more stdout.log 
2022-10-14 16:41:09,867 - root - INFO - An INFO message
2022-10-14 16:41:09,867 - root - WARNING - A WARNING message

$ more stderr.log
2022-10-14 16:41:09,868 - root - ERROR - An ERROR message
2022-10-14 16:41:09,868 - root - CRITICAL - A CRITICAL message

$ more info.log
2022-10-14 16:41:09,867 - root - INFO - An INFO message
2022-10-14 16:41:09,867 - root - WARNING - A WARNING message
2022-10-14 16:41:09,868 - root - ERROR - An ERROR message
2022-10-14 16:41:09,868 - root - CRITICAL - A CRITICAL message