Issue in sending python logs to Splunk using splunk_hec_handler

I am using Python logging library to push logs to splunk. This package use HEC method to push logs to splunk.

Issue I am facing is that out of many logger statements in my application, I want selectively only few logger statements to splunk not all. So i created one method below method which converts string logs in json (key/value) and pushes into splunk. So I am calling this method just after the logger statement I wish to push to splunk. But rest all the logger statements which i dont wish to send to splunk they are also getting pushed to splunk.

Why is this happening?

class Test:


    def __init__(self):
 
        self.logger = logging.getLogger('myapp')

    def method_test(self,url,data,headers):

            response = requests.post(url=url, data=json.dumps(data), headers=abc.headers)

            ##i dont want to push this below log message to splunk but this is also getting pushed to splunk
            self.logger.debug(f"request code:{response.request.url} request body:{response.request.body}")

            ##I wish to send this below log to splunk
            self.logger.debug(f"response code:{response.status_code} response body:{response.text}")
            log_dic = {'response_code': response.status_code,'response_body': response.text}
            splunklogger = self.logging_override(log_dic, self.splunk_host,
                                               self.index_token, self.port,
                                               self.proto, self.ssl_verify,
                                                self.source)
            splunklogger.info(log_dic)


        return response    

    def logging_override(log_dict: dict, splunk_host,index_token,splunk_port,splunk_proto,ssl_ver,source_splnk):
        """
        This function help in logging custom fields in JSON key value form by defining fields of our choice in log_dict dictionary
        and pushes logs to Splunk Server
        """
        splunklogger = logging.getLogger()
        splunklogger.setLevel(logging.INFO)
        stream_handler = logging.StreamHandler()
        basic_dict = {"time": "%(asctime)s", "level": "%(levelname)s"}
        full_dict = {**basic_dict, **log_dict}
        stream_formatter = logging.Formatter(json.dumps(full_dict))
        stream_handler.setFormatter(stream_formatter)
        if not splunklogger.handlers:
            splunklogger.addHandler(stream_handler)
        splunklogger.handlers[0] = stream_handler
        splunk_handler = SplunkHecHandler(splunk_host,
                                          index_token,
                                          port=splunk_port, proto=splunk_proto, ssl_verify=ssl_ver,
                                          source=source_splnk)
        splunklogger.addHandler(splunk_handler)
        splunklogger.addHandler(splunk_handler)

        return splunklogger

Answer

I believe that the problem is with your calls to logging.getLogger, namely when you’re configuring your app logger, you’re specifying a logger name, but when you’re configuring the splunk logger, you’re not specifying any and therefore getting, configuring, and attaching the SplunkHandler to the root logger.

As events come in to the lower level loggers by default they propagate their events to higher level loggers (e.g. the root logger) and thus get emitted to Splunk.

I suspect an easy solution would be to look at your logger names… possibly put the Splunk logger at a lower level than your component? or look into the propagation of loggers. The same docs page linked above talks a bit about logger objects and their propagation.

Advertisement

Answer