The add_to_log
function defined in the following code is called thousands of times during the execution of a script:
LOGFILENAME = 'N:\Tools\DelOldTempSave.log' def add_to_log(text): with open(LOGFILENAME, 'a') as logfile: now = str(datetime.datetime.now().replace(microsecond=0)) logfile.write(now + ' ' + text + 'n') print(now + ' ' + text)
It usually works as expected, but once in a while it crashes with:
OSError: [Errno 22] Invalid argument: 'N:\Tools\DelOldTempSave.log'
I tested it 3 times, it crashed after 3200, 1700 and 10200 calls.
I think I can use a try
/ except
block and give it another try when it fails, maybe after a time.sleep(0.1)
, but it’s a little scary!
The open
function is used also on other parts of the script. Does it mean that I need to use a try
/ except
block wherever it is called?
Also, the open
function is used on other scripts. Does it mean that I need to use a try
/ except
block wherever it is called on whatever script?
And I can’t trust python when writing to a text file?
EDIT
This version works without errors:
import os, time, datetime LOGFILENAME = 'N:\Tools\text.log' NROWS = 30000 def add_to_log(text): try: with open(LOGFILENAME, 'a') as logfile: now = str(datetime.datetime.now().replace(microsecond=0)) logfile.write(now + ' ' + text + 'n') print(now + ' ' + text) time.sleep(0.1) except OSError as e: time.sleep(0.01) add_to_log(f'Crash: {e}') add_to_log(text) if os.path.exists(LOGFILENAME): os.unlink(LOGFILENAME) for n in range(NROWS): add_to_log(str(n))
There are still three problems with this:
- Decreasing the duration in
time.sleep(0.1)
increases the chances of error, so I’m not 100% sure 0.1 is a long enough duration to ensure no error - How do I know that this problem is restricted to this function and it will not spread to other functions that call
open
- Adding a 0.1 second pause on a function that is called 30000 times, adds 3000 seconds to the duration of the script
Note: N:
is a network drive. I want the log file to be visible to all computers.
Advertisement
Answer
And I can’t trust python when writing to a text file?
Yes, it is a network drive. I want the log file to be visible to all computers.
It sounds more like you can’t trust your network file system. Reliably appending to a file would require a (distributed network) lock to be taken out on the file, anyway, and if that works, it could be slow.
Rather than opening and closing a single log file every time you need to write something, I’d recommend generating a single random log file name at the start of your program, and writing there:
import secrets LOGFILENAME = f'N:/Tools/text.{secrets.token_urlsafe(8)}.log' LOGFILE = open(LOGFILENAME, "w") # will be closed when the program ends def add_to_log(text): now = datetime.datetime.now().replace(microsecond=0).isoformat() LOGFILE.write(f'{now} {text}n') LOGFILE.flush() # if you want to ensure the text is written print(f'{now} {text}')
If you need a single file later on, you can just concatenate all of those files together and sort
them.