My python script uses subprocess to call a linux utility that is very noisy. I want to store all of the output to a log file and show some of it to the user. I thought the following would work, but the output doesn’t show up in my application until the utility has produced a significant amount of output.
#fake_utility.py, just generates lots of output over time import time i = 0 while True: print hex(i)*512 i += 1 time.sleep(0.5) #filters output import subprocess proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE) for line in proc.stdout: #the real code does filtering here print "test:", line.rstrip()
The behavior I really want is for the filter script to print each line as it is received from the subprocess. Sorta like what tee
does but with python code.
What am I missing? Is this even possible?
Update:
If a sys.stdout.flush()
is added to fake_utility.py, the code has the desired behavior in python 3.1. I’m using python 2.6. You would think that using proc.stdout.xreadlines()
would work the same as py3k, but it doesn’t.
Update 2:
Here is the minimal working code.
#fake_utility.py, just generates lots of output over time import sys, time for i in range(10): print i sys.stdout.flush() time.sleep(0.5) #display out put line by line import subprocess proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE) #works in python 3.0+ #for line in proc.stdout: for line in iter(proc.stdout.readline,''): print line.rstrip()
Advertisement
Answer
I think the problem is with the statement for line in proc.stdout
, which reads the entire input before iterating over it. The solution is to use readline()
instead:
#filters output import subprocess proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE) while True: line = proc.stdout.readline() if not line: break #the real code does filtering here print "test:", line.rstrip()
Of course you still have to deal with the subprocess’ buffering.
Note: according to the documentation the solution with an iterator should be equivalent to using readline()
, except for the read-ahead buffer, but (or exactly because of this) the proposed change did produce different results for me (Python 2.5 on Windows XP).