Running a command line program and reading the result as it runs in Python

Question

I'm using Python 2.7 under Windows and am trying to run a command line program and process the programs output as it is running. A number of web searches have indicated that the following code would work.

import subprocess

p = subprocess.Popen("D:PythonPython27Scriptspip.exe list -o",
 stdout=subprocess.PIPE,
 stderr=subprocess.STDOUT,
 bufsize=1,
 universal_newlines=True,
 shell=False)
for line in p.stdout:
 print line

When I use this code I can see that the Popen works, any code between the Popen and the for will run straight away, but as soon as it gets to the for and tries to read p.stdout the code blocks until the command
line program completes, then all of the lines are returned. Does anyone know how to get the results of the program without it blocking?

Answer 1

Is the program actually producing output progressively? I just tried your exact code with "dir /ad /s /b" and it worked fine, producing output while the dir was still spinning (obviously setting shell=True to make that work, but I don't think that'll make a difference). It may be that pip buffers its output. Is there a parameter to pip to make it pipe-compatible?

Comments

If I run pip in the command window I can see it's output appearing line  by line rather than on one block.

I tried the code with the dir command but it's too fast for me to be sure if it's working or not.

I tried again using the command "ping google.com" instead since I know that output's slowly and it something that everyone should have. In the command window I can see that the output appears over time, but from python I get nothing for a while and then suddenly get all the output in one rapid go.

Can you think of anything else I can look at?

---

A lot of programs, when their output is not going to the console, will buffer output. It's more efficient for many purposes. With Unix utilities, there's often a parameter like --pipe or --unbuffered that says "please produce output line by line", but Windows ping doesn't have that - and so I'm seeing the same thing you are.

You should be able to see the time delay in dir by looking for some particular directory name, and searching from the root directory. Unless you're on a BLAZINGLY fast drive, that'll take Windows a good while!

Answer 2

When file object is used in a for loop it works like an iterator and then it uses a hidden read-ahead buffer.
It might cause this kind of blocking. You can read more details here (description of method next):
http://docs.python.org/lib/bltin-file-objects.html

So basically non-blocking loop might look like this:

while True:
 line = p.stdout.readline()
 if not line: break
 print line