There’s this external python script that I would like to call. It provides an async mode so that it returns the task id before it completes the whole process.
The mechanism works well when I execute in the command line. The task id returns on stdout immediately. But the main process actually forks a subprocess to do the backend job. So when I want to use bash script to get the task id, it hangs until the subprocess finishes. It’s not async at all.
So my question is, how can I get the main process output immediately instead of waiting for the subprocess complete?
e.g.
$ ./cmd args
{"task": 1}
$ x=`./cmd args`
<< it hungs until entire process completed and returns all result at once.
{"task": 1} {"task": 1} {"actual_result": "xxx"}
// It's the same using python
import subprocess
p = subprocess.Popen(["cmd", "args"], stdout= subprocess.PIPE, stderr = subprocess.PIPE)
out, err = p.communicate()
<< stuck here as well
Advertisement
Answer
I would not call this a fork – read this for the difference https://stackoverflow.com/questions/49627957/what-is-the-difference-between-subprocess-popen-and-os-fork#:~:text=It%20seems%20like%20subprocess.,to%20create%20a%20child%20process. . So you want let the subprocess run while main process print something.
communicate() blocks IO, which is your main problem. You can get rid of the PIPE, just let the SP print to stdout, or any file object. More aggressively, add ‘nohup’ to the front of the child command can make the parent process free to exit itself without worrying about the child. Thought it has side effect of making the child nohup from current shell.
If you insist the parent program should manage all print, use poll() to check the status of child process before you communicate() with it.