Running a GNU parallel command using python subprocess

Here is a simple GNU parallel command that creates a file called “example_i.txt” inside an existing directory called “example_i“. It does this four times, for i from 1 to 4, with one job per core:

parallel -j 4 'cd example_{} && touch example_{}.txt' ::: {1..4}

Not very exciting, I know. The problem appears when I try to run this via python (v3.9) using the subprocess module as follows:

import subprocess
cmd = "parallel -j 4 'cd example_{} && touch example_{}.txt' ::: {1..4}"
subprocess.run(cmd, shell=True)

When doing so I get this error:

/bin/sh: 1: cd: can't cd to example_{1..4}

It looks like using the python subprocess call, bash is not triggering the call correctly as a GNU parallel command. Instead, it is substituting the {1..4} explicitly rather than dividing it up into four jobs.

I also tried this with the less advisable os.system(cmd) syntax and got back the same error.

PS: For context, this question stems from me trying to use UQpy (the RunModel module in particular) for uncertainty quantification of a Fortran code that was handed to me. Although this is not directly related to the question, it is relevant because I would like to know how to get this working using these tools as I am not at liberty to change them.

Answer

Following @Mark Setchell’s comment, indeed it appears that bash is not used by default on POSIX as can be seen in the documentation for subprocess. This is solved by explicitly telling subprocess to use bash by re-writting my python code snippet as:

import subprocess
cmd = "parallel -j 4 'cd example_{} && touch example_{}.txt' ::: {1..4}"
subprocess.run(cmd, shell=True, executable='/bin/bash')

It should be noted that although the argument executable is here being used in the subprocess.run() call, it is not directly a part of this class. The executable argument is actually part of the subprocess.Popen() class, but it is accessible to subprocess.run() by the **other_popen_kwargs argument.