python: subprocess returns nothing when running scripts which take longer time

Beginner here with subrpocess problem:

The below line works fine when I run both the scripts with less data which takes around 10-20minutes, however with bigger data to be processed the subprocess returns nothing once both the scripts are completed (lets say in an hour).

Also: Often with lesser data, it behaves abnormally as well i.e. not returning the status_code/going through. I then have to just run the snippet again and it works. Any solid reasoning would help alot!

 status_code = subprocess.run(f"python3 scriptA.py {param_1} & python3 scriptB.py {param_2}",
                                     shell=True).returncode

 if status_code == 0:
     print("subprocess ended ..")  # the control does not come here incase of huge files          
     some_other_file.main(some_param1, some_param2)

I fail to understand why that happens (or should I use a different approach?), Any help would be very much appreciated. Thank you!

Sample of scripts (both scriptA and scriptB):

def main(param_2):
    some_func_with_csv_operations()  # not returning anything
    more_funcs()

if __name__ == "__main__":
    param_2 = sys.argv[1]
    main(param_2)

Also, no scriptA or scriptB has anywhere sys.exit()

EDIT:

Screenshot:

tried printig the status_code before the if condition, nothing was printed and in the terminal, I see a cursor just blinking.

enter image description here

Also (looking for any python processes): using ps -ax | grep python shows no relevant information (Picture attached)

enter image description here

Answer

Try running your processes separately to get a better idea where the failure is occurring

running_processes = []
running_processes.append(subprocess.Popen(["python3", "scriptA.py", f"{param_1}"]))
running_processes.append(subprocess.Popen(["python3", "scriptB.py", f"{param_2}"]))

both_succeeded = True
for p in running_processes:
    ret = p.wait()
    if ret:   # anything other than zero will evaluate to True here
        both_succeeded = False
    cmd = " ".join(p.args)
    print(f'command "{cmd}" returned code {ret}'

if both_succeeded:
    do_more_things()

The above uses a list to hold the running processes so you can process them in a loop and avoid repetitive code. If you will definitely only have two subprocesses you could choose not to use the loop.

process1 = subprocess.Popen(shlex.split(f"python3 scriptA.py {param_1}"))
process2 = subprocess.Popen(shlex.split(f"python3 scriptB.py {param_2}"))

ret1 = process1.wait()
ret2 = process2.wait()

if ret1 == 0 and ret2 == 0:  # or "if not ret1 and not ret2:" would also work
    do_more_things()

notice that Popen requires a list of arguments and won’t accept a single string, but I’ve used shlex.split in this example to allow use of a single string for the command and achieve the same result.