Return value from function within a class using multiprocessing

Tags: , , , ,



I have following piece of codes, which I want to run through multiprocessing, I wonder how can I get return values after parallel processing is finished. I prefer not to make any change to getdata function.

from multiprocessing import Process

class Calculation(object):
    def __init__(self, a, b, c):
        self.A, self.B, self.C= a, b, c
        
    def getdata(self):
        self.Product = self.A * self.B
        self.Subtract = self.C - self.B
        self.Addition = self.A + self.B + self.C
        return self.Product, self.Subtract, self.Addition
    

def foo():
    EXTERNAL_C=[10, 20, 30, 40, 50, 20, 40]
    c = [Calculation(a=4, b=5, c=n) for n in EXTERNAL_C]
    return c
    
K = []
for M in foo():
    p = Process(target=M.getdata)
    p.start()
    K.append(p)
for process in K:
    process.join()
print(K)

Output:

[<Process(Process-8, stopped[1])>, <Process(Process-9, stopped[1])>, <Process(Process-10, stopped[1])>, <Process(Process-11, stopped[1])>, <Process(Process-12, stopped[1])>, <Process(Process-13, stopped[1])>, <Process(Process-14, stopped[1])>]

Answer

The Calculation objects you create in your main process are copied into the spawned process, so there is no way to extract their state or get the return value of getdata without doing something explicit.

You could use a multiprocessing.Queue to store your results like so

from multiprocessing import Process, Queue


class Calculation(object):
    def __init__(self, a, b, c):
        self.A, self.B, self.C = a, b, c

    def getdata(self, id, queue):
        self.Product = self.A * self.B
        self.Subtract = self.C - self.B
        self.Addition = self.A + self.B + self.C
        queue.put(
            {
                "id": id,
                "Product": self.Product,
                "Subtract": self.Subtract,
                "Addition": self.Addition,
            }
        )


def foo():
    EXTERNAL_C = [10, 20, 30, 40, 50, 20, 40]
    c = [Calculation(a=4, b=5, c=n) for n in EXTERNAL_C]
    return c


K = []
f = foo()
queue = Queue()
for id, M in enumerate(f):
    p = Process(target=M.getdata, args=(id, queue))
    p.start()
    K.append(p)
for process in K:
    process.join()

results = [queue.get() for i in range(len(f))]
print(results)


Source: stackoverflow