Skip to content
Advertisement

How to get passed parameters to return values in multiprocessing Pool?

For example:

from multiprocessing import Pool
from functools import partial

numbers = []
for i in range(100):
    numbers.append(i)

def add_one(number, new_numbers):
    new = number + 1
    new_numbers.append(new)

new_numbers = []
process_pool = Pool(2)
add_one_helper = partial(add_one, new_numbers=new_numbers)

process_pool.map(add_one_helper, numbers)

print(new_numbers)

I suppose the new_numbers will print: 1, 2, 3, 4…, but it is empty in print(). How to get new_numbers to be populated after the call?

Advertisement

Answer

First an observation. You have:

numbers = []
for i in range(100):
    numbers.append(i)

This is using a loop to create a list with values 0, 1, 2 … 99. But you could have just as easily and more efficiently specified:

numbers = list(range(100))

But in fact the iterable argument being passed to the Pool.map method doesn’t need to be a list (it will, however, be automatically converted to a list if it does not support the __len__ method). So instead of creating the numbers list, you could have simply passed instead range(100) as the iterable argument to the map call. Now on to the question at hand:

The Pool.map method returns a list of all the return values from calling the worker function (add_one in this case). So you just need to have add_one return the value it wants appended to the new list:

from multiprocessing import Pool

def add_one(number):
    return number + 1

process_pool = Pool(2)
new_numbers = process_pool.map(add_one, range(100))
print(new_numbers)

Prints:

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100]
User contributions licensed under: CC BY-SA
5 People found this is helpful
Advertisement