Skip to content
Advertisement

Pickle/unpickle only once per worker

I am using python multiprocessing module to spread say 10000 steps of a given task on 4 workers using a Pool. The task that is sent on the workers is a method of a complex object. If I understand right the documentation, pickle is used to dump and load the object at each step which means 10000 pickling/unpickling calls. My problem is that the object that is pickled is quite complex (it contains a lot of aggregations of nested complex objects) and the picking/unpickling process take some times. Hence, my job is very much slower when it is run using multiprocessor regarding a monoprocessor call. My question is the following: is there a way to do the pickle/unpickle process only once per worker instead of once per step ?

EDIT: The code I try to parallelize has the following (simplified) structure:

JavaScript

Advertisement

Answer

How about using Processes instead?

If such a structure is feasible for your use case, you can create another function for workers which run any target function you require. Then start the worker functions using multiprocessing.Process like below:

JavaScript

Output:

JavaScript

Pickles/Unpickles only once per worker. You could probably replicate the same thing in pools but I find this more straightforward.

Advertisement