I’m new on python. I want to learn how to parallel processing in python. I saw the following example:
JavaScript
x
21
21
1
import multiprocessing as mp
2
3
np.random.RandomState(100)
4
arr = np.random.randint(0, 10, size=[20, 5])
5
data = arr.tolist()
6
7
def howmany_within_range_rowonly(row, minimum=4, maximum=8):
8
count = 0
9
for n in row:
10
if minimum <= n <= maximum:
11
count = count + 1
12
return count
13
14
pool = mp.Pool(mp.cpu_count())
15
16
results = pool.map(howmany_within_range_rowonly, [row for row in data])
17
18
pool.close()
19
20
print(results[:10])
21
but when I run it, this error happened:
JavaScript
1
15
15
1
RuntimeError:
2
An attempt has been made to start a new process before the
3
current process has finished its bootstrapping phase.
4
5
This probably means that you are not using fork to start your
6
child processes and you have forgotten to use the proper idiom
7
in the main module:
8
9
if __name__ == '__main__':
10
freeze_support()
11
12
13
The "freeze_support()" line can be omitted if the program
14
is not going to be frozen to produce an executable.
15
What should I do?
Advertisement
Answer
If you place everything in global scope inside this if __name__ == "__main__"
block as follows, you should find that your program behaves as you expect:
JavaScript
1
19
19
1
def howmany_within_range_rowonly(row, minimum=4, maximum=8):
2
count = 0
3
for n in row:
4
if minimum <= n <= maximum:
5
count = count + 1
6
return count
7
8
if __name__ == "__main__":
9
np.random.RandomState(100)
10
arr = np.random.randint(0, 10, size=[20, 5])
11
data = arr.tolist()
12
pool = mp.Pool(mp.cpu_count())
13
14
results = pool.map(howmany_within_range_rowonly, [row for row in data])
15
16
pool.close()
17
18
print(results[:10])
19
Without this protection, if your current module was imported from a different module, your multiprocessing code would be executed. This could occur within a non-main process spawned in another Pool and spawning processes from sub-processes is not allowed, hence we protect against this problem.