How many processors should be used with multiprocessing.Pool?

I am trying to use multiprocessing.Pool to run my code in parallel. To instantiate Pool, you have to set the number of processes. I am trying to figure out how many I should set for this. I understand this number shouldn’t be more than the number of cores you have but I’ve seen different ways to determine what your system has available.

2 Methods:

1. multiprocessing.cpu_count()
2. len(os.sched_getaffinity(0))

I’m a little confused; what is the difference between the two and which should be implemented with Pool? I am working on a remote cluster, with the first, it outputs that there are 128 cpu, but the second gives 10.

Answer

The difference between the two is clearly stated in the doc:

multiprocessing.cpu_count() Return the number of CPUs in the system.

This number is not equivalent to the number of CPUs the current process can use. The number of usable CPUs can be obtained with len(os.sched_getaffinity(0)).

So even if you are on a 128-core system, your program could have been somehow limited to only run on a specific set of 10 out of the 128 available CPUs. Since affinity also applies to child threads and processes, it doesn’t make much sense to spawn more than 10. You could however try to increase the number of available CPUs through os.sched_setaffinity() before starting your pool.

import os
import multiprocessing as mp

cpu_count = mp.cpu_count() 

if len(os.sched_getaffinity(0)) < cpu_count:
    try:
        os.sched_setaffinity(0, range(cpu_count))
    except OSError:
        print('Could not set affinity')

n = max(len(os.sched_getaffinity(0)), 96)
print('Using', n, 'processes for the pool')

pool = mp.Pool(n)
# ...

See also man 2 sched_setaffinity.