I am new to the parallelizing paradigm. I already have the algorithm working in its serial form but I can’t parallelize it. I have asked around and some have told me that the way my program is written can’t be parallelized. Here is my serial code, any help or recommendation will be kindly apprecia…
Tag: parallel-processing
my Python multiprocesses are apparently not independent
I have a very specific problem with python parallelisation let’s see if I can explain it, I want to execute a function foo() using the multiprocessing library for parallelisation. The foo() function is a recursive function who explores a tree in depth until one specific event happens. Depending on how i…
Process a function on different arguments in parallel in Python
This is my simple code where I want to run printRange() in parallel: My question is different from this SO question as here, the each process is hardcoded, started and finally joined. I want to run printRange() in parallel with say 100 other printRange() worker functions. Hardcoding each time is not feasible.…
Running airflow tasks/dags in parallel
I’m using airflow to orchestrate some python scripts. I have a “main” dag from which several subdags are run. My main dag is supposed to run according to the following overview: I’ve managed to get to this structure in my main dag by using the following lines: What I want airflow to do…
Multiprocessing: use only the physical cores?
I have a function foo which consumes a lot of memory and which I would like to run several instances of in parallel. Suppose I have a CPU with 4 physical cores, each with two logical cores. My system has enough memory to accommodate 4 instances of foo in parallel but not 8. Moreover, since 4 of these 8 cores
How to iterate over consecutive chunks of Pandas dataframe efficiently
I have a large dataframe (several million rows). I want to be able to do a groupby operation on it, but just grouping by arbitrary consecutive (preferably equal-sized) subsets of rows, rather than using any particular property of the individual rows to decide which group they go to. The use case: I want to ap…
How do I parallelize a simple Python loop?
This is probably a trivial question, but how do I parallelize the following loop in python? I know how to start single threads in Python but I don’t know how to “collect” the results. Multiple processes would be fine too – whatever is easiest for this case. I’m using currently Li…