Skip to content
Advertisement

multiprocessing with moviepy

Recently I made a script that take a 5 minutes video clip and cuts for 5 video, 1 min each video, it works well, but its taking too long for pc like my, and my pc with very good part performance:

Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz, 2904 Mhz, 8 Core(s), 16 Logical Processor(s)

Installed Physical Memory (RAM) 16.0 GB

So I search on the moviepy’s docs “threads”, I found something in the “write_videofile” function that i can set my threads to speed up, I tried it, but its didnt worked, I mean its worked but its only it changed maybe to more 2 or 3 it/s.

Also I found example code with multithreading but its seems like the code doesnt work because moviepy.multithreading doesnt exists in the moviepy library, Please help me speed up the rendering, Thank you

here is the code that i found:

JavaScript

this is my code:

JavaScript

Advertisement

Answer

Using processes I reduced time only by 15-20 seconds because ffmpeg even in single process was using almost full CPU power and my computer didn’t have power to run other processes faster.


First I reduced code to make it shorter.

Code in try and except had similar elements so I moved them ouside try/except.

Next I used

JavaScript

and I didn’t need try/except at all.

Using os.makedirs(..., exist_ok=True) I don’t need to run it in try/except

Meanwhile I reduced time by 20 seconds using

JavaScript

instead of

JavaScript

This way I don’t write subclip on disk and I don’t have to read it again from disk.


JavaScript

Next I moved code to function with arguments my_process(filename, text, start, end, number, base_folder)

JavaScript

And now I can run function in separated processes using standard module multiprocessing

(or standard modules threading, concurrent.futures or external modules Joblib, Ray, etc.).

It starts single process

JavaScript

but if I use it in loop then I will start many processes at the same time.


JavaScript

Previous version for 11 subclips starts 11 processes. Using Pool(4) you can put all processes in pool and it will run 4 processes at the same time. When one process will finish task then it will start next process with new arguments.

This time I use loop to create list with arguments for all processes

JavaScript

and I use this list with Pool and it will do the rest.

JavaScript

Pool may start processes in different order but if they use return to send some result then Pool will give results in correct order.

JavaScript
User contributions licensed under: CC BY-SA
1 People found this is helpful
Advertisement