Skip to content

Tag: concurrent.futures

Retrying failed futures in Python’s ThreadPoolExecutor

I want to implement retry logic with Python’s concurrent.futures.ThreadPoolExecutor. I would like the following properties: A new future is added to the work queue as soon as it fails. A retried future can be retried again, either indefinitely or up to a maximum retry count. A lot of existing code I found online basically operates in “rounds”, where they call

Identify current thread in concurrent.futures.ThreadPoolExecutor

the following code has 5 workers …. each opens its own worker_task() BUT ….. inside each worker_task() …… I cannot identify … which of the 5 workers is currently being used (Worker_ID) If I want to print(‘worker 3 has finished’) inside worker_task() ….. I cannot do this because executor.submit does not allow Any solutions? Answer You can get name of

Predicting in parallel using concurrent.futures of tensorflow.keras models

I am trying to implement some parallel jobs using concurrent.futures. Each worker requires a copy of a TensorFlow model and some data. I implemented it in the following way (MWE) simple_model() creates the model. clone_model clones a TensorFlow model. work represents an MWE of possible work. worker assigns the work in parallel. This is not working, it just stuck and

Unable to use dynamic classes with concurrent.futures.ProcessPoolExecutor

In the code below, I am dynamically creating an object of the class inside the _py attribute by using the generate_object method. The code works perfectly if I am not using a concurrent approach. However, if I use concurrency from concurrent.futures, I do not get the desired result because of an error saying (beyond other things): After googling this error,

How to create a continuous stream of Python’s concurrent.futures.ProcessPoolExecutor.submits()?

I am able to submit batches of concurrent.futures.ProcessPoolExecutor.submits() where each batch may contain several submit(). However, I noticed that if each batch of submits consumes a significant about of RAM, there can be quite a bit of RAM usage inefficiencies; need to wait for all futures in the batch to be completed before another batch of submit() can be submitted.