I have one web scraping function which fetches data of 190 URL’s. To complete it fast I used concurrent.future.Threadpool.executor. I am saving that data to SQL Server database. I have to do these all process repeatedly to every 3 mins from 9AM to 4PM. But when I use while loop or scheduler that concurr…
Tag: multithreading
Merge using threads not working in python
I have to merge two lists and every time a full the lists in order to merge them , but what is happening that I did it like this : but what is happening that filling_buffer_thread and create_inverted_index_thread just called one time, and not working again, when I debugged the code I see that filling_buffer_t…
why there is no race condition using global python variable
I have the following code: What I don’t understand is why I do not have a race condition here? We have global variable, two threads, it suppose to give different values when subtracting and adding, but for me it’s always giving 0. How to make race condition in this example? Is the global variable …
How can I prevent other threads from running when a certain condition is met?
I created a situation in which two threads run continuously. My intention is to prevent all other threads from running when the “Printing Alphabet” portion of the “printAlphabet” function is entered, and when this prioritized thread is done running, all threads resume execution until t…
How to run periodic serial communication in a separate thread in python
I am programming a GUI with tkinter that controls the communication to multiple motor controllers that are connected serially via RS485. I have a function (get_status()) that requests the status (motor running, motor position, position reached, …) of the controllers via pySerial and displays the uptated…
How to scrape websites with threads – 1 IP per thread?
I have 60 proxies (residential, with username and password). I want to scrape 10000 webpages. I want to rotate over the IPs, so that 1 IP per thread is used every 1 second. So every second there are 60 threads, each thread scraping 1 page. But I just can’t do it. The best I was able to do is the
Worker thread does not respond to slot calls from main thread
For my project based on Python and Qt I wanted to move expensive calculations and functions providing server/client functions into separate threads, to unfreeze my GUI. While leaving them running, I still want them to check periodically if there is new data from the main thread. For testing, I therefore imple…
Right way to do parallel work in python with files and db?
I have very large number of file names (from my PC) inserted in db with status New by default. I want for every file name do some operations (change file). During change file change file status to Proccesing. After operations change status on Processed. I deside to do it with multiprocessing python module. Ri…
Python: will a thread ever unblock while hanging on a `Queue.get(block=True)` call if the queue is suddenly destroyed from another thread/process?
TLDR: Would a blocking get be unblocked by a queue that would be terminated in some way? Long question: Okay, so I know that the thread will hang if the queue (multiprocessing.Queue) remains empty forever while trying to fetch something from it with a blocking get. But suppose now that in another thread or pr…
How can I improve processing time with threads on Spyder?
I’m trying to change the date format of a column in a CSV. I know there are easier ways to do this, but the goal here is to get the threads working properly. I work with Spyder and Python 3.8. My code works as follows: I create a thread class with a function to change the date format I split