I have one web scraping function which fetches data of 190 URL’s. To complete it fast I used concurrent.future.Threadpool.executor. I am saving that data to SQL Server database. I have to do these all process repeatedly to every 3 mins from 9AM to 4PM. But when I use while loop or scheduler that concurrent future not works. No error and
Tag: multithreading
Merge using threads not working in python
I have to merge two lists and every time a full the lists in order to merge them , but what is happening that I did it like this : but what is happening that filling_buffer_thread and create_inverted_index_thread just called one time, and not working again, when I debugged the code I see that filling_buffer_thread stopped I don’t know if
How can I prevent other threads from running when a certain condition is met?
I created a situation in which two threads run continuously. My intention is to prevent all other threads from running when the “Printing Alphabet” portion of the “printAlphabet” function is entered, and when this prioritized thread is done running, all threads resume execution until the condition is met again. The “anotherThread” function continues to run even when this section is
How to run periodic serial communication in a separate thread in python
I am programming a GUI with tkinter that controls the communication to multiple motor controllers that are connected serially via RS485. I have a function (get_status()) that requests the status (motor running, motor position, position reached, …) of the controllers via pySerial and displays the uptated status on the UI. I want to run this funktion every second. The problem
How to scrape websites with threads – 1 IP per thread?
I have 60 proxies (residential, with username and password). I want to scrape 10000 webpages. I want to rotate over the IPs, so that 1 IP per thread is used every 1 second. So every second there are 60 threads, each thread scraping 1 page. But I just can’t do it. The best I was able to do is the
Worker thread does not respond to slot calls from main thread
For my project based on Python and Qt I wanted to move expensive calculations and functions providing server/client functions into separate threads, to unfreeze my GUI. While leaving them running, I still want them to check periodically if there is new data from the main thread. For testing, I therefore implemented the following demo code: My main aim was to
Right way to do parallel work in python with files and db?
I have very large number of file names (from my PC) inserted in db with status New by default. I want for every file name do some operations (change file). During change file change file status to Proccesing. After operations change status on Processed. I deside to do it with multiprocessing python module. Right now i have this solution but
Python: will a thread ever unblock while hanging on a `Queue.get(block=True)` call if the queue is suddenly destroyed from another thread/process?
TLDR: Would a blocking get be unblocked by a queue that would be terminated in some way? Long question: Okay, so I know that the thread will hang if the queue (multiprocessing.Queue) remains empty forever while trying to fetch something from it with a blocking get. But suppose now that in another thread or process, I close the queue with
How can I improve processing time with threads on Spyder?
I’m trying to change the date format of a column in a CSV. I know there are easier ways to do this, but the goal here is to get the threads working properly. I work with Spyder and Python 3.8. My code works as follows: I create a thread class with a function to change the date format I split
How to implement multiprocessing in my python script?
So the thing I’m trying to do is to have multiple cores process some stuff…. I have a py script which basically makes a feature vector for my ML and DL models. Now the thing I’m doing is that I take a txt file, I compare it with a list that I have initialized at runtime. So the code takes