What I have is a text file containing all items that need to be deleted from an online app. Every item that needs to be deleted has to be sent 1 at a time. To make deletion process faster, I divide the items in text file in multiple text files and run the script in multiple terminals (~130 for deletion time to be under 30 minutes for ~7000 items).
This is the code of the deletion script:
from fileinput import filename from WitApiClient import WitApiClient import os dirname = os.path.dirname(__file__) parent_dirname = os.path.dirname(dirname) token = input("Enter the token") file_name = os.path.join(parent_dirname, 'data/deletion_pair.txt') with open(file_name, encoding="utf-8") as file: templates = [line.strip() for line in file.readlines()] for template in templates: entity, keyword = template.split(", ") print(entity, keyword) resp = WitApiClient(token).delete_keyword(entity, keyword) print(resp)
So, I divide the items in deletion_pair.txt and run this script multiple times in new terminals (~130 terminals). Is there a way to automate this process or do in more efficient manner?
Advertisement
Answer
I used threading to run multiple functions simultaneously:
from fileinput import filename from WitApiClient import WitApiClient import os from threading import Thread dirname = os.path.dirname(__file__) parent_dirname = os.path.dirname(dirname) token = input("Enter the token") file_name = os.path.join(parent_dirname, 'data/deletion_pair.txt') with open(file_name, encoding="utf-8") as file: templates = [line.strip() for line in file.readlines()] batch_size = 20 chunks = [templates[i: i + batch_size] for i in range(0, len(templates), batch_size)] def delete_function(templates, token): for template in templates: entity, keyword = template.split(", ") print(entity, keyword) resp = WitApiClient(token).delete_keyword(entity, keyword) print(resp) for chunk in chunks: thread = Thread(target=delete_function, args=(chunk, token)) thread.start()
It worked! Any one has any other solution, please post or if the same code can be written more efficiently then please do tell. Thanks.