What I have is a text file containing all items that need to be deleted from an online app. Every item that needs to be deleted has to be sent 1 at a time. To make deletion process faster, I divide the items in text file in multiple text files and run the script in multiple terminals (~130 for deletion time to be under 30 minutes for ~7000 items).
This is the code of the deletion script:
from fileinput import filename
from WitApiClient import WitApiClient
import os
dirname = os.path.dirname(__file__)
parent_dirname = os.path.dirname(dirname)
token = input("Enter the token")
file_name = os.path.join(parent_dirname, 'data/deletion_pair.txt')
with open(file_name, encoding="utf-8") as file:
templates = [line.strip() for line in file.readlines()]
for template in templates:
entity, keyword = template.split(", ")
print(entity, keyword)
resp = WitApiClient(token).delete_keyword(entity, keyword)
print(resp)
So, I divide the items in deletion_pair.txt and run this script multiple times in new terminals (~130 terminals). Is there a way to automate this process or do in more efficient manner?
Advertisement
Answer
I used threading to run multiple functions simultaneously:
from fileinput import filename
from WitApiClient import WitApiClient
import os
from threading import Thread
dirname = os.path.dirname(__file__)
parent_dirname = os.path.dirname(dirname)
token = input("Enter the token")
file_name = os.path.join(parent_dirname, 'data/deletion_pair.txt')
with open(file_name, encoding="utf-8") as file:
templates = [line.strip() for line in file.readlines()]
batch_size = 20
chunks = [templates[i: i + batch_size] for i in range(0, len(templates), batch_size)]
def delete_function(templates, token):
for template in templates:
entity, keyword = template.split(", ")
print(entity, keyword)
resp = WitApiClient(token).delete_keyword(entity, keyword)
print(resp)
for chunk in chunks:
thread = Thread(target=delete_function, args=(chunk, token))
thread.start()
It worked! Any one has any other solution, please post or if the same code can be written more efficiently then please do tell. Thanks.