I have a file to download (download path extracted from json. eg: http://testsite/abc.zip
).
I need a help to perform, all the 5 threads should download the abc.zip
file to the output directory and the download has to be Asynchronous or concurrent.
Currently with the below code it does download the file 5 times but it downloads one by one (Synchronous).
What I want is, the download to be simultaneous.
JavaScript
x
14
14
1
def dldr(file=file_url, outputdir=out1):
2
local_fn = str(uuid.uuid4())
3
if not os.path.exists(outputdir):
4
os.makedirs(outputdir)
5
s = datetime.now()
6
urllib.urlretrieve(file, outputdir + os.sep + local_fn)
7
e = datetime.now()
8
time_diff = e - s
9
logger(out1, local_fn, time_diff)
10
11
for i in range(1, 6):
12
t = threading.Thread(target=dldr())
13
t.start()
14
I have read Requests with multiple connections post and it’s helpful, but doesn’t address the requirement of the question asked.
Advertisement
Answer
I use threading module for download threads:
Also requests, but you can change that to urllib by yourself.
JavaScript
1
19
19
1
import threading
2
import requests
3
4
def download(link, filelocation):
5
r = requests.get(link, stream=True)
6
with open(filelocation, 'wb') as f:
7
for chunk in r.iter_content(1024):
8
if chunk:
9
f.write(chunk)
10
11
def createNewDownloadThread(link, filelocation):
12
download_thread = threading.Thread(target=download, args=(link,filelocation))
13
download_thread.start()
14
15
for i in range(0,5):
16
file = "C:\test" + str(i) + ".png"
17
print file
18
createNewDownloadThread("http://stackoverflow.com/users/flair/2374517.png", file)
19