Trying to use a semaphore to control asynchronous requests to control the requests to my target host but I am getting the following error which I have assume means that my asycio.sleep()
is not actually sleeping. How can I fix this? I want to add a delay to my requests for each URL targeted.
Error:
JavaScript
x
17
17
1
RuntimeWarning: coroutine 'sleep' was never awaited
2
Coroutine created at (most recent call last)
3
File "sephora_scraper.py", line 71, in <module>
4
loop.run_until_complete(main())
5
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 571, in run_until_complete
6
self.run_forever()
7
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 539, in run_forever
8
self._run_once()
9
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 1767, in _run_once
10
handle._run()
11
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/events.py", line 88, in _run
12
self._context.run(self._callback, *self._args)
13
File "makeup.py", line 26, in get_html
14
asyncio.sleep(delay)
15
asyncio.sleep(delay)
16
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
17
Code:
JavaScript
1
29
29
1
import sys
2
import time
3
import asyncio
4
import aiohttp
5
6
async def get_html(semaphore, session, url, delay=6):
7
await semaphore.acquire()
8
async with session.get(url) as res:
9
html = await res.text()
10
asyncio.sleep(delay)
11
semaphore.release()
12
return html
13
14
async def main():
15
categories = {
16
"makeup": "https://www.sephora.com/shop/"
17
}
18
semaphore = asyncio.Semaphore(value=1)
19
tasks = []
20
async with aiohttp.ClientSession(loop=loop, connector=aiohttp.TCPConnector(ssl=False)) as session:
21
for category, url in categories.items():
22
# Get HTML of all pages
23
tasks.append(get_html(semaphore, session, url))
24
res = await asyncio.gather(*tasks)
25
26
if __name__ == "__main__":
27
loop = asyncio.get_event_loop()
28
loop.run_until_complete(main())
29
Advertisement
Answer
JavaScript
1
2
1
asyncio.sleep(delay)
2
Change it to:
JavaScript
1
2
1
await asyncio.sleep(delay)
2
asyncio.sleep
is a coroutine and should be awaited.