While working in asyncio, I’m trying to use a list comprehension to build my task list. The basic form of the function is as follows:
import asyncio import urllib.request as req @asyncio.coroutine def coro(term): print(term) google = "https://www.google.com/search?q=" + term.replace(" ", "+") + "&num=100&start=0" request = req.Request(google, None, headers) (some beautiful soup stuff)
My goal is to use a list of terms to create my task list:
terms = ["pie", "chicken" ,"things" ,"stuff"] tasks=[ coro("pie"), coro("chicken"), coro("things"), coro("stuff")]
My initial thought was:
loop = asyncio.get_event_loop() tasks = [my_coroutine(term) for term in terms] loop.run_until_complete(asyncio.wait(tasks)) loop.close()
This doesn’t create the task list it runs the function during the list comprehension. Is there a way to use a shortcut to create the task list wihout writing every task?
Advertisement
Answer
Your HTTP client does not support asyncio, and you will not get the expected results. Try this to see .wait()
does work as you expected:
import asyncio import random @asyncio.coroutine def my_coroutine(term): print("start", term) yield from asyncio.sleep(random.uniform(1, 3)) print("end", term) terms = ["pie", "chicken", "things", "stuff"] loop = asyncio.get_event_loop() tasks = [my_coroutine(term) for term in terms] print("Here we go!") loop.run_until_complete(asyncio.wait(tasks)) loop.close()
If you use asyncio.gather()
you get one future encapsulating all your tasks, which can be easily canceled with .cancel()
, here demonstrated with python 3.5+ async def
/await
syntax (but works the same with @coroutine
and yield from
):
import asyncio import random async def my_coroutine(term): print("start", term) n = random.uniform(0.2, 1.5) await asyncio.sleep(n) print("end", term) return "Term {} slept for {:.2f} seconds".format(term, n) async def stop_all(): """Cancels all still running tasks after one second""" await asyncio.sleep(1) print("stopping") fut.cancel() return ":-)" loop = asyncio.get_event_loop() terms = ["pie", "chicken", "things", "stuff"] tasks = (my_coroutine(term) for term in terms) fut = asyncio.gather(stop_all(), *tasks, return_exceptions=True) print("Here we go!") loop.run_until_complete(fut) for task_result in fut.result(): if not isinstance(task_result, Exception): print("OK", task_result) else: print("Failed", task_result) loop.close()
And finally, if you want to use an async HTTP client, try aiohttp. First install it with:
pip install aiohttp
then try this example, which uses asyncio.as_completed
:
import asyncio import aiohttp async def fetch(session, url): print("Getting {}...".format(url)) async with session.get(url) as resp: text = await resp.text() return "{}: Got {} bytes".format(url, len(text)) async def fetch_all(): async with aiohttp.ClientSession() as session: tasks = [fetch(session, "http://httpbin.org/delay/{}".format(delay)) for delay in (1, 1, 2, 3, 3)] for task in asyncio.as_completed(tasks): print(await task) return "Done." loop = asyncio.get_event_loop() resp = loop.run_until_complete(fetch_all()) print(resp) loop.close()