Python asyncio task list generation without executing the function

While working in asyncio, I’m trying to use a list comprehension to build my task list. The basic form of the function is as follows:

import asyncio
import urllib.request as req
@asyncio.coroutine
def coro(term):
    print(term)
    google = "https://www.google.com/search?q=" + term.replace(" ", "+") + "&num=100&start=0"
    request = req.Request(google, None, headers) 
    (some beautiful soup stuff)

My goal is to use a list of terms to create my task list:

terms = ["pie", "chicken" ,"things" ,"stuff"]    
tasks=[
    coro("pie"),
    coro("chicken"),
    coro("things"),
    coro("stuff")]

My initial thought was:

loop = asyncio.get_event_loop()
tasks = [my_coroutine(term) for term in terms]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

This doesn’t create the task list it runs the function during the list comprehension. Is there a way to use a shortcut to create the task list wihout writing every task?

Answer

Your HTTP client does not support asyncio, and you will not get the expected results. Try this to see .wait() does work as you expected:

import asyncio
import random

@asyncio.coroutine
def my_coroutine(term):
    print("start", term)
    yield from asyncio.sleep(random.uniform(1, 3))
    print("end", term)


terms = ["pie", "chicken", "things", "stuff"]
loop = asyncio.get_event_loop()
tasks = [my_coroutine(term) for term in terms]
print("Here we go!")
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

If you use asyncio.gather() you get one future encapsulating all your tasks, which can be easily canceled with .cancel(), here demonstrated with python 3.5+ async def/await syntax (but works the same with @coroutine and yield from):

import asyncio

import random


async def my_coroutine(term):
    print("start", term)
    n = random.uniform(0.2, 1.5)
    await asyncio.sleep(n)
    print("end", term)
    return "Term {} slept for {:.2f} seconds".format(term, n)


async def stop_all():
    """Cancels all still running tasks after one second"""
    await asyncio.sleep(1)
    print("stopping")
    fut.cancel()
    return ":-)"


loop = asyncio.get_event_loop()
terms = ["pie", "chicken", "things", "stuff"]
tasks = (my_coroutine(term) for term in terms)
fut = asyncio.gather(stop_all(), *tasks, return_exceptions=True)

print("Here we go!")
loop.run_until_complete(fut)

for task_result in fut.result():
    if not isinstance(task_result, Exception):
        print("OK", task_result)
    else:
        print("Failed", task_result)

loop.close()

And finally, if you want to use an async HTTP client, try aiohttp. First install it with:

pip install aiohttp

then try this example, which uses asyncio.as_completed:

import asyncio

import aiohttp


async def fetch(session, url):
    print("Getting {}...".format(url))
    async with session.get(url) as resp:
        text = await resp.text()
    return "{}: Got {} bytes".format(url, len(text))


async def fetch_all():
    async with aiohttp.ClientSession() as session:
        tasks = [fetch(session, "http://httpbin.org/delay/{}".format(delay))
                 for delay in (1, 1, 2, 3, 3)]
        for task in asyncio.as_completed(tasks):
            print(await task)
    return "Done."


loop = asyncio.get_event_loop()
resp = loop.run_until_complete(fetch_all())
print(resp)
loop.close()