asyncio.wait_for does not propagate CancelledError, if waited on future is “done” before cancellation

I’m working on an application that features tasks that can be started by the user. These tasks can also be aborted by the user. The notable thing is, that the cancellation can happen at any given time. I implemented this by using asyncio.tasks.Task which I cancel if the user aborts. I recently updated from Python 3.8.5 to 3.10 (on Windows 10), after which the cancellation of my Tasks stopped working as expected.

After some digging, I think I found where things go awry: These tasks naturally involve a lot of I/O that is also unreliable, so I’m working with timeouts in the form of asyncio.wait_for(). It seems that the CancelledError gets swallowed within wait_for() if the future that is waited on is already done at the time of cancellation. Consider this code snippet from Python 3.8.5:

# asyncio.tasks.wait_for() / Python 3.8.5
# wait until the future completes or the timeout
        try:
            await waiter
        except futures.CancelledError:
            fut.remove_done_callback(cb)
            fut.cancel()
            raise # reraise CancelledError -> all is fine

        if fut.done():
            return fut.result()
        else:
            fut.remove_done_callback(cb)
            # We must ensure that the task is not running
            # after wait_for() returns.
            # See https://bugs.python.org/issue32751
            await _cancel_and_wait(fut, loop=loop)
            raise futures.TimeoutError()

JavaScript
​x
 
# asyncio.tasks.wait_for() / Python 3.8.5
# wait until the future completes or the timeout
        try:
            await waiter
        except futures.CancelledError:
            fut.remove_done_callback(cb)
            fut.cancel()
            raise # reraise CancelledError -> all is fine
​
        if fut.done():
            return fut.result()
        else:
            fut.remove_done_callback(cb)
            # We must ensure that the task is not running
            # after wait_for() returns.
            # See https://bugs.python.org/issue32751
            await _cancel_and_wait(fut, loop=loop)
            raise futures.TimeoutError()
​

versus Python 3.10:

# asyncio.tasks.wait_for() / Python 3.10
# wait until the future completes or the timeout
        try:
            await waiter
        except exceptions.CancelledError:
            if fut.done(): # future already done
                return fut.result() # return result without raising -> task is not aborted
            else:
                fut.remove_done_callback(cb)
                # We must ensure that the task is not running
                # after wait_for() returns.
                # See https://bugs.python.org/issue32751
                await _cancel_and_wait(fut, loop=loop)
                raise

JavaScript
 
# asyncio.tasks.wait_for() / Python 3.10
# wait until the future completes or the timeout
        try:
            await waiter
        except exceptions.CancelledError:
            if fut.done(): # future already done
                return fut.result() # return result without raising -> task is not aborted
            else:
                fut.remove_done_callback(cb)
                # We must ensure that the task is not running
                # after wait_for() returns.
                # See https://bugs.python.org/issue32751
                await _cancel_and_wait(fut, loop=loop)
                raise
​

This piece of code reproduces the error:

import asyncio

async def im_done():
    print("done")

async def nested_wait(timeout):
    await asyncio.wait_for(im_done(), timeout=timeout)

async def test_cancel():
    task = asyncio.create_task(nested_wait(30))
    await asyncio.sleep(0)
    task.cancel()
    await asyncio.sleep(0)
    
    await task # raises on 3.8.5 / doesn't raise on 3.10

asyncio.run(test_cancel())

JavaScript
 
import asyncio
​
async def im_done():
    print("done")
​
async def nested_wait(timeout):
    await asyncio.wait_for(im_done(), timeout=timeout)
​
async def test_cancel():
    task = asyncio.create_task(nested_wait(30))
    await asyncio.sleep(0)
    task.cancel()
    await asyncio.sleep(0)
    
    await task # raises on 3.8.5 / doesn't raise on 3.10
​
asyncio.run(test_cancel())
​

Is this expected behaviour or is this a bug? Note that im_done() in my example is in reality a coroutine that does actual I/O. My tests for aborting always fail, but I don’t fully understand how the “finished” state of the future comes to pass and why the CancelledError is raised within wait_for() if the waited on future is already done. If it is by pure chance and timing or if I’m actually using the API wrong. But since everything was working perfectly before that code change in asyncio.tasks.wait_for I’m guessing this could be a bug.

Answer

Never mind, it seems it is an older bug, that just hasn’t been adressed yet. It has an open issue in Pyton’s issue tracker. I didn’t see it on my first time round the issue tracker. Of course I find it right after I post here. I’ll leave the question anyway for anyone who has the same issue but missed it in the issue tracker.

Advertisement

Answer