Fire-and-forget upload to S3 from a Lambda function

I have a lambda function where, after computation is finished, some calls are made to store metadata on S3 and DynamoDB.

The S3 upload step is the biggest bottleneck in the function, so I’m wondering if there is a way to “fire-and-forget” these calls so I don’t have do wait for them before the function returns.

Currently I’m running all the upload calls in parallel using asyncio, but the boto3/S3 put_object call is still a big bottle neck. I tried using asyncio.create_task to run coroutines without waiting for them to finish, but as expected, I get a bunch of Task was destroyed but it is pending! errors and the uploads don’t actually go through.

If there was a way to do this, we could save a lot on billing since as I said S3 is the biggest bottleneck. Is this possible or do I have to deal with the S3 upload times?

Answer

If there was a way to do this,

Sadly there is not, unless you are going to use other lambda function to do the upload for you. This way your main function would delegate time consuming file processing and upload to a second function in an asynchronous way. Your main function can then return immediately to the caller, and the second function does that heavy work in the background.

Either way, you will have to pay for the first or second function’s execution time.