Skip to content
Advertisement

Python how to download a file from s3 and then reuse

how can I download a file from s3 and then reuse instead of keeping downloading it everytime when the endpoint is called?

    @app.route("/get", methods=['GET'])
    def get():
        s3 = boto3.resource('s3')
        ids = pickle.loads(s3.Bucket('bucket').Object('file').get()['Body'].read())
        ...

Advertisement

Answer

Based on the comments.

Since the files are large (16GB) and need to be read and updated often, instead of S3, an EFS filesystem could be used for their storage:

Amazon Elastic File System (Amazon EFS) provides a simple, serverless, set-and-forget elastic file system for use with AWS Cloud services and on-premises resources.

EFS provides NFS filesystems that you can mount to your instance, or even multiple instances at the same time. You can also mount the same filesystem to ECS containers and lambda functions.

Since EFS provides regular filesystem, you can write and read the files directly in it. There is no need to copy it first as in S3 which is object storage (not filesystem).

Its worth pointing out, that convenience of EFS costs more then using S3. However, now you can reduce the cost of using EFS, if this is a problem, by using just released Amazon EFS One Zone storage class.

User contributions licensed under: CC BY-SA
5 People found this is helpful
Advertisement