Django: Use makedirs in AWS S3

I have code that will automate CSVs and it will create a dir using makedirs with uuid dirname. The code is working on my local machine but not in S3.

I am using an href to download the csv file by passing file_path in context.

views.py

def makedirs(path):
try:
    os.makedirs(path)
except OSError as e:
    if e.errno != errno.EEXIST:
        raise
return path

def ..

tmp_name = str(uuid.uuid4())
file_path = 'static/tmp/'+tmp_name+'/'
file_path = makedirs(file_path)
reviews_df.to_csv(file_path+'file_processed.csv', index=False)

Thanks a lot!

Answer

For python and s3 you need to have permissions to write to the bucket and you need to know the bucket name. Here, I’ll assume you have both.

In the AWS python SDK boto3 you have either a client or a resource-although not all services will have both the most commonly used ones usually do.

# s3 client
import boto3
s3_client = boto3.client('s3')
s3_client.upload_file(local_file_name, bucket_name, key_in_s3)

The client is usually a lower level aspect of the AWS service and is usually more useful if you are writing some infrastructure code yourself. A resource-usually-is a higher level of abstraction.

# s3 resource
s3_resource = boto3.resource('s3')
bucket = s3_resource.Bucket(bucket_name)
bucket.upload_file(local_file_name, key_in_s3)

Also choosing to use a resource does not preclude you from using a client for the same AWS service. You can do so by using the meta attribute on the resource. For example to use the client upload way above but using the s3 resource rather than client you would do:

# client from resource
s3_resource.meta.client.upload_file(local_file_name, bucket_name, key_in_s3)

but usually you wouldn’t do that because the bucket under the resource has the option for you already.

Even though the key_in_s3 doesn’t need to be the same as what you have in your local filesystem it is likely a good idea (for your sanity) to use the same value unless you have requirements otherwise.

Here it sounds like your requirement is to have the s3 key be a uuid. If you plan to download the files at a future point and want to avoid the hassle of selecting the application you might want to include the .csv extension after the uuid when you upload to s3. Not a requirement from the s3 side though.

Q: How to make a directory?

A: The bucket is really the directory. Inside a bucket, there isn’t really a concept of directory in S3. The / characters are used to delineate lists of objects in the AWS console and in the AWS CLI but that is more of a convenience. If you want to have that then you can construct it in the code where you write the s3 key. The In other words the way you have your file path and file name in a single string should work fine.