I have a s3 folder with a csv file stored on it. I’m trying to download the last modified file. I’m using this script to get the last modified file:
JavaScript
x
12
12
1
s3_client = boto3.client('s3', aws_access_key_id=s3_extra_data['aws_access_key_id'],
2
aws_secret_access_key=s3_extra_data['aws_secret_access_key'])
3
response = s3_client.list_objects_v2(Bucket='test', Prefix='file_r/')
4
all = response['Contents']
5
latest = max(all, key=lambda x: x['LastModified'])
6
response = s3_client.list_objects_v2(Bucket='test', Prefix=latest["Key"])[:-52].lower())
7
all = response['Contents']
8
latest = max(all, key=lambda x: x['LastModified'])
9
print("LATEST ->" + str(latest["Key"])[:-52].lower())
10
print("PATH ->" + str(latest["Key"]))
11
s3_client.download_file("test", latest["Key"], str(latest["Key"]))
12
This code lists my last modified object, the file name is part-00000-40f267f2-38dc-4bab-811c-4c3052fdb1ba-c000.csv
and is inside the file_r
folder.
Although, when I use s3_client.download_file
i get the following error:
JavaScript
1
2
1
'file_r/part-00000-40f267f2-38dc-4bab-811c-4c3052fdb1ba-c000.csv.8cEebaeb'
2
When i print my path and my file I get the correct values
JavaScript
1
3
1
LATEST -> file_r/part
2
PATH -> file_r/part-00000-40f267f2-38dc-4bab-811c-4c3052fdb1ba-c000.csv
3
Why the value .8cEebaeb
is appended after the .csv
extension since the PATH is correct.
Any thoughs on that?
Advertisement
Answer
To solve the problem I changed the code to:
JavaScript
1
7
1
s3_client = boto3.client('s3', aws_access_key_id=s3_extra_data['aws_access_key_id'],
2
aws_secret_access_key=s3_extra_data['aws_secret_access_key'])
3
response = s3_client.list_objects_v2(Bucket='test', Prefix='file_r/')
4
all = response['Contents']
5
latest = max(all, key=lambda x: x['LastModified'])
6
s3_client.download_file("test", latest["Key"], "FILE_NAME"))
7