I want to process/download .gz files from S3 bucket. There are more than 10,000 files on S3 so I am using This lists .txt files which I want to avoid. How can I do that? Answer The easiest way to filter objects by name or suffix is to do it within Python, such as using .endswith() to include/exclude objects. …
Tag: amazon-s3
Json.loads error on reading mutiple json files from s3 bucket
when I am passing manually json file as key to load in python shell then its working fine. code below However when using for loop to read JSON object from s3 bucket, then I am getting the error Answer Some of the entries in bucket.objects don’t have any JSON in the data, so check for that and skip them.
Amazon S3 boto3 how to iterate through objects in a bucket?
In a flask app, I was trying to iterate through objects in a S3 Bucket and trying to print the key/ filename but my_bucket.objects.all() returns only the first object in the bucket. It’s not returning the all the objects. The output is [001.pdf] instead of [001, 002, 003, 004, 005] Answer You are exitin…
How to upload a file from a Flask’s HTML form to an S3 bucket using python?
I have an HTML form (implemented in Flask) for uploading files. And I want to store the uploaded files directly to S3. The relevant part of the Flask implementation is as follows: I then use boto3 to upload the file to S3 as follows: file is a werkzeug.datastructures.FileStorage object. But I get the followin…
Tensorflow reading data from AWS s3 bucket
I want to stream my data files from an AWS s3 bucket. I’m following the setup described here, but using tensorflow 2. The setup specifies that you can use a AWS configuration file in ~/.aws/credentials, but I also tried using the environment variables. However the smoke test below keeps giving the follo…
boto3 s3 Object expiration “MalformedXML” error
I’m trying to set the lifecycle configuration of a subdirectory in Amazon S3 bucket by using boto3 put_bucket_lifecycle_configuration. I used this code from aws documentation as referece: I removed Transitions and added Expiration, to better fit my purpouses. Here is my code: The error I’m receivi…
Lambda Function to write to csv and upload to S3
I have a Python Script that gets the details of the unused security groups. I want that to write into a CSV file and upload to S3 Bucket. When I test it in local machine it writes to CSV in the local machine. But when I execute that as a lambda function, it needs a place to save the CSV.
Pandas read_pickle from s3 bucket
I am working on a Jupyter notebook from AWS EMR. I am able to do this: pd.read_csv(“s3:\mypath\xyz.csv’). However, if I try to open a pickle file like this, pd.read_pickle(“s3:\mypath\xyz.pkl”) I am getting this error: However, I can see both xyz.csv and xyz.pkl in the same path! Can a…
How to use a pretrained model from s3 to predict some data?
I have trained a semantic segmentation model using the sagemaker and the out has been saved to a s3 bucket. I want to load this model from the s3 to predict some images in sagemaker. I know how to predict if I leave the notebook instance running after the training as its just an easy deploy but doesn’t …
ERROR: Bucket name must match the regex “^[a-zA-Z0-9.-_]{1,255}$”
When I try to upload images to a bucket, it throw an error “Invalid bucket name “thum.images “: Bucket name must match the regex “^[a-zA-Z0-9.-_]{1,255}$””. I think there is nothing wrong with a bucket name. This is my code to upload image: Answer The “Invalid bucket …