Skip to content

Tag: aws-glue

Query S3 from Python

I am using python to send a query to Athena and get table DDL. I am using start_query_execution and get_query_execution functions in the awswrangler package. The code above creates a dict object that stores query results in an s3 link. The link can be accessed by res[‘ResultConfiguration’][‘OutputLocation’]. It’s a text link: s3://…..txt Can someone help me figure how to access

Get tables from AWS Glue using boto3

I need to harvest tables and column names from AWS Glue crawler metadata catalogue. I used boto3 but constantly getting number of 100 tables even though there are more. Setting up NextToken doesn’t help. Please help if possible. Desired results is list as follows: lst = [table_one.col_one, table_one.col_two, table_two.col_one….table_n.col_n] UPDATED code, still need to have tablename+columnname: Answer Adding sub-loop did

AWS region in AWS Glue

How can I get the region in which the current Glue job is executing? When the Glue job starts executing, I see the output Detected region eu-central-1. In AWS Lambda, I can use the following lines to fetch the current region: However, it seems like the AWS_REGION environment variable is not present in Glue and therefore a KeyError is raised: