I have some tables in aws glue data catalog which have been created by crawling the data from S3 buckets.I am writing my own python shell script to perform some data trasformations for data in those tables.But how can I make the connection to those tables in data catalog via python script?
Advertisement
Answer
If you want to access Glue catalog tables inside a python shell job then you can leverage aws-data-wrangler library.Refer to this on how you can import it into your python shell job.
Also this and this has more examples on how you can read tables from Glue catalog.Below is a simple example that you can use to achieve this :
dtype = wr.catalog.get_table_types(database="awswrangler_test", table="csv_crawler") df = wr.athena.read_sql_table(database="awswrangler_test", table="csv_crawler")