I was going through delta lake documentation page. There is a line like this : In the last line, we see a double assignment of same variable (spark). Does this do something different compared to : In general, in python, is there a meaning to double assignment of same variable ? Answer Short answer: It’s meaningless. Aside from a few
Tag: delta-lake
Correct Method to Delete Delta Lake Partion on AWS s3
I need to delete a Delta Lake partition with associated AWS s3 files and then need to make sure AWS Athena displays this change. The purpose is because I need to rerun some code to re-populate the data. I tried this And it completed with no errors but the files on s3 still exist and Athena still shows the data
How to write data to Delta Lake from Kubernetes
Our organisation runs Databricks on Azure that is used by data scientists & analysts primarily for Notebooks in order to do ad-hoc analysis and exploration. We also run Kubernetes clusters for non spark-requiring ETL workflows. We would like to use Delta Lakes as our storage layer where both Databricks and Kubernetes are able to read and write as first class