Tag: azure-databricks

script to get the file last modified date and file name pyspark

azure-databricks databricks pyspark python

I have a mount point location which is pointing to a blob storage where we have multiple files. We need to find the last modified date for a file along with the file name. I am using the below script and the list of files are as below: Answer If you’re using operating system-level commands to get file information, then

localhost refused to connect in a databricks notebook calling the google api

azure-databricks databricks google-api google-drive-api python

I read the Google API documentation pages (Drive API, pyDrive) and created a databricks notebook to connect to the Google drive. I used the sample code in the documentation page as follow: The CRED_PATH includes the credential file path in /dbfs/FileStore/shared_uploads. The script prompts me the URL to authorize the application but immediately after allowing access it redirects to the

Uploading files from Azure Blob Storage to SFTP location using Databricks?

azure azure-databricks pyspark python scala

I have a scenario where I need to copy files from Azure Blob Storage to SFTP location in Databricks Is there a way to achieve this scenario using pySpark or Scala? Answer Regarding the issue, please refer to the following steps (I use scala) Mount Azure Blob storage containers to DBFS Copy these file to clusters local file system Code.

install python packages using init scripts in a databricks cluster

azure-databricks bash cluster-computing linux python

I have installed the databricks cli tool by running the following command pip install databricks-cli using the appropriate version of pip for your Python installation. If you are using Python 3, run pip3. Then by creating a PAT (personal-access token in Databricks) I run the following .sh bash script: python_dependencies.sh script I use the above script to install python libraries

Removing non-ascii and special character in pyspark dataframe column

apache-spark-sql azure-databricks pyspark python

I am reading data from csv files which has about 50 columns, few of the columns(4 to 5) contain text data with non-ASCII characters and special characters. I am trying to remove all the non-Ascii and special characters and keep only English characters, and I tried to do it as below There are no spaces in my column name. I