Databricks mount to external storage

databricks_mount_tips.md

Article:

• 'Mount ADLS Gen2 or Blob Storage in Azure Databricks' from 2023-06:

  • https://techcommunity.microsoft.com/t5/azure-paas-blog/mount-adls-gen2-or-blob-storage-in-azure-databricks/ba-p/3802926

• Mount example:

storageAccountName = "<storage-account-name>"
storageAccountAccessKey = "<access-key>"
sasToken = "<sas-token>"
blobContainerName = "<blob-container-name>"
mountPoint = "<mount-point-path>"

if any(mount.mountPoint == mountPoint for mount in dbutils.fs.mounts()):
  dbutils.fs.unmount(mountPoint)

try:
  dbutils.fs.mount(
    source = f"wasbs://{blobContainerName}@{storageAccountName}.blob.core.windows.net",
    mount_point = mountPoint,
    # using access keys: extra_configs = {'fs.azure.account.key.' + storageAccountName + '.blob.core.windows.net': storageAccountAccessKey}
    # using sas tokens: extra_configs = {'fs.azure.sas.' + blobContainerName + '.' + storageAccountName + '.blob.core.windows.net': sasToken}
  )
  print("mount succeeded!")
except Exception as e:
  print("mount exception", e)

• Unmount example: dbutils.fs.unmount(mountPoint)

• List mounts: dbutils.fs.mounts()

• One faced issue example: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: java.lang.IllegalArgumentException: The String is not a valid Base64-encoded string.

  • Solution: add token into Azure Key Vault and access it using dbfs, like dbutils.secrets.get(scope = "<secret-scope>", key = "<key-vault-secret-name>")

• Additional documentation: https://learn.microsoft.com/en-us/azure/databricks/connect/storage/azure-storage