How to load data from Azure Blob Storage to Databricks Lakehouse

Begin by ensuring that you have an active Azure Storage account. Within this account, create a Blob Container where your data files are stored. Note the container name, storage account name, and the corresponding access keys since these will be necessary for establishing a connection later.

Create an Azure Service Principal to facilitate secure access to your Blob Storage. This involves registering an application in Azure Active Directory, generating a client secret, and granting the necessary permissions to the storage account. Record the client ID, tenant ID, and client secret for future use.

Access your Databricks workspace and ensure it is properly configured. Create a new cluster if you don’t have one already, and ensure that it has appropriate permissions to access Azure resources.

In a Databricks notebook, use the Azure Service Principal credentials to mount the Blob Storage. This can be accomplished using the Databricks File System (DBFS) utility. Implement the following `dbutils.fs.mount` command, substituting with your actual configuration values:

```python
configs = {
"fs.azure.account.auth.type": "OAuth",
"fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
"fs.azure.account.oauth2.client.id": "",
"fs.azure.account.oauth2.client.secret": "",
"fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com//oauth2/token"
}

dbutils.fs.mount(
source = "abfss://@.dfs.core.windows.net/",
mount_point = "/mnt/",
extra_configs = configs)
```

Once mounted, verify the connection by listing the files in the mounted directory. Use the following command in your Databricks notebook:

```python
display(dbutils.fs.ls("/mnt/"))
```

This command should display the contents of your Azure Blob Storage, confirming successful access.

Use Apache Spark's DataFrame API to read the data from the mounted Blob Storage into Databricks. Here’s an example of reading a CSV file:

```python
df = spark.read.format("csv").option("header", "true").load("/mnt//path/to/yourfile.csv")
df.show()
```

Adjust the format and options based on your data file type (e.g., JSON, Parquet).

Finally, write the DataFrame to your desired format in the Databricks Lakehouse. For example, to write as a Delta table:

```python
df.write.format("delta").mode("overwrite").save("/mnt/lakehouse/your-delta-table")
```

This operation will save the data into Databricks Lakehouse in the Delta format, making it ready for further processing or analysis.

By following these steps, you can seamlessly transfer data from Azure Blob Storage to Databricks Lakehouse using built-in capabilities, without relying on third-party connectors or integrations.