> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hightouch.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks

> Empower your operations by syncing modeled data into your warehouse

## Supported syncing

| Type | Description | Supported Sync Modes |
| - | - | - |
| Any data set | Sync data from any source to a Databricks table | Upsert, Insert, Mirror |

For more information about sync modes, refer to the [sync modes](https://docs.hightouch.com/syncs/types-and-modes.md#sync-modes) docs.

## Connect to Databricks

Go to the [**Destinations** overview page](https://app.hightouch.com/destinations?from=docs) and click the **Add destination** button. Select **Databricks** and click **Continue**. You can then authenticate Hightouch to **Databricks**.

Hightouch allows you to authenticate with Databricks using one of two methods:
Either [OAuth](https://docs.databricks.com/en/dev-tools/auth/oauth-u2m.html) or
a [Personal Access Token (PAT)](https://docs.databricks.com/en/dev-tools/auth/pat.html).

We recommend using OAuth for configuring new connections.

### Configuring a service principal with OAuth

1. Create a [service principal](https://docs.databricks.com/en/admin/users-groups/service-principals.html).

2. In your Databricks workspace, go to the user dropdown at the top right and select **Settings**.

3. In the **Identity and access** page, find **Service Principals** and click **Manage**.

4. Choose **Add new**, give your service principal a name, and click **Add**.

5. Take note of the Service Principal name as well as the **Application id** listed on the Service Principal
   Configurations page. This is a UUID (for example, `63fc7e90-a8a2-4639-afd8-36ef6bb67cfa`). You may
   need this later to complete setup.

6. Create an OAuth secret pair.

7. On the page for the Service Principal created above, to go the Secrets tab.

8. Click **Generate secret**. A dialog will pop up with a **Secret** and a **Client ID**.
   Take careful note of these values as it is only visible now; you can't look it up later.

9. Grant the necessary catalog permissions to your new Service Principal. The above steps are sufficient
   to connect and authenticate, but the Service Principal may not be *authorized* to execute
   queries against your Databricks catalog. Depending on your Databricks workspace configuration,
   it may already be configured with the necessary permissions; but if the connection test fails,
   you may need to follow these steps.

10. In the Databricks sidebar on the left, click **Catalog** and locate your schema in the tree view.

11. Select the schema and open the **Permissions** page. Click the **Grant** button.

12. In the **Principals** input box, type the **name** of your new Service Principal.

select the **Data Editor** preset or manually grant the following permissions:

* `USE SCHEMA`
* `EXECUTE`
* `CREATE VOLUME`
* `READ VOLUME`
* `SELECT`
* `MODIFY`
* `CREATE TABLE`
* Make sure to enable the checkbox at the bottom to "Also grant `USE CATALOG`"

These permissions apply to a schema you've already created. If you use Databricks as an [event storage destination](https://docs.hightouch.com/events/warehouses/warehouse.md#databricks) and want Hightouch to create the schema for you, also grant `CREATE SCHEMA` on the catalog.

1. You may also need to configure *warehouse* permissions. As with the previous step
   concerning *catalog* permissions, your Databricks workspace configuration may or may not
   automatically grant the Service Principal permission to use the compute resources.
   You can read more about it in the [Databricks docs](https://docs.databricks.com/en/compute/clusters-manage.html#compute-permissions).

2. In the Databricks sidebar, click **Compute** and find your data warehouse. It may
   be located on the **SQL warehouses** page.

3. On the far right column of the row corresponding to the desired warehouse, click
   the icon for the kebab menu **⋮**, then **Permissions**.

> [!NOTE]
>
> If you do not see the **Permissions** option, you may lack the privileges
> necessary to configure permissions—even if you are a Databricks workspace
> admin. If this happens, please contact your Databricks account admin.

1. Enter the name of your Service Principal and ensure that the setting in the
   dropdown is set to **Can use**. Click **Add**.

### Alternative: Configuring a Personal Access Token (PAT)

1. In your Databricks Account console, go the [**Workspaces** page](https://accounts.cloud.databricks.com/workspaces).
   Select the relevant workspace and then click **Open workspace**.

![Databricks workspace settings](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/sources/source-databricks-workspace.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=6e9ae4156038fb60267aceb3023c8ff7)

2. In your workspace, go the **Compute** page and click the relevant cluster. This brings you to its **Configuration** tab.

![Databricks cluster settings](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/sources/source-databricks-settings.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=d57c8275a68f7a6861bddae1244ded07)

3. At the bottom of the page, expand the **Advanced Options** toggle and open the **JDBC/ODBC** tab.

![Databricks Advanced Options](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/sources/source-databricks-advanced.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=2456806b2c79b7f80737cd633580b267)

4. This tab displays your cluster's **Server Hostname**, **Port**, and **HTTP Path**, which you need to connect to Hightouch.
   Keep the tab open, or save these to a secure location.

5. Create a **Personal access token** by following [the Databricks documentation](https://docs.databricks.com/dev-tools/auth.html#pat).

Once you've saved these Databricks credentials, you're ready to set up the connection in Hightouch.

## Sync configuration

Once you've set up your Databricks destination and have a [model](https://docs.hightouch.com/getting-started/concepts.md#models) to pull data from, you can set up your sync configuration to begin syncing data. Go to the [**Syncs** overview page](https://app.hightouch.com/syncs?from=docs) and click the **Add sync** button to begin. Then, select the relevant model and the Databricks destination you want to sync to.

Hightouch supports **Upsert** mode using the `MERGE INTO` statement, with the option to [delete removed rows](#delete-behavior), **Insert** mode using the `INSERT INTO` statement, and **Mirror** mode which will `TRUNCATE` the table and then use the `INSERT INTO` statement to insert all rows from the model.

> [!WARNING]
>
> **Mirror** mode will remove any existing data in the table before inserting all rows from the model

### Record matching

To match rows from your model to rows in Databricks, Hightouch requires you to select a unique identifier in the table you are syncing to.
The model column you select must be the same as the [primary key](https://docs.hightouch.com/models/creating-models.md#unique-primary-key-requirement) column you selected in your model.

The Databricks column you select must be set as a `PRIMARY KEY` column within Databricks.
You can see columns that fit this criteria as available options in records matching section.
If there are no fields in the dropdown, you must add a unique type column to your Databricks table. Then, click the refresh icon to access the newly created column.

### Column types

If you see type errors, it may be because your model is producing the wrong format. If so, use [typecasting](https://docs.hightouch.com/models/data-types-casting.md) or [liquid templating](https://docs.hightouch.com/syncs/mapping-data.md#liquid-template-format) to resolve the issue.

### Field mapping

You can sync columns from your model to the columns in your Databricks table.

### Batch size

You can tune the number of rows Hightouch upserts or inserts per query based on your needs and database threshold. The default is 50,000 rows per batch.

If you want to increase the sync's speed, you can increase the batch size. Keep in mind that in upsert mode, Databricks fails the entire batch of rows if it detects any erroneous row. If you suspect that you will have many bad rows, don't use a high batch size. To avoid locks, ensure you account for your database's capacity when increasing the batch size.

### Delete behavior

The [delete behavior](https://docs.hightouch.com/syncs/overview.md#delete-behavior) you select dictates what to do when a row no longer appears in your model's query results. You have the following options:

| Behavior | Description |
| - | - |
| **Do nothing** | Keep the row in Databricks |
| **Delete row** | Remove the row from Databricks entirely |

## Tips and troubleshooting

To see the exact API requests and responses from each sync run, use the [live debugger](https://docs.hightouch.com/syncs/debugger.md). To get notified by Slack, PagerDuty, SMS, or email when a sync fails, set up [sync alerts](https://docs.hightouch.com/syncs/alerting.md). If you run into an issue this page doesn't cover, [contact support](#pylon-chat).