> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hightouch.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks

> Databricks is a data science and analytics platform built on top of Apache Spark. Databricks implement the Data Lakehouse concept in a single unified, cloud based platform.

## Overview

Databricks is a popular source that supports our Reverse ETL, Composable CDP, and AI Decisioning features. This article discusses the steps for connecting to Databricks.

> [!NOTE]
>
> You can connect Databricks to Hightouch using
> [Databricks Partner Connect](https://hightouch.com/blog/hightouch-databricks-partner-connect)
> to bypass the setup steps outlined below. You can learn more about this in
> [Databricks' documentation](https://docs.databricks.com/en/partner-connect/index.html).

> [!NOTE]
>
> You may need to allowlist Hightouch's IP addresses to let our systems connect to your Databricks warehouse. Reference our [networking docs](https://docs.hightouch.com/security/networking#ip-addresses) to determine which IP addresses you need to allowlist.
>
> You can also securely connect to your Databricks warehouse using [AWS PrivateLink](https://docs.hightouch.com/security/aws/privatelink.md#databricks). AWS PrivateLink is a [Business Tier](https://hightouch.com/pricing) feature.

## Databricks credential setup

Hightouch allows you to authenticate with Databricks using one of two methods:
Either [OAuth](https://docs.databricks.com/en/dev-tools/auth/oauth-u2m.html) or
a [Personal Access Token (PAT)](https://docs.databricks.com/en/dev-tools/auth/pat.html).

We recommend using OAuth for configuring new connections.

### Configuring a service principal with OAuth

1. Create a [service principal](https://docs.databricks.com/en/admin/users-groups/service-principals.html).

   1. In your Databricks workspace, go to the user dropdown at the top right and select **Settings**.

   2. In the **Identity and access** page, find **Service Principals** and click **Manage**.

   3. Choose **Add new**, give your service principal a name, and click **Add**.

   4. Take note of the Service Principal name as well as the **Application id** listed on the Service Principal
      Configurations page. This is a UUID (for example, `63fc7e90-a8a2-4639-afd8-36ef6bb67cfa`). You may
      need this later to complete setup.

2. Create an OAuth secret pair.

   1. On the page for the Service Principal created above, go to the Secrets tab.

   2. Click **Generate secret**. A dialog will pop up with a **Secret** and a **Client ID**.
      Take careful note of these values as it is only visible now; you can't look it up later.

3. Grant the necessary catalog permissions to your new Service Principal. The above steps are sufficient
   to connect and authenticate, but the Service Principal may not be *authorized* to execute
   queries against your Databricks catalog. Depending on your Databricks workspace configuration,
   it may already be configured with the necessary permissions; but if the connection test fails,
   you may need to follow these steps.

   1. In the Databricks sidebar on the left, click **Catalog** and locate your schema in the tree view.

   2. Select the schema and open the **Permissions** page. Click the **Grant** button.

   3. In the **Principals** input box, type the **name** of your new Service Principal.

      * If you are using the Lightning sync engine (recommended; see below),
        select the **Data Editor** preset or manually grant the following permissions:
        * `USE SCHEMA`
        * `EXECUTE`
        * `READ VOLUME`
        * `SELECT`
        * `MODIFY`
        * `CREATE TABLE`
        * Make sure to enable the checkbox at the bottom to "Also grant `USE CATALOG`"

      * If you are using the Basic sync engine,
        select the **Data Reader** preset or manually grant the following permissions:
        * `USE SCHEMA`
        * `EXECUTE`
        * `READ VOLUME`
        * `SELECT`
        * Make sure to enable the checkbox at the bottom to "Also grant `USE CATALOG`"

4. You may also need to configure *warehouse* permissions. As with the previous step
   concerning *catalog* permissions, your Databricks workspace configuration may or may not
   automatically grant the Service Principal permission to use the compute resources.
   You can read more about it in the [Databricks docs](https://docs.databricks.com/en/compute/clusters-manage.html#compute-permissions).

   1. In the Databricks sidebar, click **Compute** and find your data warehouse. It may
      be located on the **SQL warehouses** page.

   2. On the far right column of the row corresponding to the desired warehouse, click
      the icon for the kebab menu **⋮**, then **Permissions**.
      > [!NOTE]
      >
      > If you do not see the **Permissions** option, you may lack the privileges
      > necessary to configure permissions—even if you are a Databricks workspace
      > admin. If this happens, please contact your Databricks account admin.

   3. Enter the name of your Service Principal and ensure that the setting in the
      dropdown is set to **Can use**. Click **Add**.

### Alternative: Configuring a Personal Access Token (PAT)

1. In your Databricks Account console, go the [**Workspaces** page](https://accounts.cloud.databricks.com/workspaces).
   Select the relevant workspace and then click **Open workspace**.

   ![Databricks workspace settings](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/sources/source-databricks-workspace.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=6e9ae4156038fb60267aceb3023c8ff7)

2. In your workspace, go the **Compute** page and click the relevant cluster. This brings you to its **Configuration** tab.

   ![Databricks cluster settings](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/sources/source-databricks-settings.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=d57c8275a68f7a6861bddae1244ded07)

3. At the bottom of the page, expand the **Advanced Options** toggle and open the **JDBC/ODBC** tab.

   ![Databricks Advanced Options](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/sources/source-databricks-advanced.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=2456806b2c79b7f80737cd633580b267)

4. This tab displays your cluster's **Server Hostname**, **Port**, and **HTTP Path**, which you need to connect to Hightouch.
   Keep the tab open, or save these to a secure location.

5. Create a **Personal access token** by following [the Databricks documentation](https://docs.databricks.com/dev-tools/auth.html#pat).

Once you've saved these Databricks credentials, you're ready to set up the connection in Hightouch.

## Connection configuration

To get started, go to the [**Sources** overview page](https://app.hightouch.com/sources?from=docs) and click the **Add source** button.
Select **Databricks** and follow the steps below.

### Configure your source

Enter the following required fields into Hightouch:

* **Server hostname**: To find your server hostname, visit the Databricks web console and locate your cluster.
  Then, click to reveal Advanced options and navigate to the JDBC/ODBC tab.

* **Port**: The default port number is 443, but yours may be different.
  To find your port, visit the Databricks web console and locate your cluster.
  Then, click to reveal Advanced options and navigate to the JDBC/ODBC tab.

* **HTTP path**: To find your HTTP path, visit the Databricks web console and locate your cluster.
  Then, click to reveal Advanced options and navigate to the JDBC/ODBC tab.

  You can choose to route all queries to a single HTTP path, or configure separate HTTP paths optimized for different workloads (recommended):

  * **Route all queries to a single HTTP path**: Use this option if you want to use a single HTTP path for all Hightouch operations.

  * **Route queries to different HTTP paths (split by workload)** (Recommended): This option allows you to optimize performance by routing different types of queries to different HTTP paths:
    * **Batch HTTP path**: For background jobs like sync runs, schema sampling, audience snapshots, and other workloads that are not latency-sensitive.
    * **Interactive HTTP path**: For latency-sensitive queries (e.g., audience previews) and heavier workloads that need more compute (e.g., identity resolution).

* (Optional) **Catalog**: You can optionally include a Databricks catalog.
  Catalogs are the first level of [Unity Catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog/)'s three-level namespace.
  If Lightning engine is enabled, the 'hightouch\_audit' and 'hightouch\_planner' [schemas](https://docs.hightouch.com/syncs/lightning-sync-engine.md#warehouse-schemas)
  need to be created in the catalog specified here. If this field is left empty,
  these schemas should be created in the [default catalog](https://docs.databricks.com/aws/en/catalogs/default).
  Note that data in other catalogs can still be accessed for model creation by specifying
  the full name (catalog.schema.table) in the Hightouch SQL interface.

* **Schema**: The initial schema to use for the connection.

* **SQL dialect**: By default, Hightouch assumes that your queries use the Databricks SQL dialect.
  You may wish to override this behavior if your queries use legacy ANSI SQL-92 syntax.
  Some features are not available when legacy syntax is used.

### Choose your sync engine

For optimal performance, Hightouch [tracks incremental changes](https://docs.hightouch.com/getting-started/concepts.md#change-data-capture) in your data model—such as added, changed, or removed rows—and only syncs those records. You can choose between two different sync engines for this work.

The **Basic engine** requires read-only access to Databricks. Hightouch executes a query in your database, reads all query results, and then determines incremental changes using Hightouch's infrastructure. This engine is easier to set up since it requires read—not write—access to Databricks.

The **Lightning engine** requires read and write access to Databricks. The engine stores previously synced data in a separate schema in Databricks managed by Hightouch. In other words, the engine uses Databricks to track incremental changes to your data rather than performing these calculations in Hightouch. Therefore, these computations are completed more quickly.

![Standard vs Lightning engine comparison](https://mintcdn.com/hightouch-3c037897/L_vQfA0JQEpeGUUo/images/get-started/sync-engine-comparison.png?fit=max\&auto=format\&n=L_vQfA0JQEpeGUUo\&q=85\&s=1202b36c58e475d85344db87dde2c825)

> [!TIP]
>
> Hightouch recommends using the Lightning sync engine when syncing more than 100,000 rows of data.

If you select the Basic engine, you can switch to the Lightning engine later. Once you've configured the Lightning engine, you can't move back to the Basic engine without recreating Databricks as a source.

To learn more, including migration steps and tips, check out the [Lightning sync engine docs](https://docs.hightouch.com/syncs/lightning-sync-engine.md).

#### Basic versus Lightning engine comparison

The **Lightning sync engine** requires granting write access to your data warehouse,
which makes its setup more involved than the Basic sync engine.
However, it is more performant and reliable than the Basic engine.
This makes it the ideal choice to guarantee faster syncs, especially with large data models.
It also supports more features, such as
[Warehouse Sync Logs](https://docs.hightouch.com/syncs/warehouse-sync-logs.md),
[Match Booster](https://docs.hightouch.com/match-booster/overview.md),
and [Identity Resolution](https://hightouch.com/platform/identity-resolution).

| Criteria | Basic sync engine | Lightning sync engine |
| - | - | - |
| Performance | Slower | Quicker |
| Ideal for large data models (over 100 thousand rows) | No | Yes |
| Reliability | Normal | High |
| Resilience to sync interruptions | Normal | High |
| Extra features | None | Warehouse Sync Logs, Match Booster, Identity Resolution |
| Ease of setup | Simpler | More involved |
| Location of change data capture | Hightouch infrastructure | Databricks schemas managed by Hightouch |
| Required permissions in Databricks | Read-only | Read and write |
| Ability to switch | You can move to the Lightning engine at any time | You can't move to the Basic engine once Lightning is configured |

> [!NOTE]
>
> Hightouch can use **Warehouse Export** with Databricks. Instead of streaming every row through Hightouch, it has Databricks write query results directly into Amazon S3, which Hightouch then reads. This speeds up syncs of large models. Warehouse Export runs automatically when your [sync engine](https://docs.hightouch.com/syncs/lightning-sync-engine.md) and [storage](https://docs.hightouch.com/security/storage.md) support it, and Hightouch falls back to standard export when they don't, so your syncs keep running either way.
>
> Databricks exports to Amazon S3 only, so your workspace storage must be S3.
>
> If Warehouse Export runs into a fixable configuration problem, such as a storage-integration or bucket-permission issue, Hightouch stops using it for the affected syncs so it isn't retried on every run, and shows this on the sync and on this source's configuration page. After you resolve the problem, re-enable Warehouse Export from an individual sync, or for all of this source's syncs at once from the source configuration page, and Hightouch uses it again on the next run.
>
> A one-off problem on a single run, such as a row that exceeds the Warehouse Export size limit, isn't treated as a configuration issue: that run falls back to standard export and says so on the run summary, and Hightouch automatically retries Warehouse Export on the next run.
>
> Databricks writes query results to storage as an external table, authorized by your Databricks workspace's own storage credential rather than credentials from Hightouch. Your workspace needs a Unity Catalog storage credential and external location with write access to the bucket Hightouch uses for this workspace. If you export to [your own storage bucket](https://docs.hightouch.com/security/storage.md), grant that external location access to it.

#### Lightning engine setup

To set up the Lightning engine, ensure that your Databricks user or service principal has the appropriate permissions.
You can do so by running the following SQL snippet.

Before running the snippet, make sure to include your own username, which can be found in the top right corner
of the Databricks web console. Alternatively, you can create an API-only service principal by following
[these instructions](https://docs.databricks.com/administration-guide/users-groups/service-principals.html).

```sql theme={null}
CREATE SCHEMA IF NOT EXISTS hightouch_audit;
CREATE SCHEMA IF NOT EXISTS hightouch_planner;
GRANT ALL PRIVILEGES ON SCHEMA hightouch_audit TO <YOUR_USER>;
GRANT ALL PRIVILEGES ON SCHEMA hightouch_planner TO <YOUR_USER>;
```

> [!NOTE]
>
> The `hightouch_audit` and `hightouch_planner` schemas exist to keep Hightouch's temporary tables, logs, and other
> artifacts separate from your source data. This simplifies governance because you can restrict write access to only
> these Hightouch-managed schemas. We recommend using `GRANT ALL PRIVILEGES` for simplicity, but you can also grant the
> [required privileges](#configuring-a-service-principal-with-oauth) separately to achieve the same effect.
> (“Temporary tables” here refers to tables that Hightouch will automatically clean up once they are no longer
> needed; it doesn't necessarily refer to special table types in the warehouse.)

> [!NOTE]
>
> If you are using a Service Principal to connect, you may need to specify the user
> above by UUID rather than name. For example, if you have created a Service Principal
> named `test_principal` with an **Application Id** of `63fc7e90-a8a2-4639-afd8-36ef6bb67cfa`,
> you may need to run
>
> ```sql theme={null}
> GRANT ALL PRIVILEGES ON SCHEMA hightouch_audit TO `63fc7e90-a8a2-4639-afd8-36ef6bb67cfa`;
> GRANT ALL PRIVILEGES ON SCHEMA hightouch_planner TO `63fc7e90-a8a2-4639-afd8-36ef6bb67cfa`;
> ```

### Provide credentials

Enter the **Access token** you generated in the [Databricks credential setup](#databricks-credential-setup).

### Unity Catalog, external locations, and managed storage

The Lightning sync engine persists change data capture state in managed tables under `hightouch_planner` and `hightouch_audit`. When Unity Catalog manages these tables, Hightouch can apply extra optimizations during the CDC phase that improve throughput and stability on large models.

Hightouch only supports S3 for self-hosted managed storage today. If your workspace uses a cloud provider other than AWS, contact Support before proceeding.

#### Before you begin

* You plan to use Lightning with Databricks Unity Catalog.
* You have the [service principal](#configuring-a-service-principal-with-oauth) Hightouch uses to connect, including its Application id for `GRANT` statements.
* You have a metastore or account administrator (or delegate) who can create Unity Catalog objects, specifically:
  * To create the [storage credential](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/manage-storage-credentials): `CREATE STORAGE CREDENTIAL` on the metastore.
  * To create the external location: `CREATE EXTERNAL LOCATION` on both the metastore and the storage credential that backs it.
  * To create the Lightning schemas: `CREATE MANAGED STORAGE` on the external location plus `USE CATALOG` and `CREATE SCHEMA` on the catalog ([Create schemas](https://docs.databricks.com/aws/en/schemas/create-schema)).
  * To run the `GRANT` statements in step 3: `MANAGE` (or ownership) on the external location and storage credential.
  * Someone who can attach IAM roles to your bucket path is usually involved ([S3 + Unity Catalog](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-cfn)).

#### What to do

1. Pick an S3 path for Hightouch-managed data — typically the same bucket as [self-hosted workspace storage](https://docs.hightouch.com/security/storage.md#amazon-s3) (for example `s3://your-company-hightouch/hightouch-databricks/`). Every `MANAGED LOCATION` you set later must sit inside the external location URL you register.

2. Create a storage credential and an external location in Databricks. Follow [CloudFormation quickstart for S3](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-cfn), [Create a storage credential](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/manage-storage-credentials), and [Create an external location](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/manage-external-locations#create-an-external-location-using-catalog-explorer). Record the external location name and storage credential name.

3. Grant file access to the Hightouch service principal. Replace the names and the Application id (in backticks) below.

   ```sql theme={null}
   GRANT BROWSE, CREATE EXTERNAL TABLE, READ FILES, WRITE FILES
   ON EXTERNAL LOCATION your_hightouch_external_location
   TO `63fc7e90-a8a2-4639-afd8-36ef6bb67cfa`;

   GRANT CREATE EXTERNAL TABLE, READ FILES, WRITE FILES
   ON STORAGE CREDENTIAL your_hightouch_storage_credential
   TO `63fc7e90-a8a2-4639-afd8-36ef6bb67cfa`;
   ```

   Always grant `READ FILES` with `WRITE FILES` — `WRITE FILES` alone returns `PERMISSION_DENIED` ([privilege reference](https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/privileges#write-files)).

4. Create the Lightning schemas inside the catalog you configured in the Hightouch source (`my_catalog` below).

   > [!NOTE]
   >
   > If your catalog already has a `MANAGED LOCATION`, omit the `MANAGED LOCATION` clause below — schemas inherit catalog storage automatically ([managed storage hierarchy](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/managed-storage)).

   ```sql theme={null}
   CREATE SCHEMA IF NOT EXISTS my_catalog.hightouch_audit
     MANAGED LOCATION 's3://your-company-hightouch/hightouch-databricks/audit';

   CREATE SCHEMA IF NOT EXISTS my_catalog.hightouch_planner
     MANAGED LOCATION 's3://your-company-hightouch/hightouch-databricks/planner';
   ```

5. Grant catalog and schema privileges to the Hightouch service principal. `USE CATALOG` is required in addition to schema grants; if you didn't already grant it via the [credential UI](#configuring-a-service-principal-with-oauth), include the first statement below.

   ```sql theme={null}
   GRANT USE CATALOG ON CATALOG my_catalog TO `63fc7e90-a8a2-4639-afd8-36ef6bb67cfa`;

   GRANT ALL PRIVILEGES ON SCHEMA my_catalog.hightouch_audit TO `63fc7e90-a8a2-4639-afd8-36ef6bb67cfa`;
   GRANT ALL PRIVILEGES ON SCHEMA my_catalog.hightouch_planner TO `63fc7e90-a8a2-4639-afd8-36ef6bb67cfa`;
   ```

6. Ensure the service principal has `Can use` on the SQL warehouse Hightouch uses ([warehouse permissions](#configuring-a-service-principal-with-oauth)).

## Test your connection

When setting up a source for the first time, Hightouch validates the following:

* Network connectivity
* Databricks credentials
* Permission to list schemas and tables
* Permission to write to `hightouch_planner` schema
* Permission to write to `hightouch_audit` schema

All configurations must pass the first three, while those with the Lightning engine must pass all of them.

Some sources may initially fail connection tests due to timeouts. Once a connection is established, subsequent API requests should happen more quickly, so it's best to retry tests if they first fail. You can do this by clicking **Test again**.

If you've retried the tests and verified your credentials are correct but the tests are still failing, don't hesitate to [reach out to our Customer Success team](#pylon-chat).

## Create models

Once your source configuration has passed the necessary validation, your source setup is complete. Next, you can set up
[models](https://docs.hightouch.com/getting-started/concepts.md#models) to define which data you want to pull from Databricks.

The Databricks source supports these modeling methods:

* writing a query in [the SQL editor](https://docs.hightouch.com/models/sql-editor.md)
* using the visual [table selector](https://docs.hightouch.com/models/table-selector.md)
* leveraging existing [dbt models](https://docs.hightouch.com/extensions/dbt-models.md)
* leveraging existing [Sigma workbooks](https://docs.hightouch.com/extensions/sigma.md)

The [SQL editor](https://docs.hightouch.com/models/sql-editor.md) allows you to query data from all databases that your Databricks credentials
[have access to](#databricks-credential-setup), unless you [specify a catalog](#configure-your-source).
The [table selector](https://docs.hightouch.com/models/table-selector.md) only supports querying from the schema specified in
the [source configuration](#configure-your-source).

You may also want to consider [storing sync logs](https://docs.hightouch.com/syncs/warehouse-sync-logs.md) in Databricks. Like using the Lightning sync engine versus the standard one, this feature lets you use Databricks instead of Hightouch infrastructure. Rather than performance gains, it makes your sync log data available for more complex analysis. Refer to the [warehouse sync logs docs](https://docs.hightouch.com/syncs/warehouse-sync-logs.md) to learn more.

> [!NOTE]
>
> You must enable the [Lightning sync engine](#choose-your-sync-engine) to store sync logs in your warehouse.

## Tips and troubleshooting

If you run into an issue this page doesn't cover, [contact support](#pylon-chat).

### Unity Catalog support

Hightouch integrates with Databricks' [Unity Catalog](https://docs.databricks.com/en/data-governance/unity-catalog/index.html) feature.
During [initial configuration](#configure-your-source), include a **catalog** when you use Unity Catalog.

### Subquery error

Some Databricks [models](https://docs.hightouch.com/getting-started/concepts.md#models) return the following error:

`java.lang.IllegalArgumentException: requirement failed: Subquery subquery#485, [id=#937] has not finished {...}.`

This error can occur when the model query contains a subquery, especially when the subquery has an `ORDER BY` clause.
For example:

```sql theme={null}
SELECT user_query.*
FROM (
   SELECT * FROM default.subscriptions_table
   ORDER BY last_name
) user_query.*
```

The best way to resolve the error is to rewrite your model query to remove subqueries.
[Common table expressions (CTE)](https://docs.databricks.com/sql/language-manual/sql-ref-syntax-qry-select-cte.html) are supported.
If you require assistance, don't hesitate to [get in touch](#pylon-chat).

### Error connecting to the database. Unauthorized/Forbidden: 403

This error can occur when [testing the source connection](#test-your-connection)
or when running a sync that uses a Databricks model.
It is typically caused by an expired Databricks access token.
To solve this, [generate a new token](https://docs.databricks.com/dev-tools/auth.html#pat)
and insert it in the source configuration.